Douyin Creator Scraping

Douyin Creator Center Scraping

The #1 Pitfall: macOS TCC Blocks launchd from Reading ~/Downloads

This is the root cause of 99% of "download timeout" failures. launchd-spawned Python processes cannot read ~/Downloads/, ~/Documents/, or ~/Desktop/ due to macOS TCC (Transparency, Consent, and Control). glob.glob(), os.listdir(), even osascript -e 'do shell script "ls/cp/cat ..."' — ALL silently return empty or zero from launchd context.

The fix: redirect Chrome's download directory to /tmp/. /tmp has zero TCC restrictions. Chrome writes there freely, Python reads there freely.

# One-time setup — persists across Chrome restarts:
defaults write com.google.Chrome DownloadDirectory -string "/tmp/douyin_dl"

After this, download detection is trivially simple:

def fetch_newest_xlsx():
    import glob
    pattern = os.path.join('/tmp/douyin_dl', '作品列表*.xlsx')
    files = glob.glob(pattern)
    if not files:
        return None, None
    newest = max(files, key=os.path.getmtime)
    return newest, os.path.basename(newest)

Detection time: ~2 seconds. Works identically from terminal and launchd.

What does NOT work (field-tested 2026-05-10)

Pitfall #2: Login Session Expiry — Two Failure Modes

Second most common failure after TCC. Douyin creator center sessions expire after a few days. There are TWO distinct failure modes depending on how expired the session is:

Mode A: Recent Expiry → JSON Error File Downloaded

When the session expired recently (hours to ~1 day), clicking "导出数据" still triggers a download — but the server returns a JSON error instead of an xlsx file:

{"BaseResp":{"StatusCode":8,"StatusMessage":"用户未登录"}}

openpyxl then fails with File is not a zip file. The PK header check catches this.

Mode B: Deep Expiry → No File Downloaded At All (2026-05-23)

When the session has been expired for longer (~1+ days), the server returns an error response that Chrome does NOT interpret as a file download. No file ever appears in the download directory. The wait_for_download PK check never fires because glob returns [] every iteration. The scraper logs only:

点击导出: clicked:3
  等待下载... 10s
  等待下载... 20s
  ...
❌ 等待下载超时

Crucially, there are NO ⚠️ 非xlsx文件 or ❌ 登录态过期 log lines — the file was never downloaded, so the PK validation code path is never reached.

Degradation: Garfish App Stops Initializing

After too many reload attempts with an expired session, the page degrades further. The Garfish micro-frontend stops initializing entirely:

• document.body.innerText shows only sidebar navigation (~66 chars): "高清发布 首页 活动管理..."
• Garfish container has __garfishmockhtml__ attribute — the placeholder HTML, not the live app
• "加载中，请稍候..." text may appear
• document.querySelectorAll('button') returns only 1 button ("高清发布")
• "投稿列表" tab and "导出数据" button are never found

This is the same root cause as Mode B (expired session) but the symptoms are different — it looks like the page failed to load, not that the download failed.

Detection: Is It Session Expiry or Something Else?

⚠️ Health Check Limitation

The pre-flight health check (health_check()) validates AppleScript JS access and tab existence — it does NOT validate the Douyin login session. A health check returning js_ok means the AppleScript pipeline works, but the user may still be logged out.

Recovery: User must re-login to creator.douyin.com in Chrome. No script changes needed. After re-login, verify with a manual scrape run: python3 ~/.hermes/scripts/douyin_hourly.py

Pitfall #3: Page Stuck on Home → Tabs/Buttons Not Found

After login, Chrome may land on the creator home page (/creator-micro/home). The Garfish data-center app never loads there.

Fix: force-navigate URL in reload_page() — never just reload t:

set URL of t to "https://creator.douyin.com/creator-micro/data-center/content"
delay 3
reload t

Pre-flight Health Check (MUST run before scrape)

Before attempting any scrape, validate the AppleScript pipeline with a lightweight JS execution test. This catches 3 common silent failures before wasting time on a full page reload + Garfish wait cycle:

def health_check():
    """Returns (ok, reason). Runs in ~2 seconds if healthy."""
    test_js = "/tmp/dy_health_check.js"
    with open(test_js, "w") as f:
        f.write("(function(){ return 'js_ok'; })()")
    
    script = f'''
    tell application "Google Chrome"
        repeat with w in windows
            repeat with t in tabs of w
                if (URL of t) contains "creator.douyin.com" then
                    set js to read POSIX file "{test_js}"
                    return execute t javascript js
                end if
            end repeat
        end repeat
        return "no_tab"
    end tell
    '''
    try:
        result = subprocess.check_output(['osascript', '-e', script], timeout=15).decode()
    except subprocess.TimeoutExpired:
        return False, "AppleScript 执行超时"
    except Exception as e:
        if "Allow JavaScript" in str(e) or "JavaScript 的功能已关闭" in str(e):
            return False, "🔧 AppleScript JS 已关闭 → Chrome 菜单勾上"
        return False, f"异常: {str(e)[:100]}"
    
    if result == "no_tab":
        return False, "📄 未找到 creator.douyin.com 标签页"
    if "js_ok" not in result:
        return False, f"⚠️ JS 返回异常: {result[:100]}"
    
    return True, "ok"

Call this at the top of main(). If it fails, send the reason string directly to Telegram — the user gets an actionable error message instead of a generic timeout.

Why this matters: AppleScript JS silently resets (Chrome updates, re-logins, desktop switches). Without this check, the scrape silently fails after 30s of polling for buttons that never appear. With it, failure is detected in ~2s with a clear fix instruction.

Troubleshooting: "no_tab" — Diagnostic to List All Douyin Tabs

When the health check returns "no_tab" but the user says Chrome is open, run this diagnostic to find ALL douyin-related tabs across every window. Use execute_code (not terminal) because the AppleScript & string concatenation operator is misinterpreted as shell backgrounding by the terminal tool:

import subprocess

script = '''
tell application "Google Chrome"
    set output to ""
    repeat with w in windows
        repeat with t in tabs of w
            if URL of t contains "douyin" then
                set output to output & (URL of t) & " | win_id:" & (id of w) & linefeed
            end if
        end repeat
    end repeat
    if output is "" then
        return "NO_DOUYIN_TABS"
    end if
    return output
end tell
'''

result = subprocess.check_output(['osascript', '-e', script], timeout=10).decode().strip()
print(result)

This reveals:

• Whether ANY douyin tab exists at all
• Which sub-page each tab is on (/content/manage, /data-center/content, /home, etc.)
• Which Chrome window each tab lives in (useful for Spaces/Desktop debugging)

Common findings:

Prerequisites

• macOS with Google Chrome open and logged into creator.douyin.com
• AppleScript JS execution enabled in Chrome: View → Developer → Allow JavaScript from Apple Events ⚠️ This setting resets frequently (Chrome updates, re-logins, desktop switches).
• Chrome download directory: defaults write com.google.Chrome DownloadDirectory -string "/tmp/douyin_dl"
• ⚠️ macOS desktop aware: Chrome must be on the current active desktop/Space. AppleScript silently fails when Chrome is on a different Space.
• ⚠️ "Allow JavaScript from Apple Events" can silently reset. Symptom: execution error: JavaScript execution via AppleScript is turned off.
• Chrome download directory set to /tmp/douyin_dl/: defaults write com.google.Chrome DownloadDirectory -string "/tmp/douyin_dl"
• Chrome must be on the active macOS desktop — switching spaces can break AppleScript tab operations
• ⚠️ macOS desktop / Space aware: Chrome must be on the current active macOS desktop (Space). AppleScript interactions silently fail or produce stale results when Chrome lives on a different Space. If the scraper suddenly stops finding tabs/buttons after working fine before, check whether Chrome got moved to another desktop.
• ⚠️ "Allow JavaScript from Apple Events" can silently reset. Chrome updates and re-logins can uncheck this setting. Symptom: execution error: "Google Chrome" has encountered an error: JavaScript execution via AppleScript is turned off. Fix: re-check the menu item.

Architecture

The Douyin creator center uses Garfish (ByteDance micro-frontend framework):

• Content lives inside #micro > #garfish_app_for_* (ID suffix changes on reload)
• Use document.querySelectorAll on document (not garfish container)
• document.body.innerText works after full page load + tab switch

Step-by-Step Flow

1. Force-navigate to data center + reload — reload_page() MUST set the URL to https://creator.douyin.com/creator-micro/data-center/content before reloading, NOT just call reload t. Douyin's login redirect often lands on the home page (/creator-micro/home), and reloading there keeps you on home — the Garfish data-center app never loads. The AppleScript MUST:

   set URL of t to "https://creator.douyin.com/creator-micro/data-center/content"
   delay 3
   reload t

   Also: activate + set active tab index + set index of w to 1 are required.
2. Wait 15s for Garfish app to initialize
3. Switch to 投稿列表 tab — charCode match: 25237,31295,21015,34920
4. Click 刷新数据 — charCode: 21047,26032,25968,25454. Wait 4s.
5. Poll for 导出数据 button — up to 15 attempts, 2s each
6. Click first 导出数据 button — charCode: 23548,20986,25968,25454
7. Wait for 作品列表*.xlsx in /tmp/douyin_dl/ — poll with glob every 1s, validate PK header, timeout 40s
8. Parse with openpyxl, delete file after

Code vs SKILL.md Consistency (2026-05-22 Fix)

The script douyin_hourly.py was out of sync with this SKILL.md on two critical points:

1. reload_page() did reload t without setting URL → now sets URL to /data-center/content
2. wait_for_download() called move_latest_xlsx_via_applescript() looking in ~/Downloads → now uses direct glob on /tmp/douyin_dl

Both fixes are in the open-source repo. The SKILL.md was always correct — the code just wasn't following it.

Critical: Tab MUST Be Foreground

Chrome blocks downloads triggered by JS click() on background tabs. Before any interaction, the AppleScript MUST:

tell application "Google Chrome"
    activate
    set active tab index of w to tabIndex
    set index of w to 1
end tell

All three (activate + active tab index + index of w to 1) are required.

AppleScript: No Unicode Escapes

Never use \uXXXX in JS via AppleScript — they fail silently. Use charCodeAt():

// 导出数据: 23548,20986,25968,25454
// 投稿列表: 25237,31295,21015,34920
// 刷新数据: 21047,26032,25968,25454
if(t.charCodeAt(0)===23548 && t.charCodeAt(1)===20986 ...) { ... }

Always write JS to /tmp/*.js file, then read POSIX file in AppleScript.

Excel Schema

作品名称 | 发布时间 | 体裁 | 审核状态 | 播放量 | 完播率 | 5s完播率 |
封面点击率 | 2s跳出率 | 平均播放时长 | 点赞量 | 分享量 | 评论量 | 收藏量 |
主页访问量 | 粉丝增量

~99 rows per export. Percentages as decimals — multiply by 100.

Database

CREATE TABLE video_stats (
    timestamp DATETIME, title TEXT, publish_date TEXT,
    plays INTEGER, avg_duration_sec INTEGER, ctr REAL,
    finish_rate REAL,
    likes INTEGER, comments INTEGER, shares INTEGER,
    favorites INTEGER, danmaku INTEGER, status TEXT
)

• ctr = 5s完播率 (%), stored from xlsx column 「5s完播率」
• finish_rate = 完播率/播放占比 (%), stored from xlsx column 「完播率」 (2026-05-23 新增)
• avg_duration_sec = average watch duration in seconds
• status = 审核状态 («公开», «自见», «私密», «未通过», etc.) — added 2026-05-23. Must use ALTER TABLE ADD COLUMN + explicit column names in INSERT to avoid column ordering bugs (see Pitfall #7). Filter status NOT IN ('自见', '私密', '未通过', '审核中', '已删除') when syncing to Feishu — feishu_sync.py applies this filter in the SELECT query.

publish_date has 3 formats mixed (short ISO / full ISO / Chinese). Always match on title only, never publish_date.

Pitfall #8: status Field Not Stored → All Videos Synced (2026-05-23)

The scraper parsed «审核状态» from xlsx but didn't store it in the DB (line 348 reads it, line 388 didn't include it in INSERT). feishu_sync.py synced ALL 102 videos including 私密/自见 ones. Fix: ① ALTER TABLE video_stats ADD COLUMN status TEXT ② INSERT includes status with explicit column names ③ feishu_sync.py filters WHERE status NOT IN ('自见', '私密', '未通过', '审核中', '已删除'). Cleanup: after fix, manually delete stale Feishu records that match private video titles — sync is upsert-only, never deletes.