Full Record of Implementing Douyin Parser via Cloudflare Workers Reverse Proxy

Background

When developing the AstrBot Douyin parsing plugin, a common but tricky problem was encountered: the server IP was flagged by Douyin's risk control, causing the API request to return empty data. This article records the complete process from a simple retry mechanism to finally using Cloudflare Workers reverse proxy to successfully solve the problem.

Problem Analysis

Initial Problem

When the plugin parses a Douyin link, dysk.py returns None, manifested as:

Videos can be parsed occasionally successfully
Images and live photos cannot be parsed at all
Logs show the API returned an empty response

Root Cause

Douyin's risk control mechanism detects the request source IP. When an abnormal request pattern is detected, it returns an empty response (HTTP 200 but body is empty).

Solution Evolution

Phase 1: Implementing Retry Mechanism

Idea: When parsing fails, recreate a DouyinDownloader instance and retry.

Implementation Points:

Retry up to 5 times
Each retry interval is 5 seconds
Force recreate the instance on retry (get a new Cookie)

Code Snippet:

PYTHON

def _parse_douyin_sync(self, url):
    max_retries = 5
    retry_delay = 5

    for attempt in range(max_retries):
        try:
            current_time = time.time()

            if attempt == 0:
                # First attempt: reuse instance (within 5 minutes)
                if self.dy_downloader is None or (current_time - self.dy_downloader_time) > 300:
                    self.dy_downloader = DouyinDownloader()
                    self.dy_downloader_time = current_time
            else:
                # On retry: force recreate instance
                self.dy_downloader = DouyinDownloader()
                self.dy_downloader_time = current_time

            result = self.dy_downloader.get_detail(url)

            if result is not None:
                return (result, self.dy_downloader)

            if attempt < max_retries - 1:
                time.sleep(retry_delay)
        except Exception as e:
            if attempt < max_retries - 1:
                time.sleep(retry_delay)
            else:
                raise

    return (None, self.dy_downloader)

Effect: The retry mechanism can improve success rate, but it is a temporary fix; once the IP is flagged, parsing still fails.

Phase 2: Introducing Cloudflare Workers Reverse Proxy

Idea:

API requests are proxied through CF Workers, utilizing CF's IP pool to avoid risk control
Video/image downloads connect directly to CDN to save CF traffic

Expected Success Rate: 75-93%

Phase 3: Pitfalls During Implementation

Pitfall 1: CF Workers Response Body Empty

Phenomenon:

Response status: 200
Response headers: {'content-length': '0', 'content-type': 'text/plain'}
Response text length: 0

Cause: CF Workers' automatic gzip compression causes response body transmission failure.

Attempted Solutions:

❌ Return fetch() result directly - response body lost
❌ Use response.arrayBuffer() - returns empty ArrayBuffer
❌ Set Content-Length header - CF still forces compression
✅ Base64 encoding transmission
- successfully bypassed the compression issue

Phenomenon:

Original response length: 20
Original response content (hex): 1f8b08000000000000ff03000000000000000000
Decompressed length: 0

This is an empty gzip file, indicating the Douyin server returned an empty response.

Cause Analysis:

ttwid API request succeeded (Response text length: 205)
Douyin detail API request failed (Response text length: 0)
CF Workers log shows: Request Cookie: No Cookie

Root Cause: Although Python's requests.Session() manages cookies, when proxied through CF Workers, the Cookie is not automatically sent to CF Workers.

Solution: Manually construct the Cookie request header.

Final Solution

Architecture Design

┌─────────────┐      ┌──────────────────┐      ┌─────────────┐
│   Python    │─────▶│  CF Workers      │─────▶│   Douyin API│
│   Client    │      │  (Proxy + Base64) │      │             │
└─────────────┘      └──────────────────┘      └─────────────┘
      │                                                │
      │                                                │
      └────────────────────────────────────────────────┘
                    Direct download video/image

Complete Code

1. Cloudflare Workers Reverse Proxy Script

// cloudflare_worker.js
// Cloudflare Workers reverse proxy script - proxy Douyin API
export default {
  async fetch(request) {
    // Handle OPTIONS preflight request
    if (request.method === "OPTIONS") {
      return new Response(null, {
        status: 200,
        headers: {
          "Access-Control-Allow-Origin": "*",
          "Access-Control-Allow-Methods": "GET, POST, PUT, DELETE, OPTIONS",
          "Access-Control-Allow-Headers": "*",
          "Access-Control-Max-Age": "86400",
        },
      });
    }

    const url = new URL(request.url);

    // Support multiple target domains
    const targetHosts = {
      "douyin": "www.douyin.com",
      "ttwid": "ttwid.bytedance.com"
    };

    // Extract target type from path
    // e.g.: /douyin/aweme/v1/web/aweme/detail/ or /ttwid/ttwid/union/register/
    const pathMatch = url.pathname.match(/^\/(douyin|ttwid)(\/.*)/);
    if (!pathMatch) {
      return new Response("Invalid path. Use /douyin/* or /ttwid/*", { status: 400 });
    }

    const targetType = pathMatch[1];
    const targetPath = pathMatch[2];
    const targetHost = targetHosts[targetType];

    // Build target request URL
    const targetUrl = `https://${targetHost}${targetPath}${url.search}`;

    try {
      // Copy request headers
      const headers = new Headers(request.headers);
      headers.set("Host", targetHost);

      // Debug: print Cookie in request (should be commented out in production)
      // console.log('Request Cookie:', headers.get('Cookie') || 'No Cookie');

      // Remove CF related headers
      headers.delete("cf-connecting-ip");
      headers.delete("cf-ipcountry");
      headers.delete("cf-ray");
      headers.delete("cf-visitor");

      // Send request
      const response = await fetch(targetUrl, {
        method: request.method,
        headers: headers,
        body: request.body,
        redirect: 'follow'
      });

      // Debug: print response status (should be commented out in production)
      // console.log('Response status:', response.status);
      // console.log('Response Content-Type:', response.headers.get('Content-Type'));

      // Read response completely
      const responseText = await response.text();

      // Debug: print response length (should be commented out in production)
      // console.log('Response text length:', responseText.length);

      // Base64 encode to bypass CF automatic compression
      const base64Data = btoa(unescape(encodeURIComponent(responseText)));

      // Return Base64 encoded data
      return new Response(JSON.stringify({
        data: base64Data,
        encoding: 'base64'
      }), {
        status: response.status,
        headers: {
          'Content-Type': 'application/json; charset=utf-8',
          'Access-Control-Allow-Origin': '*',
        },
      });
    } catch (error) {
      return new Response(
        JSON.stringify({
          error: "Proxy request failed",
          message: error.message,
        }),
        {
          status: 500,
          headers: {
            "Content-Type": "application/json",
            "Access-Control-Allow-Origin": "*",
          },
        }
      );
    }
  },
};

2. Python Key Code (dysk.py snippet)

PYTHON

class DouyinDownloader:
    def __init__(self, enable_cf_proxy=False, cf_proxy_url=""):
        self.session = requests.Session()
        self.session.headers.update({
            "User-Agent": USERAGENT,
            "Referer": "https://www.douyin.com/",
        })
        self.ab = ABogus(USERAGENT)
        self.extractor = Extractor()
        self.enable_cf_proxy = enable_cf_proxy
        self.cf_proxy_url = cf_proxy_url.rstrip("/") if cf_proxy_url else ""

        print("Initializing (getting ttwid/msToken)...")
        self._init_tokens()

    def _init_tokens(self):
        base_str = string.digits + string.ascii_letters
        ms_token = "".join(random.choice(base_str) for _ in range(156))
        self.session.cookies.set("msToken", ms_token, domain=".douyin.com")
        data = {"region": "cn", "aid": 1768, "needFid": False, "service": "www.ixigua.com",
                "migrate_info": {"ticket": "", "source": "node"}, "cbUrlProtocol": "https", "union": True}

        # Use CF proxy or direct connection
        if self.enable_cf_proxy and self.cf_proxy_url:
            url = f"{self.cf_proxy_url}/ttwid/ttwid/union/register/"
            resp = self.session.post(url, json=data, timeout=30)
            if resp.status_code == 200:
                # Extract ttwid cookie from response and set to session
                if 'set-cookie' in resp.headers or 'Set-Cookie' in resp.headers:
                    cookie_header = resp.headers.get('set-cookie') or resp.headers.get('Set-Cookie')
                    # Debug: print Set-Cookie (should be commented out in production)
                    # print(f"Received Set-Cookie: {cookie_header[:100] if cookie_header else 'None'}")

                    if cookie_header and 'ttwid=' in cookie_header:
                        ttwid_match = re.search(r'ttwid=([^;]+)', cookie_header)
                        if ttwid_match:
                            ttwid_value = ttwid_match.group(1)
                            self.session.cookies.set("ttwid", ttwid_value, domain=".douyin.com")
                            # Debug: print the set cookie (should be commented out in production)
                            # print(f"Set ttwid cookie: {ttwid_value[:50]}...")
        else:
            url = "https://ttwid.bytedance.com/ttwid/union/register/"
            resp = self.session.post(url, json=data, timeout=30)

        if resp.status_code != 200:
            raise Exception(f"Initializing ttwid failed: HTTP {resp.status_code}")

    def get_detail(self, url_input):
        url = url_input.strip()
        aweme_id = self._resolve_short_url(url)

        if not aweme_id:
            return None

        print(f"Resolved ID: {aweme_id}")

        params = {
            "device_platform": "webapp",
            "aid": "6383",
            "channel": "channel_pc_web",
            "aweme_id": aweme_id,
            "update_version_code": "170400",
            "pc_client_type": "1",
            "version_code": "190500",
            "version_name": "19.5.0",
            "cookie_enabled": "true",
            "platform": "PC",
            "downlink": "10",
            "msToken": self.session.cookies.get("msToken")
        }

        params["a_bogus"] = self.ab.get_value(params)

        try:
            # Use CF proxy or direct connection
            if self.enable_cf_proxy and self.cf_proxy_url:
                api = f"{self.cf_proxy_url}/douyin/aweme/v1/web/aweme/detail/"
            else:
                api = "https://www.douyin.com/aweme/v1/web/aweme/detail/"

            self.session.headers.update({"User-Agent": USERAGENT})

            # If using CF proxy, manually add Cookie to request headers
            if self.enable_cf_proxy and self.cf_proxy_url:
                # Get all cookies and build Cookie header
                cookie_str = "; ".join([f"{k}={v}" for k, v in self.session.cookies.items()])
                # Debug: print the sent Cookie (should be commented out in production)
                # print(f"Sent Cookie: {cookie_str[:100]}...")

                headers_with_cookie = {"Cookie": cookie_str}
                resp = self.session.get(api, params=params, timeout=30, headers=headers_with_cookie)
            else:
                resp = self.session.get(api, params=params, timeout=30)

            # Debug: print request info (should be commented out in production)
            # print(f"API request: {api}")
            # print(f"Response status code: {resp.status_code}")
            # print(f"Response headers: {dict(resp.headers)}")

            if resp.status_code == 200:
                try:
                    resp_json = resp.json()

                    # If using CF proxy, the response will be Base64 encoded
                    if self.enable_cf_proxy and self.cf_proxy_url and isinstance(resp_json, dict) and 'encoding' in resp_json:
                        if resp_json.get('encoding') == 'base64':
                            import base64
                            decoded_text = base64.b64decode(resp_json['data']).decode('utf-8')
                            data = json.loads(decoded_text)
                            # Debug: print decoding success info (should be commented out in production)
                            # print(f"Base64 decoded successfully, JSON keys: {list(data.keys())}")
                        else:
                            data = resp_json
                    else:
                        data = resp_json

                    if data.get("aweme_detail"):
                        return self.extractor.extract_data(data["aweme_detail"])
                    else:
                        print(f"No aweme_detail retrieved")
                except Exception as e:
                    print(f"JSON parsing failed: {e}")
                    # Debug: print response content (should be commented out in production)
                    # print(f"Response content first 500 chars: {resp.text[:500]}")
            else:
                print(f"API request failed: {resp.status_code}")
                # Debug: print response content (should be commented out in production)
                # print(f"Response content: {resp.text[:500]}")
        except Exception as e:
            print(f"Request exception: {e}")
        return None

3. Plugin Configuration File (confschema.json)

JSON

{
  "enable_cf_proxy": {
    "description": "Whether to enable Cloudflare proxy",
    "type": "bool",
    "hint": "When enabled, API requests to Douyin will be proxied through CF Workers, which can effectively avoid IP being blocked by risk control",
    "default": false
  },
  "cf_proxy_url": {
    "description": "Cloudflare Workers proxy address",
    "type": "string",
    "hint": "Fill in the CF Workers address you deployed, e.g.: https://your-worker.workers.dev",
    "default": ""
  }
}

Technical Summary

1. CF Workers Automatic Compression Issue

Problem: CF Workers automatically compresses responses with gzip, even if the Content-Length header is set, it cannot be disabled.

Solution: Transmit data using Base64 encoding.

Principle:

CF's automatic compression targets text content
Base64 encoded data is wrapped in JSON
JSON format responses are also compressed, but the response body will not be lost

Problem: Cookie management of requests.Session() fails under proxy scenario.

Solution:

Manually extract Set-Cookie from response headers
Manually build the Cookie request header

Key Code:

PYTHON

# Extract Cookie
cookie_header = resp.headers.get('set-cookie')
ttwid_match = re.search(r'ttwid=([^;]+)', cookie_header)
self.session.cookies.set("ttwid", ttwid_match.group(1), domain=".douyin.com")

# Send Cookie
cookie_str = "; ".join([f"{k}={v}" for k, v in self.session.cookies.items()])
headers_with_cookie = {"Cookie": cookie_str}
resp = self.session.get(api, headers=headers_with_cookie)

Performance and Success Rate

Test Results

Scenario	Without CF Proxy	With CF Proxy
Video Parsing	60-70%	95%+
Image Parsing	0-10%	95%+
Live Photo Parsing	0-10%	95%+

Advantages

High success rate: Utilize CF's IP pool to avoid single IP being flagged
Traffic saving: Only proxy API requests; downloads connect directly to CDN
Free tier: CF Workers free plan provides 100,000 requests per day
Global acceleration: CF's edge nodes provide low-latency access

Precautions

CF Workers limits:
- Free plan: 100,000 requests/day
- CPU time limit: 10-50ms/request
- Response body size: unlimited (but recommended < 10MB)
Base64 encoding overhead:
- Data increases by about 33%
- Encoding/decoding has CPU overhead
- For small data (< 100KB) the impact is negligible
Cookie management:
- Need to manually handle Cookie extraction and sending
- Pay attention to Cookie domain and path settings

Deployment Guide

1. Deploy CF Workers

BASH

# 1. Log in to Cloudflare Dashboard
# 2. Go to Workers & Pages
# 3. Create a new Worker
# 4. Paste the cloudflare_worker.js code
# 5. Deploy and get the Worker URL

2. Configure the Plugin

In AstrBot WebUI:

Find the Douyin parsing plugin
Click "Configure"
Enable "Enable Cloudflare proxy"
Fill in "Cloudflare Workers proxy address"
Save and reload the plugin

3. Test

Send a Douyin link to the bot and observe log output:

Success: Should see parsed result
Failure: Check CF Workers logs and Python logs

Troubleshooting

Issue 1: Empty Response Body

Symptom: Response text length: 0

Steps:

Check Request Cookie in CF Workers logs
Confirm Cookie contains ttwid and msToken
Check if Python side correctly built the Cookie header

Issue 2: Base64 Decoding Failed

Symptom: JSON parsing failed: Expecting value

Steps:

Check response format is {"data": "...", "encoding": "base64"}
Confirm CF Workers correctly encoded the response
Check if response was truncated

Issue 3: CF Workers Timeout

Symptom: Proxy request failed: timeout

Cause: Douyin API slow response or CF Workers CPU time exceeded

Solution:

Increase timeout setting on Python side
Optimize CF Workers code (reduce unnecessary operations)
Consider using CF Workers paid plan

Conclusion

Through Cloudflare Workers reverse proxy, the Douyin API risk control problem was successfully solved. Key technical points:

Base64 encoding: Bypass CF automatic compression
Manual Cookie management: Ensure authentication information is correctly passed
Retry mechanism: Improve fault tolerance
Separation architecture: API goes through proxy, downloads go direct

This solution is not only applicable to Douyin but can also be extended to other platforms with risk control mechanisms (such as Xiaohongshu, Bilibili, etc.).

Full Record of Implementing Douyin Parser via Cloudflare Workers Reverse Proxy

Background

Problem Analysis

Initial Problem

Root Cause

Solution Evolution

Phase 1: Implementing Retry Mechanism

Phase 2: Introducing Cloudflare Workers Reverse Proxy

Phase 3: Pitfalls During Implementation

Pitfall 1: CF Workers Response Body Empty

Pitfall 2: Cookie Not Passed Correctly

Final Solution

Architecture Design

Complete Code

1. Cloudflare Workers Reverse Proxy Script

2. Python Key Code (dysk.py snippet)

3. Plugin Configuration File (confschema.json)

Technical Summary

1. CF Workers Automatic Compression Issue

2. Cookie Passing Issue

Performance and Success Rate

Test Results

Advantages

Precautions

Deployment Guide

1. Deploy CF Workers

2. Configure the Plugin

3. Test

Troubleshooting

Issue 1: Empty Response Body

Issue 2: Base64 Decoding Failed

Issue 3: CF Workers Timeout

Conclusion

References