Rnote API Rnote API
Back to Blog

Batch Crawls That Don't Die Halfway: Error Classes, Backoff and Resumability

Rnote API Team · · 4 views · 中文
Xiaohongshu Data Batch Collection Error Handling Retries

Getting one call working and getting ten thousand to finish are different problems. A crawl that dies at item 3,000 costs you a full re-run — and if you re-charge for rows you already had, you pay twice for the same data.

This post covers three things for batch work: how to classify errors, how to retry, and how to make a job resumable.

Sort errors into three buckets

Not every failure should be retried. Misclassify them and you waste quota at best, spin forever at worst.

Bucket Typical status What to do
Retryable 429, 5xx, connection timeout Exponential backoff, then retry
Not retryable 400, 404 Record and skip — ten thousand retries change nothing
Stop the job 401, 402 Halt immediately and alert

The third bucket is the one people get wrong. Plenty of jobs treat 402 (insufficient balance) as an ordinary failure and retry it — the balance does not grow because you retried. You end up with tens of thousands of useless requests burying the actual problem in the logs.

Back off, and cap the attempts

import time, requests

RETRYABLE = {429, 500, 502, 503, 504}
FATAL = {401, 402}

def fetch(url, params, max_tries=5):
    headers = {"X-API-Key": "YOUR_API_KEY"}
    delay = 1.0
    for attempt in range(max_tries):
        try:
            r = requests.get(url, headers=headers, params=params, timeout=30)
        except requests.RequestException:
            time.sleep(delay); delay *= 2; continue
        if r.status_code in FATAL:
            raise SystemExit(f"halting: {r.status_code} {r.text[:200]}")
        if r.status_code in RETRYABLE:
            time.sleep(delay); delay *= 2; continue
        return r          # 2xx, or a 4xx the caller should judge
    return None           # retries exhausted -> record as failed

Three details: backoff starts at one second and doubles (1→2→4→8→16); FATAL raises rather than returns; exhausted retries return None instead of raising, so the main loop can log the item and keep going.

Make the job resumable

Batch jobs will be interrupted — network, deploy, an accidental Ctrl+C. The fix is simple: persist "done" to disk, not to memory.

import json, pathlib

done = set()
state = pathlib.Path("done.txt")
if state.exists():
    done = set(state.read_text().split())

with state.open("a") as f:
    for note_id in all_ids:
        if note_id in done:          # already fetched — skip, don't pay again
            continue
        r = fetch(url, {"note_id": note_id})
        if r is None or r.status_code >= 400:
            continue                 # leave failures for round two
        save(r.json())
        f.write(note_id + "\n"); f.flush()   # the important bit

That flush() is the line people drop, and then a killed process loses the few hundred entries sitting in the buffer — failing at exactly the moment it was needed.

Two passes: finish first, patch second

Run everything in pass one, recording failures without retrying them. Run only the failure list in pass two. Pass one never gets stuck behind a handful of stubborn items, and pass two tends to have a high success rate because the rate limit window has moved on and upstream has recovered.

Whatever still fails after two passes is usually genuinely unavailable (note deleted, account gone). Move it to a "confirmed unavailable" list instead of leaving it in the queue forever.

Get started

Failed requests aren't billed, so sane retries cost nothing extra — but they still consume rate limit. Give batch jobs their own API key with lower concurrency. Endpoints are in the docs; usage is reconcilable by date and endpoint on the billing page.

Back to Blog