Add resumable news history backfill

Let the crawler console set a backfill target per source. Each scheduled
run pages backwards from a checkpoint for at most 20s so real-time
crawling is not blocked. The checkpoint is saved only after a successful
run and is also kept in process memory, because the next trigger may be
queued before the database write lands. cls_depth is excluded since its
list API cannot page into history.
This commit is contained in:
eezd committed 2026-10-13 14:58:48 +08:00
1 parent 0831971e83
commit 5c8d773ce1
10 files changed
+229 -17

No files matched your search

+2
View File
@@ -11,6 +11,8 @@ LOG_CHANNEL = f"{KEY_PREFIX}pub:logs"
NEWS_CHANNEL = f"{KEY_PREFIX}pub:news"
CONTROL_CHANNEL = f"{KEY_PREFIX}ctl:crawler"
STATUS_KEY = f"{KEY_PREFIX}crawler:status"
# 列表接口支持按时间点往前翻的来源;cls_depth 列表只能翻约 3 页,无法回补历史
HISTORY_SOURCES = ("cls_telegraph", "wscn_live", "wscn_news")
def start_urls_key(source_code: str) -> str: