Add resumable news history backfill
Let the crawler console set a backfill target per source. Each scheduled run pages backwards from a checkpoint for at most 20s so real-time crawling is not blocked. The checkpoint is saved only after a successful run and is also kept in process memory, because the next trigger may be queued before the database write lands. cls_depth is excluded since its list API cannot page into history.
This commit is contained in:
1 parent
0831971e83
commit
5c8d773ce1
10 files changed
+229
-17
No files matched your search
@@ -11,6 +11,8 @@ LOG_CHANNEL = f"{KEY_PREFIX}pub:logs"
|
||||
NEWS_CHANNEL = f"{KEY_PREFIX}pub:news"
|
||||
CONTROL_CHANNEL = f"{KEY_PREFIX}ctl:crawler"
|
||||
STATUS_KEY = f"{KEY_PREFIX}crawler:status"
|
||||
# 列表接口支持按时间点往前翻的来源;cls_depth 列表只能翻约 3 页,无法回补历史
|
||||
HISTORY_SOURCES = ("cls_telegraph", "wscn_live", "wscn_news")
|
||||
|
||||
|
||||
def start_urls_key(source_code: str) -> str:
|
||||
|
||||
Reference in new issue
Block a user