Add resumable news history backfill

Let the crawler console set a backfill target per source. Each scheduled
run pages backwards from a checkpoint for at most 20s so real-time
crawling is not blocked. The checkpoint is saved only after a successful
run and is also kept in process memory, because the next trigger may be
queued before the database write lands. cls_depth is excluded since its
list API cannot page into history.
This commit is contained in:
eezd committed 2026-10-13 14:58:48 +08:00
1 parent 0831971e83
commit 5c8d773ce1
10 files changed
+229 -17

No files matched your search

+4
View File
@@ -21,6 +21,10 @@ class Source(Base):
last_success_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
last_error: Mapped[str | None] = mapped_column(Text)
paused_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# 历史回补:目标时间(为空表示不回补)、已回补到的最早时间(断点)、是否已结束(到达目标或来源无更早数据)
history_target: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
history_position: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
history_done: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
class NewsItem(Base):