Add resumable news history backfill
Let the crawler console set a backfill target per source. Each scheduled run pages backwards from a checkpoint for at most 20s so real-time crawling is not blocked. The checkpoint is saved only after a successful run and is also kept in process memory, because the next trigger may be queued before the database write lands. cls_depth is excluded since its list API cannot page into history.
This commit is contained in:
1 parent
0831971e83
commit
5c8d773ce1
10 files changed
+229
-17
No files matched your search
@@ -21,6 +21,10 @@ class Source(Base):
|
||||
last_success_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
last_error: Mapped[str | None] = mapped_column(Text)
|
||||
paused_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
# 历史回补:目标时间(为空表示不回补)、已回补到的最早时间(断点)、是否已结束(到达目标或来源无更早数据)
|
||||
history_target: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
history_position: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
history_done: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
|
||||
|
||||
|
||||
class NewsItem(Base):
|
||||
|
||||
Reference in new issue
Block a user