Prioritize realtime news and make historical backfill resumable
Development CI / check-and-build (push) Successful in 3m0s
GitHub Git mirror / mirror (push) Successful in 5s
Development CI / deploy-development (push) Successful in 14m7s

This commit is contained in:
eezd committed 2026-11-06 14:28:39 +08:00
1 parent c90691d9a0
commit 5f347f8ab4
17 files changed
+697 -143

No files matched your search

+39
View File
@@ -455,3 +455,42 @@ some documents were imported: ECB dynamic index version coverage, inaccessible
BoE speeches and CoinDesk access blocks are reported rather than labeled complete.
The deployment lock serializes writes; crawler/calendar processes are restored in
finally blocks. Do not start a second driver or remove its state to renew quotas.
### Realtime priority and historical AI backfill
Migration `dd4567890123` adds the persistent realtime boundary and pending-event
priority, plus partial unique indexes for active package attempts and event
versions. Pause and drain news AI before applying it; duplicate active records
must be investigated rather than deleted to force migration success. Preserve
originals, attempts and version history. Deployment requires the candidate's
schema head and must leave automatic processing paused during verification.
The model/Skills drawer saves the historical start time with model settings via
**Save and pause**. New settings default to 2026-11-05 00:00 Asia/Shanghai; existing
saved values remain intact. Settings responses expose `control_revision`,
`realtime_from`, `timezone` and queue counts. PUT accepts optional `start_at` and
`base_revision`; version conflicts return 409. Enable uses the saved start only;
a differing legacy enable body returns 422. The first successful enable fixes
`realtime_from`; later saves, pauses and restarts never move it.
Publication time, not ingestion time, determines realtime membership. Realtime
work takes the next available slot without cancelling accepted calls. Historical
work uses batches of at most 30 originals, ordered chronologically. Saving a new
start pauses dispatch; enabling after drain retires unfinished batch schedules
and rebuilds from durable stage states without repeating successes. Shrinking
history does not exclude realtime backlog. Known in-range dependencies inherit
a waiting realtime event's priority. Out-of-range or failed members block event
research without silently dropping evidence or retrying paid failures.
Ordinary event research uses the existing five-minute quiet window with a
15-minute maximum; urgent events become ready immediately. A running research
call is not duplicated when newer members arrive. New changes start another
aggregation window and obsolete output cannot overwrite current conclusions.
After a worker interruption, started calls become failed with an unknown-result
message and need manual retry; unclaimed work resumes normally. Strict realtime
priority can starve historical work under sustained overload.
CI additionally runs `scripts/smoke_priority_queue.py` in its disposable internal
network with synthetic credentials and no model calls. It covers preemption,
late historical ingestion, stable realtime eligibility, range changes,
dependencies, aggregation, control revisions and interrupted-call recovery.