eezd/AlphaPulse
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(news): index news packages and research persistent events

Browse Source 
Extract reusable literal keywords once per source revision, assign complete news packages to long-lived events, and research all event members with immutable evidence and references.

Add additive research storage, vocabulary snapshots, the two-stage workspace, isolated acceptance coverage, and a paused rollout with an initial fixed 50-news production trial.
This commit is contained in:
eezd committed 2026-11-04 12:09:58 +08:00
1 parent 07252fea0f
commit 4929a32a0a
69 files changed
+8215 -2092
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

deploy/README.md
+61 -95
View File
@@ -79,31 +79,74 @@ Git mirroring is independent of build results. Push/create/delete events, hourly
Only resources labeled for the current CI attempt are cleaned; there is no global Docker prune. The existing runner registration, capacity and other projects' mount allowances are preserved.
## Three-stage news research
## Two-stage news research
The `news-risk` service runs three model stages with separate durable queues for grouping, package review, event timelines, reading and explicit reprocessing. Original news and revisions remain in their existing tables; AI packages, package versions, memberships and research evidence are stored separately.
A news item has one stable package, identified by `news_items.id`. Reposts remain separate packages; a digest is never split into new packages. All original news, revisions and legacy AI tables are retained.
1. **Grouping:** the selected Luna model processes every domain and content type, including political news, statements, data, explanations and opinions. Reposts of the same concrete fact share a package with original links and normalized tags. New developments and opinions form separate packages; topic similarity alone is insufficient. One article can contribute to multiple packages.
2. **Package review:** a separately selected model receives the package fact, title, category, tags and content type alongside its original articles, so separate facts within one roundup retain distinct review scopes. It records facts, timeline, claims, verification verdicts, counter-evidence, related packages, opinions and watchpoints. Reading importance (1–5) is independent of risk severity (0–5), so major positive developments can enter the important reading list. Each attempt permits four local searches, eight local reads, two external searches and four external reads, across at most eight tool rounds followed by submission. Context planning reserves the research tool definitions before extracting or compacting long articles. News/package/calendar search and SearXNG are read-only. Search snippets and other AI summaries are leads, not independent verification. Quotes must occur in a source actually supplied or read; insufficient evidence must remain explicit. Tool responses, failed attempts and previously successful versions are retained. A manual retry can reuse saved successful tool responses; old failures cannot requeue a revision that already succeeded.
3. **Reading and trends:** the independently selected advanced model ranks every successful event or independent package of importance 3–5 within the last 72 hours. Large inputs are processed in batches; truncated ranking outputs split into smaller batches, while a truncated single-package output fails explicitly. The complete reading list is retained and paginated, with strict package/version coverage checks. Up to three lower-importance packages from different domains provide expansion reading. Synthesis distinguishes facts from inference and gives market implications, 24–72-hour and 1–4-week trends, conditions, counter-evidence and watchpoints. Ordinary changes aggregate for 15 minutes; importance-5 developments can trigger earlier, with at least 60 seconds between starts. Citation-only changes do not repeat advanced-model calls; changed facts, ratings or verification verdicts still trigger updates. Reposts and analysis execution alone do not advance package source-progress timestamps. Window expiry also triggers an update. Failed refreshes retain the previous successful briefing.
1. **Keyword indexing and event assignment:** read the complete news once to persist canonical TAGs, literal aliases and original supporting quotes. This stage generates no summaries or ratings. Assignment is a separate persisted subtask, so a classification retry reuses the keyword index. Packages and events have many-to-many membership; a digest can support several distinct events, and a single package may start an event. Event candidates cover all time, bounded to sixty lexical matches. Shared names or sectors alone do not establish a common occurrence. Source-content changes explicitly invalidate the index; oversized single-news input fails without truncation.
2. **Event research:** read all current member packages and originals together, producing one event summary, claims, importance (1–5), independent risk (0–5), verification, factors, uncertainty, opinions, impacts, counter-evidence, watchpoints and timeline. Every successful result and failed attempt retains input, output, rules, usage and tool evidence. Long event inputs use evidence-preserving extraction/compaction across all originals or fail explicitly. Timeline nodes identify the corresponding member packages and their original news IDs; background nodes supported only by supplementary material have an empty package list. Literal quotes and source identities are validated before publication.
An event is optional and requires at least two distinct packages describing the same developing incident or process. A standalone package remains readable and can join an event later. Review must actually read the other packages before proposing membership. Each package has at most one main event; cross-event links remain separate. Event timelines preserve known, partial and unknown occurrence times, reporting times, verification status, original quotes and exact package versions. Unknown occurrence times are never replaced by publication times. A later event refresh cannot publish over changed member versions. Successful events aggregate their represented packages into one reading entry; typed `(kind, id, revision)` references distinguish events from packages.
An event has its own continuing timeline. Research may refer to another event's successful immutable version as background, supporting material or subject. A changed referenced event is marked for the reader and does not automatically start another paid call. AI summaries are clues, never primary evidence. Tools allow four local searches, eight local reads, two external searches and four external reads per attempt; supplied articles and cached reads are reused. Tool failures remain visible as research limitations.
Package review can request atomic split/merge corrections. Corrections lock and verify input revisions, require all current source assignments and literal evidence, retain old IDs/history, and preserve the earliest ID on merges. Repeating a correction with the same source hashes and Skill rules is rejected to prevent loops. Corrections are versioned separately; superseded package URLs link to replacements. A pure repost only adds a source; new facts, independent evidence and corrections request another review. Original news rows are never rewritten by AI.
Ordinary new facts, independent evidence and corrections aggregate for five minutes, with a fifteen-minute hard deadline. Urgent developments are immediately eligible. Pure reposts retain their own packages and citations without requesting a new successful event review. Indexing, assignment and research share the configured concurrency pool (1–16, default 3). Pausing stops new admissions; in-flight work drains. Interrupted tasks recover from PostgreSQL. Third-stage reading/trends, legacy queue/backfill jobs and old three-model configuration are not exposed or scheduled by this workspace.
The homepage offers a publication-range reprocessing preview followed by explicit cost confirmation. Starting captures the latest source hashes, three model/effort configurations and published Skill contents; it does not reconstruct a historical information state. Jobs yield to pending realtime work, persist position and failure reasons, and resume after restart. Cancellation stops subsequent articles; an in-flight call and already queued package/event reviews may finish. Pause stops new work. Completion waits for package/event review and a reading snapshot; failures retain prior successful outputs. Failures identify their stage and object. Retrying grouping failures reruns only those news items; package review, event timeline and reading failures resume their own stages without repeating successful grouping. Retries can incur additional model costs. Pausing before a grouping call leaves the job checkpoint unchanged, even when an older successful analysis exists. Source revisions invalidate stale job items; select a new range to include the new revision. Shared packages/events may update even when some of their articles fall outside the selected range.
The homepage provides event and package searches, pagination, membership details, timelines, evidence and version history. `GET /api/risk/settings` and `PUT /api/risk/settings` manage independent `index` and `research` model/effort/Skill bindings and concurrency; saving pauses admission. Clearing effort inherits provider settings. `POST /api/risk/enable` validates the two models with synthetic material, including separate index/assignment contracts. `POST /api/risk/packages` enrolls up to 500 explicit news IDs; while enabled the worker additionally admits news published since November 1, 2026, 00:00 Asia/Shanghai. Failed assignment retries reuse indexing; event retries never repeat indexing. Earlier source revisions are not shown with current revision annotations.
The first successful enable operation records a permanent start timestamp. Only source publications or source updates after that timestamp enter the pipeline; ingestion of older history does not trigger analysis. Pause/resume retains the timestamp. Historical records remain available to research tools. No trading actions are triggered.
Create the two default Skill drafts in homepage settings, review and publish them in Skills, then bind them before enabling. Template generation never publishes or changes existing Chat/trading Skills. The additive `a71234567890` migration follows `f60123456789` and creates seven `news_research_*` tables; `b81234567890` adds the eighth table for immutable vocabulary snapshots. There is no conversion, truncation or deletion of original/legacy tables. Production schema execution, code deployment, commits and pushes require the corresponding task authorization; preparing a migration alone does not grant it.
During package review, long-article slices with no facts relevant to the package return empty facts and evidence together and are omitted from compaction. Relevant slices must retain literal source quotes; if every slice is irrelevant, review fails explicitly. Grouping extraction, compaction and final package summaries still require nonempty evidence.
### First rollout of two-stage research
Create or update the three default news Skill drafts in homepage settings, review and publish them in the Skills page's News tab, then bind the grouping, package-analysis and reading Skills. Updating appends a new draft only when the latest content differs; it preserves node IDs, previous versions and published versions. Templates never publish themselves. `POST /api/risk/skill-templates` without a body, or with `refresh_existing=false`, only creates missing templates; the homepage explicitly sends `refresh_existing=true` to prepare updated drafts. News-scoped Skills never enter Chat/trading prompts. Publishing a Skill does not enqueue a historical reanalysis.
The first rollout deploys the new workspace and binds the two published Skills while automatic news AI remains paused. The user requested a first production trial of exactly 50 frozen news items after deployment. Run only their indexing, assignment and resulting event research through the same runtime services, then leave automatic admission paused. Fifty news items require more than fifty model requests because assignment and event research are separate tasks, with bounded tool rounds and retries.
All three stages independently select a model and reasoning effort (`low`, `medium`, `high`, `xhigh`, `max`). Model selections support provider model lists and manual IDs. Clearing a stage's effort inherits its provider setting; if both are unset, no effort parameter is sent. Overrides do not change the shared provider. Each attempt retains the selected override, effective effort, rules and model identity; changing either the override or effective settings changes its configuration fingerprint. Test-and-enable makes real validation calls and reports unsupported settings without silently lowering effort; saving paused settings makes no model calls.
Do not use the normal enable endpoint for this trial: it makes real synthetic model calls and admits news published since November 1, 2026, 00:00 Asia/Shanghai. A read-only check on November 4 found 3,388 eligible news items; this count will grow. Explicitly enrolling 50 IDs does not limit automatic admission after enabling. The acceptance allowance of 180 requests belongs to the earlier isolated acceptance runner and does not impose a production spending limit.
The additive `c3d5e7f90123` migration follows `b2c4d6e8f012` and adds three nullable stage-effort columns. Existing settings inherit provider effort. Apply it only with separate migration authorization through the normal deployment lock; it does not update live Skill contents, publish drafts, enable AI or reprocess history.
The following steps are prepared operator instructions, not an instruction to run production changes without authorization:
The subsequent `d4e6f8012345` migration adds events, membership and cross-event links, grouping versions, correction records, reprocessing jobs and active article assignments. It seeds grouping history from existing analyses and pauses AI while leaving raw news and prior successful summaries intact. Both pending migrations require separate authorization under the deployment lock. Update and review the default Skill drafts before enabling the new output contracts; no historical job starts automatically. CI exercises migration and pipeline behavior using synthetic models and a disposable database, including cancelled/failed reprocessing, reposts and correction history.
1. Review the complete working tree, including new files, and obtain task authorization for commit/push, the independent schema migration, deployment, and publication/binding of the two news Skills. A `dev` push publishes accepted images and automatically attempts deployment. Review the proposed scope before pushing; GitHub receives a public mirror. Keep all original news and legacy AI data.
2. After the authorized commit/push, wait for the isolated check/build job and download its `build-receipt` artifact. Record the full commit, version and accepted image identities. The first automatic deploy is expected to stop at `Database migration requires separate authorization` while the database is at `f60123456789`; this check runs before stopping or replacing services. Do not disable the guard or bypass the deployment lock. If a different failure occurs, diagnose it first.
3. Under `/docker/gitea/dev/AlphaPulse/.deploy.lock`, confirm no pending deployment, the approved commit is still current `dev`, legacy news AI is paused, the database is at `f60123456789`, and the candidate has the single head `b81234567890`. Validate both image identities using `scripts/deploy.py`'s `validate_images` and the downloaded receipt. Preserve the current deployment receipt, Compose and image selection. Create a private full database backup with the owning PostgreSQL container, record its checksum, and verify restoration in a separate empty database before migrating. Preserve the existing encryption key through its private configuration mechanism; never put it in reports or Git. A file existing or `pg_restore --list` succeeding alone does not establish recoverability.
4. Still holding the same lock, use the verified candidate backend image with the existing instance `.env` and `dev_default` network to run `alembic upgrade b81234567890`. Only these two reviewed migrations may run. Check the resulting revision and eight new tables; verify original-news samples and legacy settings remain intact. The migration updates `alembic_version` as well as creating the new tables. If it fails, inspect the revision and tables before retrying; do not stamp a revision or drop tables to bypass a failure.
5. Release the migration lock, then run the existing update command with that same accepted receipt:
```sh
python3 scripts/deploy.py update --build /absolute/path/to/accepted/build.json
```
The updater takes the lock itself and restarts the existing nine services using their existing configuration. Verify the served frontend/API commit, all service states, restart counts and search health. A newer `dev` commit causes the update to skip; obtain its accepted receipt and reassess migrations before continuing.
6. In homepage settings, create the two default drafts through `POST /api/risk/skill-templates`. Review and publish **新闻关键词索引与事件归属** and **新闻事件整体研究** in Skills. Record their returned node IDs, published version IDs and content hashes, and verify their scope is `news` and they are enabled. The first content must equal `INDEX_SKILL + "\n" + ASSIGN_SKILL`; the second must equal `RESEARCH_SKILL` in the deployed candidate. Do not copy the sample-specific official-URL hints from isolated acceptance Skills into these production defaults. Preserve the legacy Skills and all Chat/trading bindings.
7. Bind the returned Skill IDs to `index` and `research` with the reviewed provider/model/effort values and concurrency 3. Save settings, then read them back and confirm `enabled=false`. Published content is used together with the runtime base instructions; production has no fallback to an unbound default template. Configuration/publication does not call the models. Do not click test-and-enable, call `/api/risk/enable`, or import isolated acceptance outputs during this rollout.
8. Verify homepage event/package lists, news details, vocabulary reads and authentication. Empty new event/package lists are expected before the fixed trial; originals and legacy history remain stored. Record the migration, served commit, Skill versions/bindings and paused state in the private rollout receipt.
9. For the fixed trial, freeze exactly 50 distinct original news IDs and their content hashes before processing. In the absence of another selection, use the latest 50 and order them by publication time and ID for assignment. Keep the instance deployment lock throughout execution, stop only the existing `news-risk` service through the deployment controller's Compose command, and acquire the worker's PostgreSQL advisory lock `(20261030, 1)` before dispatch. Keep the research-settings row locked and `enabled=false`, and invalidate concurrent enable validation by advancing its control revision. Reject an existing research package outside the frozen selection; do not clear it. Validate the published model/Skill bindings without calling the normal enable endpoint. Index with concurrency 3, assign chronologically and serially, then research the resulting events with concurrency 3; each event reads all its member packages. Reuse completed stages instead of repeating successful TAG extraction. Abort if a frozen original changes. Preserve JSON checkpoints, attempts, versions, evidence, usage and failures; automated completion is not semantic quality approval. In a `finally` block, retain the paused setting, release the advisory lock and restore the previously running news worker under the deployment lock. Check the stored package IDs equal the frozen 50 and the automatic setting is still paused, then review TAGs, memberships, event conclusions, evidence and model usage. General production enablement remains a separate decision; neither this bounded trial nor historical samples prove long-running production quality.
For a failure after replacement starts, `scripts/deploy.py` attempts restoration of the previous images/configuration. Use `python3 scripts/deploy.py recover` only when a pending deployment remains; use `python3 scripts/deploy.py rollback` for an explicitly authorized rollback after a successful deployment. Image rollback retains the eight new tables and their data and does not downgrade or stamp Alembic. Before rolling back an enabled system, pause news AI and check in-flight work; pause stops admission rather than cancelling requests already sent. Keep legacy news AI disabled so the old worker does not resume the three-stage flow. If new Skills have already been published, retain their versions for a later forward deployment. Do not restore the database backup over ongoing news/trading writes as a routine image rollback. The older image's Alembic CLI cannot resolve the new revision; run any subsequent schema inspection or migration using the candidate that owns it.
Uncommitted preflight images are local test artifacts, not deployable CI receipts. Build them under distinct local tags and run isolated smoke tests against those exact images. `scripts/ci.py check` identifies images by committed `HEAD` and may reuse its published images; running it on a dirty working tree cannot by itself establish that the uncommitted runtime was exercised. After committing, the normal clean-checkout CI must build and accept the committed version before rollout.
### Event boundaries, vocabulary and pinned references
An event follows one occurrence, its direct consequences, responses, investigation, corrections and remediation across months or years. A new occurrence, independent decision/action or statistical period starts another event. A market reaction explicitly attributed to a particular release belongs to that release. Shared subjects or TAGs establish candidate relevance, not membership.
Candidate recall also matches whole ASCII words from literal compound aliases against event TAGs and titles, so an existing `bybit` TAG can be found from `Bybit Hack` without regenerating historical keywords. Full-phrase matches rank first, retired events are excluded and the sixty-candidate cap remains. The model decides membership. New keyword extraction keeps independently searchable subject names alongside incident phrases. Research quotes must retain enough continuous original context to support the claim, including original line breaks around names; literal substring validation alone does not establish semantic support.
`GET/PUT /api/risk/tag-vocabulary` reads/saves the reviewed phrase vocabulary. PUT accepts `{base_revision, entries: [{canonical, aliases}]}`; stale revisions return 409, ambiguous or invalid entries return 422. Each save creates an immutable snapshot in the additive `news_research_vocabulary_versions` table (migration `b81234567890`, following `a71234567890`). The homepage provides manual editing. Vocabulary changes expand historical searches without rewriting prior model output or triggering paid reprocessing. Attempts and event research freeze their vocabulary snapshot; literal news aliases and quotes remain unchanged.
`GET /api/risk/events/{event_id}/versions/{version_id}` reads a successful pinned version, including frozen member articles and references; a mismatched event or unsuccessful version returns 404. Reference links open that snapshot. Current target changes are marked but do not enqueue referring events.
`scripts/accept_news_research.py` runs only against an empty local disposable database with CI credentials and an explicitly supplied private model configuration. Use the frozen 30-news report or `backend/tests/fixtures/news_research_scenarios.json` (`--synthetic`). Separate databases share a `--budget` counter file, which atomically limits all provider requests, including retries and repair/tool/extraction rounds, to 240. Never reset the counter for the same acceptance run. The script reports automated assertions separately from human quality review and never automatically grants quality acceptance. Synthetic external searches are controlled; real-source availability limitations remain visible. Reports retain exact inputs, outputs, rules, vocabulary, evidence and usage; private provider configuration must be removed after the run.
### Chronological scenario acceptance
`accept_news_research.py --scenario-manifest MANIFEST --credentials PRIVATE --output RUN_DIRECTORY --budget NEW_COUNTER` freezes twelve real originals and chronological batches. The manifest specifies explicit same-event, separation and pinned-reference expectations, official URLs with text anchors, and the immutable historical counter hash (199). The independent new counter permits 90 requests initially; `--resume --finish` permits the reserved continuation up to 120 additional requests total. All retries, repairs and tool rounds share this counter; never reset it. Reports and attempts are retained separately, while `state.json` holds resumable progress.
After an explicitly authorized budget extension, `--resume --finish --request-limit LIMIT` sets the cumulative supplemental cap for that continuation and records it in the new state/report. It never changes the frozen manifest or resets the counter. Omission retains the original 90/120 defaults; the option is rejected without both resume flags or below 120. A counter already above the selected cap fails explicitly. Completed-checkpoint restoration retains the extended cap from its report; paid continuation still needs the explicit authorized limit. Authorization remains an operator responsibility.
Only the disposable local CI database is writable. Published isolated Skills use the same configuration path as runtime; both the base instructions and published content must reach the model. Original news enters the test database one batch at a time. External tools are disabled during historical replay; final live verification publishes an isolated Skill version with explicitly supplied official URL hints, which are leads rather than evidence. These explicit reruns do not count as natural revisions. Known public URLs may be read directly through the same DNS, redirect, size and tool-budget checks. Tool snapshots distinguish `discovered` and `direct` reads. Search availability and successful official text reads are assessed separately.
The manifest, configuration, vocabulary and existing originals must remain frozen for resume. A diagnosed missing incident reference can use `--resume --finish --repair-references` before the final natural-update batch: retain the old version, publish a repaired isolated Skill, and retry the affected consumer without repeating successful TAG extraction. This explicit repair is retained in the audit and is not evidence of autonomous recovery. Automated checks never grant human quality acceptance; limited acceptance requires review of facts, timelines, rating rationale, original quotes and immutable references. No result establishes long-running production quality, autonomous URL discovery, or savings. Live migration, deployment and real Skill publication remain separate actions.
Official-evidence checks require a successful version of the intended event, a successful read from its expected official domain with the frozen fact anchors, and a supported/partial claim citing that read's literal text. A read retained only in a failed version cannot pass. Each event receives only the official leads associated with its member news; unrelated official pages are not required reading. Reference checks require a later natural target update without another consumer attempt and call the application reference reader to verify that the pinned version remains unchanged while `changed=true`.
Resume requires the disposable database as well as report files. If that database has been removed, `scripts/restore_news_scenario.py --report ORIGINAL_REPORT --snapshot COMPLETED_BATCH_SNAPSHOT --credentials PRIVATE --output NEW_RUN_DIRECTORY --budget ORIGINAL_NEW_COUNTER` restores a completed chronological prefix into an empty local `ci` database with synthetic CI credentials. Apply migrations to that isolated database first. The restore checks historical versions, originals, vocabulary and checkpoint consistency, preserves successful keyword attempts, and carries forward the original counter without resetting or spending requests. Historical rules stay frozen; subsequent work uses newly frozen isolated Skills and requires separate real-model validation. Old reports are never overwritten. Offline mock replay demonstrates code behavior only and cannot grant quality acceptance.
### Search and subscriptions
@@ -111,17 +154,13 @@ Compose includes a version-tagged SearXNG service on the existing internal Docke
The crawler console accepts administrator-managed public RSS/Atom subscriptions, names, intervals and enable/disable controls. New subscriptions are paused, no default feeds are added, and the manager discovers new subscriptions without a restart. Only feed-provided content is stored; a summary is labeled as a summary. Arbitrary private-network feed URLs are rejected.
### Authorized transition only
### Isolated acceptance and original-data protection
The additive `b2c4d6e8f012` migration requires separate authorization and pauses the existing news AI setting. It adds package data, evidence, Skill scope, three-stage configuration and the source adapter; it does not delete old results. For the initial transition, stop the old worker under the instance `.deploy.lock` before applying the authorized migration. Using the accepted new-image receipt, preview legacy cleanup with:
Use `scripts/smoke_news_research.py` only with disposable CI PostgreSQL credentials (`ci` / `ci-only`, database `ci`). It mocks model I/O and verifies one-news identity, multiple event memberships, indexing reuse, strict quote rejection and durable failed outputs. CI runs this after migration; all collectors and trading remain stopped.
```sh
python3 scripts/deploy.py reset-news-ai --build /path/to/accepted-build.json
```
Real-news acceptance requires explicit permission for the bounded sample and necessary model access. Export original records in a read-only repeatable-read transaction, copy only necessary provider fields to a private isolated configuration, and never copy the production `.env`. After processing, review keyword coverage, event boundaries, reposts, evidence, dates and ratings, retain a credential-free report, and remove temporary keys and test containers. The initial November 3 round used thirty news items and additive tables only; the subsequent chronological supplement uses twelve frozen originals and a separate additional request budget. Legacy sample replay/reset commands target a different schema and are not part of this workflow; do not run them for this transition. Legacy reports and task counts do not prove current quality or savings. Missing usage is not zero cost.
After explicit maintenance authorization, the same command with `--confirm-delete-old-news-ai` stops the news worker and clears only `risk_analyses`, `risk_events` and `risk_snapshots` in one transaction. It refuses if the new pipeline already contains data. It preserves provider configuration, source news, revisions, calendar, Skills, Chat and trading. It uses the deployment lock and does not start the worker or enable models. Deploy through the normal `scripts/deploy.py update` path, then configure, publish and enable from the UI. Do not roll back to pre-scope code with enabled news Skills: pause news AI and disable news-scoped nodes first, since older code does not understand that boundary. Downgrading the additive migration drops the new AI tables and requires an independent data-retention decision.
The homepage displays the latest successful reading briefing and its refresh status, plus searchable package archives with time, importance, risk, tag and category filters. Details include source links, claims, tool evidence and version history. Failed tasks can be explicitly retried by stage. Local acceptance uses synthetic models/search and disposable databases; real model quality, upstream search availability and live deployment need separate operational validation.
News transport uses at most two retries after 30/60 seconds, honoring valid Retry-After up to 120 seconds while retaining the worker slot. Partial output and permanent failures are not replayed. Structural submission repair is bounded to one correction using already available evidence. Default Skill changes are not automatically published to live nodes. External search must use a working test endpoint to establish external-verification coverage; unavailable search is explicitly reported.
## Global economic calendar
@@ -136,76 +175,3 @@ Each complete daily response withdraws missing records only within its own day a
Official and Jin10 observations retain separate identities and revisions. The read API merges only verified indicator IDs, definitions, units, reference periods, release stages and exact publication times. Official nonmissing fields win independently; Jin10 fills missing fields and market forecasts with explicit provenance. Unverified definitions remain distinct records. Original and revised previous values remain separate. Forecast observations captured after publication are displayed with that limitation and excluded from historical surprise judgments. Jin10's gold/USD direction flags are not BTC/ETH signals. Tentative times and all-day holidays never display an invented exact midnight time.
CI acceptance runs `scripts/smoke_calendar.py` against the isolated database using public fixtures and synthetic account material. It never starts a browser collector or connects a real Jin10 account. Local unit tests likewise isolate all external I/O. The live database migration, credential import/reconnection, history collection and dev deployment are separate operator actions; ordinary local validation does not authorize them.
### News AI concurrency
All background news AI stages share `risk_settings.max_concurrency` (default 5).
A slot covers one complete stage task, including its sequential model and verification
calls. Completed tasks release their slots immediately; runnable real-time stages rotate
before historical items use spare capacity. Pending work remains in PostgreSQL.
The home settings dialog saves this operational limit independently of model settings.
Increasing it admits more work; decreasing it drains existing tasks without cancelling
paid requests. Pausing stops new admissions. The overview reports active slots and stage
counts; an expired worker heartbeat shows an unknown active count.
`PATCH /api/risk/settings/concurrency` accepts a positive integer `max_concurrency`.
The limit is excluded from frozen reprocessing configurations and model fingerprints.
“Test and enable” requires paused analysis and an idle pipeline; an exclusive PostgreSQL
advisory gate prevents concurrent validation and background work. Historical items persist
individual states, so completion order does not change progress or skip unfinished items.
Migration `e5f701234567` adds the limit and converts legacy historical checkpoints to
item states. Deployment and execution of this migration require separate authorization.
### Historical news AI backfill and queue
The home page's **AI 历史回补** selects publication times in `[start, end)` using
stored current news revisions. `missing` completes unfinished grouping/package analysis
and reuses published successes; `review` explicitly repeats grouping/TAG and package
verification, preserving previous successful versions. Selecting both includes all
news in the range. Related event timelines are refreshed; a batch waits only for its
own package/event targets, while the home reading window stays at 72 hours.
Corrections register every resulting package in the same transaction, including new
branches when a split retains the original ID. An empty reading window saves a cleared
snapshot without a model call or concurrency slot.
`POST /api/risk/reprocess/preview` accepts `start`, `end`, and `modes`. Its opaque
`preview_id` expires after 15 minutes and fixes news IDs, content hashes, models,
reasoning efforts, and published Skills. `POST /api/risk/reprocess` accepts that ID;
changed configuration requires a new preview, and changed source content fails its
individual item explicitly. A missing item completed before execution is skipped.
Only one batch runs at a time. Batches use spare slots in the shared concurrency
pool, respect global AI pause, support cancellation of further news admissions and
explicit retry of failed items, and reclaim interrupted news items after restart.
Already scheduled downstream package/event updates continue after cancellation.
An in-flight news item also keeps its identity reserved after cancellation, so a new
batch waits for that item to finish while unrelated news can still use spare slots.
`GET /api/risk/reprocess/{id}` returns paginated per-news states and downstream
stage counts. `GET /api/risk/queue` filters by `state`, `stage`, `origin`, `job_id`
and page. The home queue refreshes every three seconds while visible. Actual model
waits, research rounds, tool calls and save steps are transient telemetry under
`ap:risk:active`; PostgreSQL remains the durable queue. Heartbeat expiry means unknown
execution state. Empty outlook polls and batch completion checks do not consume AI
slots or appear as running model tasks.
Batch filters include durable package/event targets even after real-time updates
change their execution configuration; displayed model/source metadata describes the
current execution. Open news details refresh with the batch list, retaining their page
and rejecting responses from an older selection or session.
Migration `f60123456789` adds normalized batch news items and package/event targets,
retaining legacy job JSON and execution semantics. Isolated acceptance includes
`scripts/smoke_backfill.py`; it mocks all model/Redis I/O and checks mode behavior,
frozen previews, source changes, target completion and queue queries. Live migration,
deployment and real backfill execution require separate authorization.
### Complete news units and exclusive rechecks
Grouping retains each coherent publication as a complete news unit: related metrics and background remain inside the package. Independent developments remain separate. Local research accepts up to six keywords and ranks partial matches; search outages are distinct from successful empty searches. Planned dates and statistical periods remain separate from actual occurrence timestamps.
Reading ranks all important inputs, shortlists at most twelve low-star candidates across domains, selects up to three expansion entries, then synthesizes once. Saved priorities are reused by synthesis.
Reprocess previews accept `selection=attempted` to freeze distinct previously attempted news IDs at their latest source hashes and force regrouping. `run_mode=history_only` requires paused, idle AI; startup uses the existing three-stage validation gate, enables only this durable batch, and pauses on completion, cancellation or failure. It does not consume realtime backlog. Grouping finishes before downstream review; completion also waits for the final reading snapshot. Existing range/shared behavior remains the default. These additions use existing JSON storage and require no schema migration.
An exclusive batch also enrolls unfinished member packages required by its events, using the same frozen configuration. This may add package reviews beyond the selected news; unrelated realtime work stays queued. Failed members remain failed until explicitly retried. Any grouping, package or event failure blocks advanced reading generation. The first exclusive start initializes the dispatch boundary while preserving an existing boundary. Event plan dates must match the verified time point's precision, original time text and at least one corresponding time quote.