Coordinate AI retries and recover shared provider outages

Retry transient requests after one and three minutes and coordinate provider recovery across processes with bounded probe intervals. Preserve completed results when recovery bookkeeping fails, keep concurrent failure accounting accurate, and expose retry state in the UI.

deploy/README.md

@@ -179,7 +179,9 @@ Use `scripts/smoke_news_research.py` only with disposable CI PostgreSQL credenti
 
 Real-news acceptance requires explicit permission for the bounded sample and necessary model access. Export original records in a read-only repeatable-read transaction, copy only necessary provider fields to a private isolated configuration, and never copy the production `.env`. After processing, review keyword coverage, event boundaries, reposts, evidence, dates and ratings, retain a credential-free report, and remove temporary keys and test containers. The initial November 3 round used thirty news items and additive tables only; the subsequent chronological supplement uses twelve frozen originals and a separate additional request budget. Legacy sample replay/reset commands target a different schema and are not part of this workflow; do not run them for this transition. Legacy reports and task counts do not prove current quality or savings. Missing usage is not zero cost.
 
-News transport uses at most two retries after 30/60 seconds, honoring valid Retry-After up to 120 seconds while retaining the worker slot. Partial output and permanent failures are not replayed. Structural submission repair is bounded to one correction using already available evidence. Default Skill changes are not automatically published to live nodes. Research tools are restricted to local sources; evidence gaps are reported without attempting external access.
+All AI generation requests share one transport retry owner; SDK retries are disabled. A single request retries after 60 and then 180 seconds while retaining its task slot. Partial text, thinking or tool arguments, ambiguous timeouts and permanent errors are not replayed. Three distinct transiently failing requests within sixty seconds without a completed success open a Redis-coordinated circuit scoped to protocol, normalized endpoint and credential fingerprint. It waits 180, 600 and then 900 seconds, repeating 900-second waits until recovery. Only one real request probes recovery across processes; successful completion closes the circuit. An expiring, renewed lease recovers crashed probes. Normal concurrent successes clear the failure window without invalidating peers that subsequently fail. Complete responses and known usage remain available when post-response bookkeeping or lease cleanup fails; admission for new requests still requires Redis coordination. These fixed intervals replace upstream Retry-After scheduling, including five-hour limit hints. Redis coordination failure blocks outgoing generation rather than bypassing protection.
+
+Chat emits additive `retry` SSE events and cancellation stops waiting. `GET /api/settings/provider-availability` reports normal/waiting/probing per configured provider without credentials. News queue status adds `waiting_retry`; waiting requests retain their accepted task positions. Trading retries refresh market input without replaying previous tools or trades. Synchronous connection tests and news enable validation fail promptly while the shared circuit is waiting; they do not enable processing after a failed validation. CI runs `scripts/smoke_ai_recovery.py` against its disposable Redis, using `AI_RECOVERY_TEST_REDIS_URL`. Structural submission repair is bounded to one correction using already available evidence. Default Skill changes are not automatically published to live nodes. Research tools are restricted to local sources; evidence gaps are reported without attempting external access.
 
 ## Global economic calendar