ivang/ecommerce-tool-monorepo

Fork 0

[PRD] AI-native Goal Supervisor 与失败自动恢复闭环 #1

New issue

Open

opened 2026-07-15 01:09:31 +00:00 by ivang · 0 comments

ivang commented

2026-07-15 01:09:31 +00:00

Owner

Problem Statement

用户向 AI 提交 pipeline 任务时，关注的是输入能否转化为最终制品，而不是理解 DataScraping、UPM 映射、LLM 优化、图片处理、聚合或结算中的具体失败步骤。当前系统把某次执行尝试的失败直接投影成用户可见的 failed 或 partial_failed，并要求用户查看错误、选择失败记录和手动重试。这把内部实现复杂度泄漏给用户，也使飞书 Bot、Web、Codex 和各 pipeline 各自承担一部分恢复逻辑。

一次失败本质上只是执行过程中观察到的内部事实。它可能包含网络超时、模型输出异常、源数据不可用、权限不足、程序缺陷或验收不通过等细节，但不应天然等同于用户目标失败。系统需要在保留完整失败事实和审计链的同时，持续尝试确定性恢复或受约束 Agent 修复，直到产生满足验收标准的制品、确实需要用户输入，或耗尽有界恢复预算。

现有 task-hub 已承担飞书渠道、Codex 派发和事件回流，但其 Task 使用 JSON Store，failed 是不可退出终态，部分 watcher 依赖进程内状态；现有 Reliable Task Engine 已具备 Postgres task/item/attempt ledger、lease 和 transactional outbox。两者尚未形成以 Goal 为中心的持久化自动恢复闭环。

Solution

把系统升级为 AI-native Goal Supervisor：用户提交的是带验收标准的 Goal，pipeline Task 和每次 Attempt 都只是实现 Goal 的内部执行。任何 Attempt 失败时，由领域执行模块原子记录 Failure Case 并通过 outbox 发布标准化失败事件；task-hub 中新增持久化 Recovery Coordinator，消费这些事件，优先执行领域内确定性恢复，必要时向 Codex 派发带上下文、授权范围、恢复预算和验证规则的结构化 Repair Plan。修复后的结果必须经过独立 Verifier 验证，验证成功才把 Goal 投影为 delivered。

用户默认只看到 accepted -> working -> verifying -> delivered。内部失败在恢复预算未耗尽时仍属于 working，不要求用户理解具体步骤。只有缺少输入或授权时进入 needs_input，只有所有允许的恢复策略和预算均耗尽后进入 exhausted。原始失败、后续 Attempts、Repair Plans 和制品版本全部追加保存，不通过覆盖历史记录伪造成功。

总体流程：

用户输入 + 验收标准
        |
        v
       Goal
        |
        v
  Pipeline Task / Attempt
        |
        +----------------------+
        |                      |
        v                      v
    中间状态              Attempt Failed
        |                      |
        |                      v
        |                Failure Case + Outbox
        |                      |
        |                      v
        |              Recovery Coordinator
        |                      |
        |          +-----------+-----------+
        |          |                       |
        |          v                       v
        |   确定性领域恢复            受约束 Codex 修复
        |          |                       |
        |          +-----------+-----------+
        |                      |
        |                      v
        |                   新 Attempt
        |                      |
        +----------------------+
                               |
                               v
                            Verifier
                               |
                   +-----------+-----------+
                   |                       |
                   v                       v
               delivered          needs_input / exhausted

User Stories

As a pipeline user, I want to submit an input and receive the requested artifact, so that I do not need to understand internal pipeline stages.
As a pipeline user, I want transient internal failures to be repaired automatically, so that normal infrastructure instability does not interrupt my work.
As a pipeline user, I want the task to remain visibly working while bounded recovery is in progress, so that intermediate failures are not presented as premature final outcomes.
As a pipeline user, I want to see when the system is verifying the artifact, so that completion means the result was checked rather than merely generated.
As a pipeline user, I want the system to ask me only for information or authority it cannot derive, so that I am interrupted only when my action is genuinely required.
As a pipeline user, I want a clear final explanation when recovery is exhausted, so that I understand what was delivered and what could not be produced.
As a pipeline user, I want partial usable artifacts to remain downloadable when some inputs are irrecoverable, so that successful work is not discarded.
As a pipeline user, I want the final artifact to include newly recovered records automatically, so that I do not need to manually re-aggregate results.
As a pipeline user, I want old artifact revisions to remain auditable, so that I can distinguish the original result from repaired versions.
As a pipeline user, I want duplicate retries to be prevented, so that repeated clicks, network retries or service restarts do not duplicate work or charges.
As a pipeline user, I want billing to include only newly successful work, so that recovery does not charge again for records already completed.
As a Feishu user, I want the Bot to continue supervising my Goal after dispatch, so that I can stay in my normal work channel instead of monitoring an engineering dashboard.
As a Feishu user, I want the Bot card to report working, verifying, delivered or needs-input states, so that technical Attempt failures remain internal.
As a Feishu user, I want a direct action when authorization or approval is required, so that I can unblock the Goal without reading raw logs.
As a Web user, I want a concise Goal status with optional diagnostic details, so that the default experience is simple while advanced inspection remains possible.
As an operator, I want every failed Attempt preserved immutably, so that automated recovery does not destroy forensic evidence.
As an operator, I want Failure Cases grouped by Goal, Task, item and process, so that I can inspect systemic failure patterns without coupling user experience to them.
As an operator, I want large logs, model responses and screenshots stored behind references, so that the transactional database remains queryable and sensitive details can have separate retention controls.
As an operator, I want recovery cases to have leases and deadlines, so that crashed workers can be recovered without concurrent duplicate repair.
As an operator, I want poison failures to become exhausted after a bounded budget, so that the system cannot create infinite retry loops.
As an operator, I want stale failure events ignored when a later Attempt has already succeeded, so that delayed events cannot regress current state.
As an operator, I want duplicate outbox deliveries handled idempotently, so that at-least-once delivery remains safe.
As an operator, I want each Repair Plan and its generation recorded, so that I can explain why the system chose a particular recovery action.
As an operator, I want deterministic recovery attempted before invoking Codex, so that cheap and predictable repairs do not consume Agent time.
As an operator, I want Codex constrained by allowed actions, authorization and budget, so that automatic recovery cannot silently expand its authority.
As an operator, I want high-risk actions to remain approval-gated, so that Goal supervision does not bypass existing safety controls.
As a pipeline maintainer, I want domain modules to own Task, Attempt, checkpoint and artifact facts, so that task-hub does not become a second pipeline engine.
As a pipeline maintainer, I want failures emitted as a stable Failure Envelope, so that task-hub does not parse process-specific log strings.
As a pipeline maintainer, I want structured error codes and classes, so that recovery decisions survive wording and localization changes.
As a pipeline maintainer, I want process checkpoints referenced from Failure Cases, so that repair can resume from an appropriate durable point.
As a pipeline maintainer, I want optional-process failure semantics preserved, so that non-critical enhancement failures do not unnecessarily block artifact delivery.
As a pipeline maintainer, I want root-level aggregation, conversion, settlement and publication failures recoverable separately from item failures, so that every execution layer uses the correct repair command.
As a task-hub maintainer, I want a single Recovery Coordinator interface, so that classification, budgeting, Agent dispatch, verification and status projection remain local to one deep module.
As a task-hub maintainer, I want recovery state stored durably rather than in process-local watchers, so that service restarts do not abandon Goals.
As a task-hub maintainer, I want Goal records correlated with Codex thread/turn IDs and domain task IDs, so that events can be traced across all execution planes.
As a task-hub maintainer, I want Codex to return a structured Repair Plan, so that free-form assistant text is never treated as an executable recovery command.
As a task-hub maintainer, I want Verifier results independent from Codex self-reporting, so that an Agent saying “done” cannot complete a Goal without evidence.
As a developer, I want production adapters and in-memory test adapters at owned network seams, so that recovery behavior can be tested without real Feishu, Codex or pipeline dependencies.
As a developer, I want the highest-level recovery behavior covered by a single integration seam, so that internal refactors do not require rewriting the acceptance suite.
As a developer, I want explicit observability for Goal, Failure Case and Recovery Attempt IDs, so that logs and metrics can be correlated without exposing secrets.
As a product owner, I want recovery success rate, needs-input rate, exhausted rate, repair latency and incremental cost measured, so that the value and risk of autonomy can be evaluated.
As a product owner, I want recovery policies configurable by task type and process, so that different pipelines can adopt autonomy gradually.
As a product owner, I want a kill switch for Agent repair while deterministic recovery remains available, so that operational risk can be controlled independently.
As a security owner, I want Failure Envelopes sanitized before Agent dispatch, so that secrets, credentials and unrelated customer data are not placed into prompts.
As a security owner, I want tenant identity propagated through every recovery command, so that one tenant's Failure Case cannot be repaired using another tenant's context.
As an auditor, I want immutable links from Goal to Task, Attempts, failures, repairs, verification evidence and artifact revisions, so that final delivery is reproducible and explainable.

Implementation Decisions

Introduce the domain terms Goal, Execution Task, Attempt, Failure Case, Recovery Case, Repair Plan, Verification Evidence and Artifact Revision. A failed Attempt is an internal fact; only exhausted is a final Goal failure.
Keep domain execution modules as the source of truth for Task, item, Attempt, checkpoint, billing and artifact state. task-hub must not update pipeline tables directly.
Build Recovery Coordinator as a deep module inside task-hub. Its external interface is event-driven: consume a normalized Failure Envelope and eventually project one of working, needs_input, delivered or exhausted. Classification, budgets, Codex dispatch and verification remain hidden in its implementation.
Use the existing transactional outbox pattern to publish task.failure.recorded, task.attempt.succeeded, task.artifact.created and related lifecycle events atomically with domain state changes.
Use at-least-once delivery with a durable task-hub inbox, unique event keys and idempotent commands. Exactly-once transport is not required; exactly-once observable effects are required.
Add a Postgres-backed Goal and Recovery ledger. JSON Store and in-process maps may remain for non-critical Bot MVP data but are not authoritative for recovery.
Store structured Failure Case fields in Postgres. Store large or sensitive diagnostics in object storage and reference them through detailRef; apply sanitization, encryption and retention policies before persistence or Agent dispatch.
A Failure Envelope must include tenant identity, Goal ID, Task ID, optional item ID, Attempt ID, process/stage, structured error code and class, checkpoint reference, observed entity version, allowed recovery actions and correlation IDs.
Do not require complete classification before recording a failure. Raw structured facts are persisted first; classification is a versioned recovery decision that can be improved later.
Use separate state machines for the user Goal, internal Attempts and Failure Cases. The following state machine comes from the architecture prototype in this discussion:

Goal:
  accepted -> working -> verifying -> delivered
                         |            
                         +-> needs_input
                         +-> exhausted

Attempt:
  queued -> running -> succeeded
                    +-> failed
                    +-> timed_out

Failure Case:
  open -> classified -> repair_queued -> repairing -> verifying -> resolved
                             |              |             |
                             |              +-> retry_scheduled
                             +----------------------------> needs_input
                             +----------------------------> exhausted

Attempt failure does not project the Goal to a terminal red state while a permitted recovery path and budget remain.
Implement three recovery levels: L0 deterministic domain recovery, L1 constrained Agent repair, and L2 needs-input/exhausted escalation. L0 always precedes L1 unless the Failure Envelope explicitly declares that deterministic recovery is unavailable.
Deterministic recovery includes transient retry, lease recovery, dependency waiting, idempotent replay, re-aggregation, conversion retry and settlement retry through domain-owned commands.
Agent repair is allowed for non-deterministic but bounded problems such as malformed model output, data-specific Prompt behavior, recoverable mapping gaps or parameter adjustment from a checkpoint.
Source loss, missing authorization, insufficient funds, safety approval and deterministic implementation defects cannot be silently bypassed. They become needs-input or exhausted according to policy.
Codex must return a validated structured Repair Plan, not executable free text. The decision-rich shape from the prototype is:

{
  "strategy": "repair_prompt",
  "targetProcess": "UpmTextLocalizationProcess",
  "resumeFrom": "checkpoint://...",
  "temporaryInstruction": "...",
  "expectedEffect": "...",
  "verification": ["output conforms to UPM schema"]
}

Recovery Coordinator executes Repair Plans only through injected domain control ports. Production uses internal HTTP/event adapters; tests use in-memory adapters. Codex does not receive direct database write authority.
Every Recovery Case has a retry budget, Agent-turn budget, elapsed-time deadline, optional credit budget, generation number, lease and next eligible execution time.
Recovery commands use failureId + repairGeneration as the idempotency identity. Repeated events, HTTP retries and restarts must resolve to the same observable execution.
Before executing a repair, compare the Failure Envelope's observed version with the current Task/item state. A later success resolves a stale Failure Case without another repair.
A repair creates a new Attempt; it never rewrites the terminal facts of a previous Attempt.
Recovered item results trigger incremental aggregation and a new immutable Artifact Revision. Only newly successful items are eligible for incremental settlement.
Add independent Verifier modules per Goal/task type. Verification consumes Goal acceptance criteria and artifact references and returns structured Verification Evidence. Codex completion text is not verification evidence.
Correlate each Goal with source channel, user/tenant, Codex workspace/thread/turn IDs, and all domain execution references. Tool calls that start domain tasks must return and persist those references.
Change task-hub Task semantics so an underlying turn or pipeline failure can enter recovering rather than the current irreversible failed transition. A truly terminal state is named exhausted to distinguish it from Attempt failure.
Replace process-local recovery watchers with durable inbox claims, leases and a recovery sweeper. SSE/history remains a Codex event adapter but is not the sole source of recovery durability.
Default user interfaces display Goal progress and delivery state. Raw process errors and Failure Cases move to an optional diagnostic surface for operators.
Feishu and Web display working, verifying, needs_input, delivered and exhausted. While recovery is active they may report counts such as “18/21 generated, 3 recovering” without asking the user to choose repair steps.
needs_input includes a specific requested action, reason, expiry and continuation command. User input resumes the existing Recovery Case rather than creating an unrelated new Goal.
Emit metrics for recovery success, deterministic-versus-Agent strategy, latency, attempts, incremental cost, needs-input, exhausted cases and stale-event suppression.
Provide task-type and process-level policy configuration plus an operational kill switch for L1 Agent repair.

Testing Decisions

The primary and highest test seam is Recovery Coordinator's external behavior. A test feeds durable domain events and observes Goal state, issued domain commands, Agent dispatch, verification evidence, billing effects and Artifact Revision references. Tests do not assert private classifier functions or internal step ordering.
Use a real isolated Postgres schema for Goal, inbox and Recovery Case persistence so transaction, lease, uniqueness and restart behavior are verified at the same seam used by production.
Treat pipeline engines and codex-web-backend as remote-but-owned dependencies. Define narrow injected ports with production HTTP/event adapters and in-memory test adapters.
Extend the existing task-hub full-chain testing pattern that drives intake, dispatch, Fake Codex events and final Task state. The new suite drives Failure Envelopes and recovery outcomes through the same top-level hub interface.
Reuse Reliable Task Engine testing patterns for atomic ledger writes, FOR UPDATE SKIP LOCKED, lease expiry, immutable Attempts and transactional outbox creation.
Reuse async-job worker recovery tests for duplicate signals, cooldowns, retry scheduling, lease heartbeat, worker crashes and terminal projection.
Verify a transient infrastructure failure resolves through L0 without dispatching Codex.
Verify a recoverable model-output failure dispatches one structured L1 Repair Plan and reaches delivered only after Verifier success.
Verify duplicate failure events do not create duplicate Recovery Cases, Attempts, reservations or charges.
Verify a crash after inbox claim and before completion is recovered after lease expiry.
Verify a stale failure event received after item success is resolved without a repair command.
Verify multiple concurrent failure events for the same item serialize by entity version and repair generation.
Verify source unavailable, authorization missing, insufficient funds and approval-required cases enter needs-input with the correct user action.
Verify deterministic code/schema defects cannot be converted into arbitrary Prompt fallback.
Verify retry, Agent-turn, time and credit budgets terminate poison cases as exhausted.
Verify original Attempts, Failure Cases and Artifact Revisions remain immutable after recovery.
Verify recovered items are added to a new Artifact Revision and only newly successful work is settled.
Verify an Agent response that claims success but fails artifact verification does not deliver the Goal.
Verify task-hub restart preserves active Recovery Cases and resumes event consumption without relying on in-memory watchers.
Verify tenant identity and allowed actions are enforced by every production adapter and reflected in audit records.
Add Web and Feishu adapter tests ensuring recoverable internal failures remain presented as working, while needs-input and exhausted render actionable final states.
Good tests assert observable outcomes at module interfaces and survive changes to classifiers, worker layout, queue transport or internal strategy selection.

Out of Scope

Replacing existing domain pipelines, Reliable Task Engine, async-job workers or Codex runtime with task-hub.
Giving Codex unrestricted database, credential, deployment or production mutation authority.
Automatically approving high-risk commands, payments, permissions or external publication actions.
Infinite retries, unlimited Agent turns or unsupported fallback behavior.
Treating Redis, a message queue, vector database or raw logs as the source of truth for Failure Cases.
Building a general cross-company incident-management or AIOps platform.
Redesigning Feishu Bot registration, account binding or the main authentication system except where Goal correlation requires additional identifiers.
Guaranteeing recovery for permanently unavailable source data or missing user authority.
Hiding exhausted outcomes or fabricating a complete artifact when acceptance criteria were not met.
Removing advanced diagnostics, audit history or manual operator intervention surfaces.

Further Notes

Recommended delivery order:

M1  Goal/Failure Case schema + structured events + user-state projection
M2  task-hub Postgres inbox + durable Recovery Coordinator + L0 strategies
M3  Codex Repair Plan contract + L1 dispatch + independent Verifier
M4  incremental artifact/settlement integration + Web/Feishu Goal UX
M5  metrics, policy controls, kill switch and staged rollout

The existing per-item repair endpoint remains useful as an operator command, but it should become one adapter behind Recovery Coordinator rather than the primary user workflow.
The existing Task ledger and outbox provide the correct fact-producing seam. The major architectural change is promoting task-hub from channel dispatcher to durable Goal Supervisor without moving domain execution ownership into it.
Rollout should begin with one task type and deterministic L0 recovery enabled. L1 Agent repair should be activated per process only after structured Failure Envelopes and Verifiers exist.
The principal product metric is not “retry succeeded”; it is “Goal delivered without unnecessary user intervention, within configured safety and cost budgets.”

## Problem Statement 用户向 AI 提交 pipeline 任务时，关注的是输入能否转化为最终制品，而不是理解 DataScraping、UPM 映射、LLM 优化、图片处理、聚合或结算中的具体失败步骤。当前系统把某次执行尝试的失败直接投影成用户可见的 `failed` 或 `partial_failed`，并要求用户查看错误、选择失败记录和手动重试。这把内部实现复杂度泄漏给用户，也使飞书 Bot、Web、Codex 和各 pipeline 各自承担一部分恢复逻辑。一次失败本质上只是执行过程中观察到的内部事实。它可能包含网络超时、模型输出异常、源数据不可用、权限不足、程序缺陷或验收不通过等细节，但不应天然等同于用户目标失败。系统需要在保留完整失败事实和审计链的同时，持续尝试确定性恢复或受约束 Agent 修复，直到产生满足验收标准的制品、确实需要用户输入，或耗尽有界恢复预算。现有 task-hub 已承担飞书渠道、Codex 派发和事件回流，但其 Task 使用 JSON Store，`failed` 是不可退出终态，部分 watcher 依赖进程内状态；现有 Reliable Task Engine 已具备 Postgres task/item/attempt ledger、lease 和 transactional outbox。两者尚未形成以 Goal 为中心的持久化自动恢复闭环。 ## Solution 把系统升级为 AI-native Goal Supervisor：用户提交的是带验收标准的 Goal，pipeline Task 和每次 Attempt 都只是实现 Goal 的内部执行。任何 Attempt 失败时，由领域执行模块原子记录 Failure Case 并通过 outbox 发布标准化失败事件；task-hub 中新增持久化 Recovery Coordinator，消费这些事件，优先执行领域内确定性恢复，必要时向 Codex 派发带上下文、授权范围、恢复预算和验证规则的结构化 Repair Plan。修复后的结果必须经过独立 Verifier 验证，验证成功才把 Goal 投影为 `delivered`。用户默认只看到 `accepted -> working -> verifying -> delivered`。内部失败在恢复预算未耗尽时仍属于 `working`，不要求用户理解具体步骤。只有缺少输入或授权时进入 `needs_input`，只有所有允许的恢复策略和预算均耗尽后进入 `exhausted`。原始失败、后续 Attempts、Repair Plans 和制品版本全部追加保存，不通过覆盖历史记录伪造成功。总体流程： ```text 用户输入 + 验收标准 | v Goal | v Pipeline Task / Attempt | +----------------------+ | | v v 中间状态 Attempt Failed | | | v | Failure Case + Outbox | | | v | Recovery Coordinator | | | +-----------+-----------+ | | | | v v | 确定性领域恢复受约束 Codex 修复 | | | | +-----------+-----------+ | | | v | 新 Attempt | | +----------------------+ | v Verifier | +-----------+-----------+ | | v v delivered needs_input / exhausted ``` ## User Stories 1. As a pipeline user, I want to submit an input and receive the requested artifact, so that I do not need to understand internal pipeline stages. 2. As a pipeline user, I want transient internal failures to be repaired automatically, so that normal infrastructure instability does not interrupt my work. 3. As a pipeline user, I want the task to remain visibly working while bounded recovery is in progress, so that intermediate failures are not presented as premature final outcomes. 4. As a pipeline user, I want to see when the system is verifying the artifact, so that completion means the result was checked rather than merely generated. 5. As a pipeline user, I want the system to ask me only for information or authority it cannot derive, so that I am interrupted only when my action is genuinely required. 6. As a pipeline user, I want a clear final explanation when recovery is exhausted, so that I understand what was delivered and what could not be produced. 7. As a pipeline user, I want partial usable artifacts to remain downloadable when some inputs are irrecoverable, so that successful work is not discarded. 8. As a pipeline user, I want the final artifact to include newly recovered records automatically, so that I do not need to manually re-aggregate results. 9. As a pipeline user, I want old artifact revisions to remain auditable, so that I can distinguish the original result from repaired versions. 10. As a pipeline user, I want duplicate retries to be prevented, so that repeated clicks, network retries or service restarts do not duplicate work or charges. 11. As a pipeline user, I want billing to include only newly successful work, so that recovery does not charge again for records already completed. 12. As a Feishu user, I want the Bot to continue supervising my Goal after dispatch, so that I can stay in my normal work channel instead of monitoring an engineering dashboard. 13. As a Feishu user, I want the Bot card to report working, verifying, delivered or needs-input states, so that technical Attempt failures remain internal. 14. As a Feishu user, I want a direct action when authorization or approval is required, so that I can unblock the Goal without reading raw logs. 15. As a Web user, I want a concise Goal status with optional diagnostic details, so that the default experience is simple while advanced inspection remains possible. 16. As an operator, I want every failed Attempt preserved immutably, so that automated recovery does not destroy forensic evidence. 17. As an operator, I want Failure Cases grouped by Goal, Task, item and process, so that I can inspect systemic failure patterns without coupling user experience to them. 18. As an operator, I want large logs, model responses and screenshots stored behind references, so that the transactional database remains queryable and sensitive details can have separate retention controls. 19. As an operator, I want recovery cases to have leases and deadlines, so that crashed workers can be recovered without concurrent duplicate repair. 20. As an operator, I want poison failures to become exhausted after a bounded budget, so that the system cannot create infinite retry loops. 21. As an operator, I want stale failure events ignored when a later Attempt has already succeeded, so that delayed events cannot regress current state. 22. As an operator, I want duplicate outbox deliveries handled idempotently, so that at-least-once delivery remains safe. 23. As an operator, I want each Repair Plan and its generation recorded, so that I can explain why the system chose a particular recovery action. 24. As an operator, I want deterministic recovery attempted before invoking Codex, so that cheap and predictable repairs do not consume Agent time. 25. As an operator, I want Codex constrained by allowed actions, authorization and budget, so that automatic recovery cannot silently expand its authority. 26. As an operator, I want high-risk actions to remain approval-gated, so that Goal supervision does not bypass existing safety controls. 27. As a pipeline maintainer, I want domain modules to own Task, Attempt, checkpoint and artifact facts, so that task-hub does not become a second pipeline engine. 28. As a pipeline maintainer, I want failures emitted as a stable Failure Envelope, so that task-hub does not parse process-specific log strings. 29. As a pipeline maintainer, I want structured error codes and classes, so that recovery decisions survive wording and localization changes. 30. As a pipeline maintainer, I want process checkpoints referenced from Failure Cases, so that repair can resume from an appropriate durable point. 31. As a pipeline maintainer, I want optional-process failure semantics preserved, so that non-critical enhancement failures do not unnecessarily block artifact delivery. 32. As a pipeline maintainer, I want root-level aggregation, conversion, settlement and publication failures recoverable separately from item failures, so that every execution layer uses the correct repair command. 33. As a task-hub maintainer, I want a single Recovery Coordinator interface, so that classification, budgeting, Agent dispatch, verification and status projection remain local to one deep module. 34. As a task-hub maintainer, I want recovery state stored durably rather than in process-local watchers, so that service restarts do not abandon Goals. 35. As a task-hub maintainer, I want Goal records correlated with Codex thread/turn IDs and domain task IDs, so that events can be traced across all execution planes. 36. As a task-hub maintainer, I want Codex to return a structured Repair Plan, so that free-form assistant text is never treated as an executable recovery command. 37. As a task-hub maintainer, I want Verifier results independent from Codex self-reporting, so that an Agent saying “done” cannot complete a Goal without evidence. 38. As a developer, I want production adapters and in-memory test adapters at owned network seams, so that recovery behavior can be tested without real Feishu, Codex or pipeline dependencies. 39. As a developer, I want the highest-level recovery behavior covered by a single integration seam, so that internal refactors do not require rewriting the acceptance suite. 40. As a developer, I want explicit observability for Goal, Failure Case and Recovery Attempt IDs, so that logs and metrics can be correlated without exposing secrets. 41. As a product owner, I want recovery success rate, needs-input rate, exhausted rate, repair latency and incremental cost measured, so that the value and risk of autonomy can be evaluated. 42. As a product owner, I want recovery policies configurable by task type and process, so that different pipelines can adopt autonomy gradually. 43. As a product owner, I want a kill switch for Agent repair while deterministic recovery remains available, so that operational risk can be controlled independently. 44. As a security owner, I want Failure Envelopes sanitized before Agent dispatch, so that secrets, credentials and unrelated customer data are not placed into prompts. 45. As a security owner, I want tenant identity propagated through every recovery command, so that one tenant's Failure Case cannot be repaired using another tenant's context. 46. As an auditor, I want immutable links from Goal to Task, Attempts, failures, repairs, verification evidence and artifact revisions, so that final delivery is reproducible and explainable. ## Implementation Decisions - Introduce the domain terms Goal, Execution Task, Attempt, Failure Case, Recovery Case, Repair Plan, Verification Evidence and Artifact Revision. A failed Attempt is an internal fact; only `exhausted` is a final Goal failure. - Keep domain execution modules as the source of truth for Task, item, Attempt, checkpoint, billing and artifact state. task-hub must not update pipeline tables directly. - Build Recovery Coordinator as a deep module inside task-hub. Its external interface is event-driven: consume a normalized Failure Envelope and eventually project one of `working`, `needs_input`, `delivered` or `exhausted`. Classification, budgets, Codex dispatch and verification remain hidden in its implementation. - Use the existing transactional outbox pattern to publish `task.failure.recorded`, `task.attempt.succeeded`, `task.artifact.created` and related lifecycle events atomically with domain state changes. - Use at-least-once delivery with a durable task-hub inbox, unique event keys and idempotent commands. Exactly-once transport is not required; exactly-once observable effects are required. - Add a Postgres-backed Goal and Recovery ledger. JSON Store and in-process maps may remain for non-critical Bot MVP data but are not authoritative for recovery. - Store structured Failure Case fields in Postgres. Store large or sensitive diagnostics in object storage and reference them through `detailRef`; apply sanitization, encryption and retention policies before persistence or Agent dispatch. - A Failure Envelope must include tenant identity, Goal ID, Task ID, optional item ID, Attempt ID, process/stage, structured error code and class, checkpoint reference, observed entity version, allowed recovery actions and correlation IDs. - Do not require complete classification before recording a failure. Raw structured facts are persisted first; classification is a versioned recovery decision that can be improved later. - Use separate state machines for the user Goal, internal Attempts and Failure Cases. The following state machine comes from the architecture prototype in this discussion: ```text Goal: accepted -> working -> verifying -> delivered | +-> needs_input +-> exhausted Attempt: queued -> running -> succeeded +-> failed +-> timed_out Failure Case: open -> classified -> repair_queued -> repairing -> verifying -> resolved | | | | +-> retry_scheduled +----------------------------> needs_input +----------------------------> exhausted ``` - Attempt failure does not project the Goal to a terminal red state while a permitted recovery path and budget remain. - Implement three recovery levels: L0 deterministic domain recovery, L1 constrained Agent repair, and L2 needs-input/exhausted escalation. L0 always precedes L1 unless the Failure Envelope explicitly declares that deterministic recovery is unavailable. - Deterministic recovery includes transient retry, lease recovery, dependency waiting, idempotent replay, re-aggregation, conversion retry and settlement retry through domain-owned commands. - Agent repair is allowed for non-deterministic but bounded problems such as malformed model output, data-specific Prompt behavior, recoverable mapping gaps or parameter adjustment from a checkpoint. - Source loss, missing authorization, insufficient funds, safety approval and deterministic implementation defects cannot be silently bypassed. They become needs-input or exhausted according to policy. - Codex must return a validated structured Repair Plan, not executable free text. The decision-rich shape from the prototype is: ```json { "strategy": "repair_prompt", "targetProcess": "UpmTextLocalizationProcess", "resumeFrom": "checkpoint://...", "temporaryInstruction": "...", "expectedEffect": "...", "verification": ["output conforms to UPM schema"] } ``` - Recovery Coordinator executes Repair Plans only through injected domain control ports. Production uses internal HTTP/event adapters; tests use in-memory adapters. Codex does not receive direct database write authority. - Every Recovery Case has a retry budget, Agent-turn budget, elapsed-time deadline, optional credit budget, generation number, lease and next eligible execution time. - Recovery commands use `failureId + repairGeneration` as the idempotency identity. Repeated events, HTTP retries and restarts must resolve to the same observable execution. - Before executing a repair, compare the Failure Envelope's observed version with the current Task/item state. A later success resolves a stale Failure Case without another repair. - A repair creates a new Attempt; it never rewrites the terminal facts of a previous Attempt. - Recovered item results trigger incremental aggregation and a new immutable Artifact Revision. Only newly successful items are eligible for incremental settlement. - Add independent Verifier modules per Goal/task type. Verification consumes Goal acceptance criteria and artifact references and returns structured Verification Evidence. Codex completion text is not verification evidence. - Correlate each Goal with source channel, user/tenant, Codex workspace/thread/turn IDs, and all domain execution references. Tool calls that start domain tasks must return and persist those references. - Change task-hub Task semantics so an underlying turn or pipeline failure can enter recovering rather than the current irreversible `failed` transition. A truly terminal state is named `exhausted` to distinguish it from Attempt failure. - Replace process-local recovery watchers with durable inbox claims, leases and a recovery sweeper. SSE/history remains a Codex event adapter but is not the sole source of recovery durability. - Default user interfaces display Goal progress and delivery state. Raw process errors and Failure Cases move to an optional diagnostic surface for operators. - Feishu and Web display `working`, `verifying`, `needs_input`, `delivered` and `exhausted`. While recovery is active they may report counts such as “18/21 generated, 3 recovering” without asking the user to choose repair steps. - `needs_input` includes a specific requested action, reason, expiry and continuation command. User input resumes the existing Recovery Case rather than creating an unrelated new Goal. - Emit metrics for recovery success, deterministic-versus-Agent strategy, latency, attempts, incremental cost, needs-input, exhausted cases and stale-event suppression. - Provide task-type and process-level policy configuration plus an operational kill switch for L1 Agent repair. ## Testing Decisions - The primary and highest test seam is Recovery Coordinator's external behavior. A test feeds durable domain events and observes Goal state, issued domain commands, Agent dispatch, verification evidence, billing effects and Artifact Revision references. Tests do not assert private classifier functions or internal step ordering. - Use a real isolated Postgres schema for Goal, inbox and Recovery Case persistence so transaction, lease, uniqueness and restart behavior are verified at the same seam used by production. - Treat pipeline engines and codex-web-backend as remote-but-owned dependencies. Define narrow injected ports with production HTTP/event adapters and in-memory test adapters. - Extend the existing task-hub full-chain testing pattern that drives intake, dispatch, Fake Codex events and final Task state. The new suite drives Failure Envelopes and recovery outcomes through the same top-level hub interface. - Reuse Reliable Task Engine testing patterns for atomic ledger writes, `FOR UPDATE SKIP LOCKED`, lease expiry, immutable Attempts and transactional outbox creation. - Reuse async-job worker recovery tests for duplicate signals, cooldowns, retry scheduling, lease heartbeat, worker crashes and terminal projection. - Verify a transient infrastructure failure resolves through L0 without dispatching Codex. - Verify a recoverable model-output failure dispatches one structured L1 Repair Plan and reaches delivered only after Verifier success. - Verify duplicate failure events do not create duplicate Recovery Cases, Attempts, reservations or charges. - Verify a crash after inbox claim and before completion is recovered after lease expiry. - Verify a stale failure event received after item success is resolved without a repair command. - Verify multiple concurrent failure events for the same item serialize by entity version and repair generation. - Verify source unavailable, authorization missing, insufficient funds and approval-required cases enter needs-input with the correct user action. - Verify deterministic code/schema defects cannot be converted into arbitrary Prompt fallback. - Verify retry, Agent-turn, time and credit budgets terminate poison cases as exhausted. - Verify original Attempts, Failure Cases and Artifact Revisions remain immutable after recovery. - Verify recovered items are added to a new Artifact Revision and only newly successful work is settled. - Verify an Agent response that claims success but fails artifact verification does not deliver the Goal. - Verify task-hub restart preserves active Recovery Cases and resumes event consumption without relying on in-memory watchers. - Verify tenant identity and allowed actions are enforced by every production adapter and reflected in audit records. - Add Web and Feishu adapter tests ensuring recoverable internal failures remain presented as working, while needs-input and exhausted render actionable final states. - Good tests assert observable outcomes at module interfaces and survive changes to classifiers, worker layout, queue transport or internal strategy selection. ## Out of Scope - Replacing existing domain pipelines, Reliable Task Engine, async-job workers or Codex runtime with task-hub. - Giving Codex unrestricted database, credential, deployment or production mutation authority. - Automatically approving high-risk commands, payments, permissions or external publication actions. - Infinite retries, unlimited Agent turns or unsupported fallback behavior. - Treating Redis, a message queue, vector database or raw logs as the source of truth for Failure Cases. - Building a general cross-company incident-management or AIOps platform. - Redesigning Feishu Bot registration, account binding or the main authentication system except where Goal correlation requires additional identifiers. - Guaranteeing recovery for permanently unavailable source data or missing user authority. - Hiding exhausted outcomes or fabricating a complete artifact when acceptance criteria were not met. - Removing advanced diagnostics, audit history or manual operator intervention surfaces. ## Further Notes - Recommended delivery order: ```text M1 Goal/Failure Case schema + structured events + user-state projection M2 task-hub Postgres inbox + durable Recovery Coordinator + L0 strategies M3 Codex Repair Plan contract + L1 dispatch + independent Verifier M4 incremental artifact/settlement integration + Web/Feishu Goal UX M5 metrics, policy controls, kill switch and staged rollout ``` - The existing per-item repair endpoint remains useful as an operator command, but it should become one adapter behind Recovery Coordinator rather than the primary user workflow. - The existing Task ledger and outbox provide the correct fact-producing seam. The major architectural change is promoting task-hub from channel dispatcher to durable Goal Supervisor without moving domain execution ownership into it. - Rollout should begin with one task type and deterministic L0 recovery enabled. L1 Agent repair should be activated per process only after structured Failure Envelopes and Verifiers exist. - The principal product metric is not “retry succeeded”; it is “Goal delivered without unnecessary user intervention, within configured safety and cost budgets.”

ivang self-assigned this

2026-07-15 01:11:28 +00:00

No labels

ready-for-agent

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

ivang/ecommerce-tool-monorepo#1

No description provided.

Rows
Columns