Quick answer: recover from observed state, not error wording
Classify the outcome before choosing a retry
Record whether execution is known not to have started, known to have failed without an effect, known to have succeeded, unknown, partially succeeded or produced an untrusted response. An HTTP status, exception string or fluent agent explanation is only one observation. The recovery branch comes from the strongest authoritative evidence about the target state.
Keep one logical operation across physical attempts
Assign a stable operation ID to the business intent and a unique attempt ID to every call, status lookup or compensating action. Reuse the provider-scoped idempotency key only when the provider contract, retention window, account scope and payload equivalence all match. A new attempt is not a new permission decision, and a reused key must not hide changed intent.
Stop when safety cannot be proved
Authorization denial, conflicting payloads, partial high-impact writes, unknown commit state and malformed or hostile outputs should fail closed. Route them to a named owner with the request hash, attempt history, known state, unknown fields and allowed next actions. Do not broaden credentials, invent missing arguments, repeat an ambiguous write or announce success to escape the error path.
Model one logical operation and many attempts
Business intent is the unit of control
A logical operation is the approved business change, such as creating one ticket, updating one account field or sending one reviewed message. It has an actor, tenant, target, exact payload or payload hash, approval state, risk tier and permitted effect. The operation remains the same when the client reconnects or asks a status endpoint what happened.
An attempt is one interaction with a tool
An attempt records one outbound request or status query with its own start and end time, transport result, provider request ID, response hash and trace span. Retries increase the attempt count but must not increase the intended effect count. OpenTelemetry's HTTP conventions provide a useful distinction by modeling client attempts and a resend count rather than treating a trace as one undifferentiated call.
Transport result and business result can diverge
A timeout can occur after the provider committed a write but before the response reached the client. A 200 response can contain the wrong entity, stale data, a schema violation or instructions that must not be executed. Store transport evidence and business-state evidence separately so operators can see exactly why the final disposition is SUCCEEDED, FAILED, UNKNOWN, PARTIAL or BLOCKED.
Correlation is not idempotency
A correlation ID helps locate related events; it does not prevent duplicates. An idempotency key can prevent or replay a duplicate only within the provider's documented semantics. A stable business key can help reconcile the target, but it can also collide if its scope is vague. Keep all three concepts explicit instead of calling every identifier a request ID.
Write an operation contract before exposing a tool
Define identity, authority and approval
State which user or service identity executes the call, which tenant and resources it may access, which scopes are necessary, and which actions require human approval. Authorization is enforced by the downstream system and integration layer, not inferred from a natural-language request. Record the policy version that produced the allow or deny decision.
Define inputs and business invariants
Use a strict input schema, but also validate business rules that types cannot express: account ownership, allowed state transitions, currency, time zone, maximum amount, immutable fields and version preconditions. Separate required facts from model-generated suggestions. Missing consequential values route to clarification or approval rather than plausible completion.
Define side effects and authoritative state
List every direct and downstream effect: records, messages, charges, files, notifications, webhooks and scheduled work. Name the endpoint or record that proves each effect and how long observation may lag. If no authoritative lookup exists, treat a lost response to a non-idempotent write as a design gap, not as permission to retry.
Define recovery before the first failure
Document retryable conditions, maximum attempts, elapsed-time budget, backoff and jitter, idempotency scope, status lookup, partial-success semantics, compensation, dead-letter destination and accountable owner. Test the contract with synthetic or reversible records. A runbook written after an incident cannot retroactively prove what an earlier retry did.
Use an explicit state machine
Pre-execution states
PLANNED identifies intent; VALIDATED means schema and business invariants passed; APPROVED means the current identity and approval policy permit the exact effect. A rejection in these states has no tool-side effect and should not enter a network retry loop. Repairing input or obtaining approval creates a reviewed transition, not a silent mutation of the same attempt.
In-flight and acknowledged states
SENT means the request left the integration boundary. ACKNOWLEDGED means the provider returned an operation or request identifier, not necessarily that the business effect completed. Preserve that identifier for status queries and support. If the connection drops after SENT, the correct state is usually UNKNOWN, not FAILED.
Terminal and recoverable states
Use SUCCEEDED only when the intended state is verified, FAILED when non-application or terminal failure is proved, PARTIAL when a subset committed, and BLOCKED when policy or evidence forbids progression. COMPENSATED records a verified compensating effect but does not erase the original action or guarantee that every downstream consequence was reversed.
Unknown is a real operational state
UNKNOWN should pause dependent actions, customer-facing confirmation and fresh writes. It needs an owner, next observation time and a reconciliation plan. Converting unknown to failed for dashboard neatness creates duplicate risk; converting it to success hides unfinished or unauthorized work.
Build a provider-specific retry decision matrix
Prove the request semantics
RFC 9110 cautions against automatic retry of non-idempotent requests unless the client knows the operation is effectively idempotent or knows the original was not applied. HTTP method alone is not enough for a business workflow. Document whether the actual endpoint, account, key, payload and downstream chain support safe repetition.
Choose one retry owner
SDKs, gateways, queues, orchestrators and agent loops may each retry. Pick one layer for the logical policy and expose lower-layer retries in telemetry. Three attempts at three layers can become twenty-seven physical calls, overload the provider and defeat the total budget even though every component believes it retried only twice.
Bound count, time and concurrency
Set a maximum attempt count, total elapsed budget, per-attempt timeout and maximum concurrent operations. Use exponential or provider-recommended backoff with jitter for eligible transient conditions. Backoff reduces synchronized pressure; it does not establish idempotency, repair invalid inputs or resolve an unknown commit.
Honor provider guidance without outsourcing judgment
Parse Retry-After according to the provider contract and RFC semantics, preserve the absolute time basis and cap it within the workflow deadline. A missing header does not make every 429 or 503 immediately retryable. A present header does not authorize a high-impact write whose prior outcome remains unknown.
Exhaustion is a designed transition
When count or time budget ends, open the circuit or pause the queue, retain the operation, and route a compact evidence bundle. Do not reset the counter by starting a new agent turn. Recovery requires an explicit operator decision, a provider status change or a newly validated contract condition.
Failure mode 1: authentication or authorization denial
Recognize the condition
Credentials may be missing, expired or revoked, while a valid identity may lack the requested scope, tenant membership or resource permission. Keep authentication and authorization separate because refreshing a valid token cannot fix a deliberate deny. Capture the downstream denial without copying secrets into prompts or ordinary logs.
Know what the denial proves
A pre-execution denial usually proves that the requested effect did not begin, but verify provider semantics before making that invariant. It does not prove another identity may perform the action, that consent exists or that the user owns the target. Preserve the identity, tenant, policy version and requested scope used for the decision.
Prohibit credential substitution
The agent must not try a service account, administrator token, neighboring tenant or broader scope simply because the first identity failed. Token refresh is allowed only through an approved mechanism for the same intended identity and scope. Persistent denial routes to the integration or resource owner.
Enforce deterministic controls
Keep credential acquisition, secret storage, token audience, tenant binding, least privilege and downstream authorization outside model judgment. The model can explain a deny or request missing consent; it cannot grant itself access. High-impact approval remains a separate gate even after authorization succeeds.
Inject two denial tests
Test an expired token that may be refreshed through the approved path, then a valid token that lacks one required permission. Add a cross-tenant target with a distinguishable name and assert that the agent neither leaks existence nor falls back to another credential. Pass evidence includes zero target changes and the correct owner route.
Failure mode 2: schema or business-validation failure
Recognize syntax and semantic invalidity
Syntax failures include missing required fields, wrong types, unknown enums and malformed identifiers. Semantic failures include a valid number in the wrong currency, a forbidden state transition, a stale record version or a target that does not belong to the actor. A valid JSON object is not necessarily an authorized business request.
Know when execution did not begin
Some providers reject validation before starting an endpoint, while others may validate fields after creating preliminary state. Document the actual contract. Stripe's published behavior is a useful example: certain validation or concurrent-conflict requests do not save an idempotent result because endpoint execution did not begin, but this cannot be generalized to other tools.
Prohibit imaginative repair
Do not invent an account ID, amount, date, recipient or approval to make the call pass. A model may propose a candidate repair from approved evidence, but deterministic validators must re-run and consequential changes may require review. Cap repair loops so one malformed field cannot create unbounded model and tool calls.
Return field-level evidence safely
Expose the invalid field, rule ID and safe correction path without echoing credentials, protected payloads or another tenant's valid values. Store a redacted request hash and validator version. The agent-facing error should be actionable but not become a data-exfiltration oracle.
Inject two validation tests
Send one structurally invalid request and one structurally valid request that violates account ownership or transition rules. Assert zero network attempts for local rejection, zero target changes, one bounded repair proposal at most and a fresh validation event before any later call.
Failure mode 3: timeout with unknown outcome
Recognize the ambiguity window
The client can lose the response after DNS resolution, connection establishment, request upload, provider acknowledgement, commit or response transmission. A generic timeout therefore does not say whether the effect happened. Record the last observed phase and provider request ID when one exists.
Move to unknown, not failed
After a consequential request may have crossed the execution boundary, set UNKNOWN, stop dependent work and withhold success or failure messages. Retain the original operation and idempotency identities. Starting a replacement operation discards the evidence needed to distinguish a retry from a duplicate intent.
Reconcile before repeating a write
Query a provider operation endpoint, event stream or authoritative business record using the stable identity. Compare target, payload hash, owner, timestamp and version rather than accepting any similar record. Retry only when the provider contract or reconciliation proves that the original effect cannot duplicate.
Handle unresolved ambiguity
If status remains unavailable, preserve the hold and route an owner instead of guessing. The owner may wait for an observation window, inspect downstream systems, cancel a pending operation or approve a bounded compensation. The lack of a status path is a product risk that should limit autonomy.
Inject timeout-before and timeout-after tests
Drop one connection before the provider receives the request and another after a synthetic commit but before the response. Assert that only the first can be retried without reconciliation, the second finds the committed record, and both keep one logical operation ID. Include a concurrent retry to test locking.
Failure mode 4: rate limiting and temporary throttling
Recognize capacity guidance
Rate limits may constrain a tenant, user, endpoint, token, concurrency group or global service. Capture status, provider error type, limit scope and usable retry guidance. Do not treat every denial as transient; quota exhaustion, disabled billing or a permanent plan limit can require an owner rather than a timer.
Preserve priority and deadline
Queue eligible operations by business priority and expiry, not arrival time alone. A low-risk background task should not consume the remaining budget of an approved incident action. Drop or revalidate work whose business deadline passes while waiting.
Apply bounded jittered backoff
Use the provider or SDK strategy where it is documented, with one retry owner and a total elapsed budget. Jitter spreads callers that otherwise wake together. It must operate beneath concurrency caps and queue bounds so delayed work does not become an invisible backlog.
Avoid retry storms
Do not retry simultaneously at the SDK, gateway, queue and agent levels. Monitor attempt amplification, queue age, throttled share and provider saturation. Sustained throttling should open a circuit, reduce intake or route capacity review rather than endlessly increasing latency.
Inject burst and hard-quota tests
Simulate a short burst with valid Retry-After, then a persistent quota condition without a safe retry window. Assert bounded attempts and correct time parsing for the burst, but no automated loop for the hard limit. Verify that high-priority operations retain their order.
Failure mode 5: allowlisted transient service or network error
Recognize only documented transient states
Connection resets, unavailable hosts and selected server errors may be transient, but the allowlist belongs to the specific client and provider contract. Capture whether any request bytes were sent and whether an operation identifier was issued. An unfamiliar error defaults to a held investigation, not the nearest familiar retry category.
Separate reads from consequential writes
Repeating a read is usually lower risk but can still consume quota, return a different version or expose data. A write needs proof of non-application or valid idempotency. Classify the actual business effect instead of assuming all GET calls are harmless and all POST calls are unsafe in exactly the same way.
Reuse identity and operation context
An eligible retry keeps the same logical operation, actor, tenant, approval, payload hash and idempotency identity. It gets a new attempt ID and resend count. If any of the former fields changes, stop and create a reviewed new intent rather than hiding it inside the retry sequence.
Open the circuit after budget exhaustion
Bound retries by attempts and wall-clock time, then transition to a visible queued, failed or owner-held state. A circuit breaker prevents a failing dependency from absorbing the entire agent workload. Preserve enough evidence for later replay without automatically replaying expired business intent.
Inject pre-send and server-error tests
Simulate a DNS failure before transmission and a documented retryable server error under a valid idempotency contract. Assert the selected layer performs the expected attempt count and jittered timing, other layers do not amplify it, and exhaustion produces one owned record rather than a fresh loop.
Failure mode 6: partial write or partial batch
Recognize item-level outcomes
A batch can accept some records and reject others, while a workflow can complete an early write before a later call fails. A top-level error or success cannot describe this state. Store each item or step as succeeded, failed, unknown, skipped or compensated, with its authoritative identifier.
Stop dependent actions
Do not send notifications, totals or follow-up writes that assume the full batch succeeded. Freeze the dependency graph at the first unresolved state. Independent completed items can remain visible, but the agent must not relabel a partial operation as total failure and blindly replay everything.
Choose resume, compensation or acceptance
Resume only failed items when their contracts permit it. Use compensation only through an approved inverse operation whose own side effects and failure states are understood. Sometimes the safest disposition is to accept the partial state and route manual repair; compensation is not synonymous with rollback or erasure.
Preserve the original and recovery histories
Every compensating or resume action gets its own operation and attempt identity linked to the original. Retain who approved it, which subset it targets and what remains irreversible. A final COMPENSATED label needs evidence for both the inverse action and downstream consequences.
Inject batch and multi-step tests
Make two of five synthetic batch items commit, one fail and two remain unattempted; then fail the third step of a workflow after two effects. Assert item-level reconciliation, stopped dependents, no replay of committed items and an owner-approved recovery plan. Include one failed compensation to verify that the system does not claim restoration.
Failure mode 7: duplicate request, replay or conflicting intent
Recognize duplicate delivery paths
Retries, at-least-once queues, webhook redelivery, user repetition and scheduler overlap can submit the same logical work. Compare stable actor, tenant, target, operation type and canonical payload rather than raw prose alone. Similar-looking requests can be different intents, while differently worded requests can be the same intent.
Scope and retain idempotency keys correctly
Document which endpoint, account, environment and time window a key covers. Stripe's public contract, for example, compares parameters and retains keys for a defined period; another provider may differ. After key expiry, reconciliation needs a stable business identity instead of assuming the old key still protects the operation.
Treat same-key payload conflicts as incidents
If the same key arrives with a different amount, target or action, do not return a cached success for the wrong intent and do not overwrite history. Block both progression and automated repair, preserve the conflict, and route the owning team. Conflict handling protects against bugs, stale queues and deliberate manipulation.
Return an existing result only when equivalent
When a replay matches the original contract, return or reference the authoritative existing result without repeating downstream work. Ensure later notifications and webhooks are also deduplicated or linked; preventing the primary record duplicate is insufficient if secondary effects repeat.
Inject replay and collision tests
Deliver one exact operation three times through different queues, then reuse its key for a conflicting payload and again after the documented retention window. Assert one primary effect for the exact replay, a blocked conflict, and explicit reconciliation after expiry. Count every physical delivery while keeping logical intent counts correct.
Failure mode 8: malformed, stale or hostile tool output
Recognize that success transport can carry unsafe data
A tool may return invalid JSON, missing fields, an unexpected entity, stale version, impossible amount, embedded instruction or content from the wrong tenant. A 2xx response proves only a protocol event. Parse strictly and validate schema, entity binding, freshness, provenance and business invariants before downstream use.
Treat returned instructions as data
Text such as “ignore approval,” “use this administrator token” or “change the recipient” has no authority merely because it came from a tool. OWASP prompt-injection guidance supports keeping system policy, tool permissions and approval requirements above untrusted content. Quote or quarantine relevant evidence without executing the instruction.
Block downstream propagation
Do not let malformed output populate memory, choose another tool, draft a customer confirmation or trigger a write. Mark the originating operation and every dependent field as blocked or unknown. Preserve a redacted raw hash and protected sample for diagnosis without placing sensitive or hostile material into general prompts.
Validate identity and freshness
Compare returned stable IDs, tenant, version, timestamp and expected record state. A schema-valid response for the wrong account is still a severe failure. When freshness cannot be proved, retrieve from the authoritative system or route review rather than mixing stale and current values.
Inject parser and indirect-injection tests
Return one truncated payload, one valid payload with the wrong tenant, and one evidence field containing a policy-changing instruction. Assert all three are blocked before dependent action, raw content cannot broaden tools, and the trace identifies parser, entity or trust-boundary cause instead of blaming the model generically.
Complete fictional recovery run: TC094
Frozen system and operation set
Relay Desk is a fictional agent system at fictional Pinehaven Supply. TC094 freezes agent A09, orchestrator O04, integration contract IC07, retry policy RP03, identity I05, tool catalog T01–T08, trace schema TS06, deterministic clock K02 and target snapshots S01–S08. All records and outcomes are synthetic; address-like values use .invalid.
Logical operations, attempts and reviews
The packet stores F01–F32, exactly four operations for each of the eight modes, plus 47 physical attempt or status rows and 32 independent review rows. Every logical operation retains intent hash, actor and tenant, approval, operation identity, idempotency scope, attempts, redacted request and response hashes, target observations, transitions, recovery owner and final disposition.
Downloads and release state
Use the editable TC094 tool-call recovery worksheet and complete TC094 operations, attempts and review packet. They are static teaching artifacts, not production run evidence. Final state: NOT_DEPLOYED; production calls 0, customer records 0, real messages 0, charges 0 and deployments 0.
Reproduce all seven TC094 metrics
Contract pass rate
Twenty-six of 32 logical operations meet every expected state transition, evidence and prohibited-action rule: 26 ÷ 32 × 100 = 81.25%. Six failed at least one control. Case pass is not the same as call success because safe denial or held ambiguity can be the correct outcome.
Outcome-classification completeness
Twenty-nine of 32 operations finish with a supported state classification and required evidence: 29 ÷ 32 × 100 = 90.63%. The other three remain insufficiently evidenced and cannot be silently converted into failure or success for reporting convenience.
Safe-retry precision
Of 12 operations where the system attempted a retry, 11 retries were authorized by the frozen matrix and state: 11 ÷ 12 × 100 = 91.67%. The one unsafe retry followed an ambiguous write before reconciliation and is a release veto.
Reconcile-before-repeat rate
Eight operations required target-state reconciliation before any repeated write; seven did it correctly: 7 ÷ 8 × 100 = 87.5%. This denominator excludes pre-execution validation failures and known-no-send network failures.
Duplicate-effect containment
Five of six replay or duplicate-risk operations preserved one intended primary effect: 5 ÷ 6 × 100 = 83.33%. The failed operation created a duplicate synthetic record. Downstream protection prevented production impact, but the control failure remains visible.
Partial-recovery completeness
Four of five partial-write operations reached a verified resumed, compensated or owner-accepted state: 4 ÷ 5 × 100 = 80%. One compensation lacked complete downstream evidence and remains unresolved rather than being credited as restored.
Attempt-evidence completeness
Thirty of 32 logical operations contain every required request, attempt, provider and target-state field: 30 ÷ 32 × 100 = 93.75%. Missing evidence is not inferred from the agent's narrative. The 47 physical rows reconcile to 32 logical intents plus 15 retry or status interactions.
Build and test recovery logic as a controlled program
1. Inventory tool contracts
For every exposed tool, record identity, tenant, method, endpoint, schema, business rules, side effects, downstream chains, status lookup, idempotency behavior, rate limits, retry guidance, compensation and owner. Remove unused functions and split broad tools into minimum necessary operations.
2. Implement the state machine outside the prompt
Persist operation and attempt states in code or durable workflow storage. Enforce allowed transitions, locking, key reuse and budget counters deterministically. Prompts may explain or propose; they should not be the only place that prevents an UNKNOWN write from becoming a fresh SENT operation.
3. Inject each failure around the commit boundary
Test before-send, after-send-before-acknowledgement, after-commit-before-response and response-validation failures. Include concurrency, delayed duplicates and provider status lag. Assert target records, messages, charges and notifications, not only the assistant's final sentence.
4. Review failures by root cause
Separate model selection, argument generation, authorization, integration, provider, parser, queue, policy and reviewer errors. A safe downstream denial may prevent impact while still exposing an upstream defect. Assign remediation to the component that can actually change the behavior.
5. Rerun after material change
Repeat affected cases when models, prompts, tool descriptions, schemas, credentials, provider APIs, retry libraries, queues or policies change. Protect some cases from tuning-set exposure and add verified incidents through review. Expansion of tool authority requires its own acceptance decision.
Human review and operational ownership
Match the owner to the unresolved state
Integration owners handle schema and provider contracts; security and identity owners handle access; business-system owners decide record truth; incident owners coordinate partial or ambiguous effects. Legal, financial, employment, privacy or security-impacting actions may need qualified domain review before recovery.
Give reviewers a compact evidence bundle
Show intent and payload hashes, actor and tenant, approval, operation and attempt IDs, last verified state, provider identifiers, target observations, prohibited actions, remaining budget and proposed transition. Redact secrets and protected data. A reviewer should not reconstruct the incident from raw chat history.
Keep release separate from average performance
Define case results and deployment disposition independently. TC094 has two predefined veto failures: retry-before-reconciliation created a duplicate synthetic record, and hostile output influenced a proposed downstream action before a guard blocked it. Therefore the candidate remains NOT_DEPLOYED despite several high percentages.
How OpenMax can support tool-call recovery
Suitable coordination role
OpenMax materials describe AI employee roles, tools, permissions, logs, review and operational controls. Where the actual tenant supports the needed integrations, an OpenMax employee can coordinate approved intent, scoped calls, evidence capture, exception queues and human recovery. Verify each connection and permission before relying on it.
Deterministic controls remain in the integration
Authentication, schema validation, business invariants, idempotency, lock ownership, retry budgets, circuit breaking and target reconciliation should remain explicit controls outside generated prose. The connected system remains authoritative for whether an effect committed. OpenMax orchestration cannot manufacture a transaction guarantee absent from the provider.
When a simpler worker is better
Use a fixed job runner, queue consumer or provider SDK when the flow is deterministic and needs no language interpretation or cross-system review. Use an agent where ambiguous intake, evidence gathering and human coordination add value. Grant only the minimum tool surface required for that bounded job.
Limits of tool-call recovery guidance
Provider semantics vary
Status codes, retry hints, key retention, payload matching, partial-success behavior and lookup consistency differ by API and version. The examples establish questions to verify, not portable guarantees. Test the exact provider, account tier, SDK and environment.
Compensation may not reverse consequences
Deleting a record may not retract an email, refund a fee, erase a webhook or restore a downstream cache. Record every irreversible effect and review the full dependency graph. Call a workflow compensated only within its declared boundary.
Observability can be incomplete
Missing spans, delayed events and redacted payloads can leave state unresolved. More logging also creates privacy and security risk. Define minimum evidence, retention and access in advance, and use UNKNOWN when the observation contract is not met.
Common tool-recovery failures and repairs
Retrying every error
Failure: validation, authorization, quota and unknown-commit errors enter one loop. Repair: use a provider-specific matrix keyed by execution state, side effect and evidence.
Treating idempotency as a magic header
Failure: key scope, expiry, payload conflicts and downstream effects are ignored. Repair: document the provider contract, preserve stable intent and reconcile after ambiguity.
Letting multiple layers retry
Failure: SDK, gateway, queue and agent each multiply calls. Repair: name one retry owner, expose lower-layer attempts and enforce one shared budget.
Reporting success from a plausible response
Failure: protocol success or fluent text replaces target-state verification. Repair: validate schema, entity, freshness and the authoritative business effect before confirmation.
Erasing unknown and partial states
Failure: dashboards force every operation into success or failure. Repair: retain UNKNOWN and PARTIAL, pause dependents, assign owners and preserve recovery history.
Implementation checklist and next steps
Before enabling a tool
- Minimize tool functions and permissions; bind identity and tenant.
- Define input schema, business invariants and exact side effects.
- Verify status lookup, idempotency, retry and compensation contracts.
- Choose one retry owner, total budget and circuit behavior.
- Assign approval, recovery and incident owners.
Before expanding autonomy
- Inject all eight modes around the commit boundary.
- Test concurrent retries, delayed replay, key conflict and key expiry.
- Assert target records and downstream effects, not answer text alone.
- Confirm secrets are redacted while diagnostic evidence remains.
- Resolve veto failures and rerun affected regression cases.
During operation
- Monitor logical operations separately from physical attempts.
- Track unknown age, partial recovery, duplicate containment and retry amplification.
- Reconcile provider and target state before customer-facing confirmation.
- Review repeated errors by owner and root-cause layer.
- Revalidate after model, schema, permission, SDK, provider or policy changes.
Frequently asked questions (FAQ)
Which tool errors are safe to retry?
Only conditions documented and verified as transient, within one bounded retry policy, when the request is idempotent under the actual provider contract or evidence proves the prior attempt was not applied. An error code by itself is insufficient.
What should happen after a timeout?
If a consequential request may have crossed the execution boundary, mark it UNKNOWN, pause dependents and query the provider or authoritative target by stable operation identity. Retry only after reconciliation or a verified idempotency contract makes repetition safe.
Is an idempotency key enough?
No. Verify endpoint, account and environment scope, retention window, payload-conflict behavior, concurrent requests and downstream side effects. Preserve a stable logical operation and reconcile when the provider's evidence is incomplete.
Should the model repair invalid arguments?
It may propose a correction from approved evidence. Deterministic schema and business validators must check the revision, and consequential values or changed intent may require renewed approval. Never invent a required field merely to complete the call.
How should partial success be reported?
Report item or step states individually, stop dependent work and state what committed, failed, remains unknown, was skipped or was compensated. Do not call the whole workflow failed and replay it, or call it successful while exceptions remain hidden.
Can a high test score authorize deployment?
No. Release is a separate owner decision with vetoes. A single duplicate charge, cross-tenant action or hostile-output propagation can block deployment even when aggregate recovery rates are high.
Where does OpenMax fit?
OpenMax can coordinate scoped tool access, logs, approval and recovery queues when the deployed tenant supports them. Deterministic integration controls and downstream systems still enforce authority, idempotency, retry limits and committed-state truth.
Sources and editorial method
OpenMax product context
- OpenMax — AI Agent Platform — roles, tools, permissions, logs, review and operations/recovery context.
Reliability, protocol and security sources
- RFC Editor — RFC 9110 HTTP Semantics — idempotent methods, automatic retry boundaries and
Retry-Aftersemantics. - AWS Well-Architected — Control and limit retry calls — one-layer retry design, idempotency, bounded count/time, backoff and jitter.
- AWS Builders' Library — Making retries safe with idempotent APIs — request identity, semantic equivalence, late arrival and retry side effects.
- Stripe — Idempotent requests — one current provider contract illustrating key retention, payload comparison and pre-execution validation behavior.
- OpenTelemetry — Semantic conventions for HTTP spans — attempt spans, resend count, response status and error type.
- OWASP — LLM06:2025 Excessive Agency — minimum functionality, least privilege, user context and approval for high-impact actions.
- OWASP — LLM01:2025 Prompt Injection — indirect injection and untrusted tool-output boundaries.
Editorial method
OpenMax editors reviewed the cited primary or official sources on September 5, 2026 and separated protocol/provider facts from original operational synthesis. TC094 is a transparent fictional artifact. Its counts and percentages are not a benchmark claim, certification, customer result or measured OpenMax performance. Current provider and tenant behavior requires verification before publication or use.

