The Open Process Durability Benchmark
For AI Coding Agents
Evaluating autonomous agent frameworks under physical host SIGKILL (137) fault injection across atomic state retention, torn-tail recovery, and token re-execution waste.
DCP-2.0 Agent Durability Leaderboard
Tested across 4 deterministic kill windows under 500 physical process terminations (SIGKILL 137). Real-world systems have trade-offs — we measure both recovery rate and persistence overhead.
| Agent Runtime & Reference Architecture | Durability Mechanism | Crash Recovery (Rcrash) | Resumption Latency (Tresume) | Duplicate Token Waste (Wtoken) | Per-Step Write Overhead | Proof / Audit Trail |
|---|---|---|---|---|---|---|
| LetItLoop (LIL v0.7.0) | Single-File Atomic WAL (LILWAL02) | 98.6% PASS | 14.2 ms | 2.8% (interrupted step only) | +3.8 ms (fsync journal) | HMAC-SHA256 Sealed |
| Temporal (Durable Workflows) | Distributed Event Sourcing (Cluster) | 99.2% PASS | 74.0 ms | 1.9% | +18.5 ms (gRPC roundtrip) | Cluster Event History |
| LangGraph (SQLite Saver) | Superstep Graph Checkpointing | 84.5% PARTIAL | 38.4 ms | 16.8% (re-executes active node) | +1.2 ms (SQLite transaction) | Database Row Logs |
| CrewAI | In-memory process queue | 0.0% LOSS | N/A (Full restart) | 100.0% (Total wipe) | 0.0 ms (Zero disk writes) | None |
| Microsoft AutoGen | In-memory ConversableAgent state | 0.0% LOSS | N/A (Full restart) | 100.0% (Total wipe) | 0.0 ms (Zero disk writes) | None |
| Raw Python Script (Unmanaged CLI) | Standard Python runtime globals | 0.0% LOSS | N/A (Full restart) | 100.0% (Total wipe) | 0.0 ms (Zero disk writes) | None |
- Why 100% durability is physically impossible: If a non-maskable SIGKILL strikes precisely during an in-flight, non-idempotent external HTTP/LLM API request before the response is committed to disk, that specific step must be reconciled upon resume, producing ~1.4%–2.8% token re-execution overhead.
- The Persistence Overhead Trade-off: LetItLoop's single-file WAL adds ~3.8ms disk fsync latency per step. For sub-millisecond CPU loops, persistence is unnecessary overhead; for multi-step LLM and tool pipelines costing $0.10 to $2.00 per step, paying 3.8ms disk I/O to guarantee zero lost progress under process termination is an overwhelming net win.
- Reproduce Locally: Run
pip install agent-durability-bench && python -m harness.runner --full-matrixto execute the physical OS subprocess fault injector on your own hardware.
The 4 Deterministic Kill Windows
DCP-2.0 injects host OS signals at the exact four failure boundaries of autonomous execution.
SENTINEL_PROMPT
Injected during LLM prompt compilation and context assembly. Validates supervisor memory isolation and dirty-bit state tracking.
SENTINEL_EXEC
Injected during subagent tool invocation and shell command execution. Tests orphan process group termination via PR_SET_PDEATHSIG.
SENTINEL_WRITE
Injected mid-write into the Write-Ahead Log. Asserts CRC32 torn-tail recovery and automatic truncation to the last valid frame prefix.
SENTINEL_VERIFY
Injected during AST verification and receipt generation. Validates file-lock release, dead-lock stealing, and receipt tamper detection.
Zero Mocking Harness
DCP-2.0 executes real Python subprocesses under host OS signals rather than simulated in-memory mock passes.
Cryptographic Proof
Every benchmark trial generates a machine-verifiable HMAC-SHA256 proof receipt recording AST scope diffs and timing receipts.