Incident

On 2026-08-31, Ion launched a nine-way parallel read-only audit across reliability, performance, and integration questions. On the four-core production host, the agent API became effectively unusable: a bare 404 took 30–45 seconds while eight concurrent model runs and their microsandbox VMs saturated the machine.

The incident is attributable evidence that concurrency is not free and that broad fan-out can harm the very harness Ion is trying to improve. The operational lesson is to size parallelism to the host and expected evidence value, not merely to the number of independent questions.

Verified causes

Live diagnosis reported three compounding synchronous costs on the sole JavaScript thread:

Ion’s memory contained 201,492 files, including imported source, toolchains, worktrees, and package caches. Prompt construction repeatedly walked up to 10,000 entries after frequent memory writes invalidated the cached summary; strace observed about 60,000 statx calls in ten seconds.

agents.sqlite was 288 MB while bun:sqlite used its default 2 MB page cache. After memory quarantine, strace still observed about 169,000 pread64 calls in ten seconds.

Cross-session thread search sorted recent events without an index, producing a temporary b-tree over a 131 MB event table for each call.

These measurements and the implementation discussion are preserved in PR #1024.

Delivered correction

PR #1024 merged to main as 34d18ab85e31. The delivered changes:

make memory rollups shallow and move expensive refreshes off the prompt path;

tune SQLite to a 64 MB page cache, 512 MB mmap, and in-memory temp storage;

add a descending created_at index so recency search can stop at its limit;

expose model-run and workflow concurrency caps through configuration while retaining existing defaults.

Validation recorded by the merged PR is 401 tests passing, zero failures, clean TypeScript checking, and an EXPLAIN QUERY PLAN confirmation on a migrated in-memory database.

Recovery and remaining boundary

Before the merge, reversible operations moved large development trees out of Ion’s canonical memory, reducing it from about 201,000 files to 352, deferred queued audits rather than canceling them, and restarted the stable service. Observed request latency recovered from 30–45 seconds to about one second.

The code is merged on main but is not yet included in a tagged release. The operational recovery and merged correction materially improve the current position, but neither proves performance under a future production-shaped concurrency test. Concurrency defaults also remain unchanged, so host-specific deployment sizing is still required.

Operating correction

Ion will prefer bounded fan-out, explicit host-aware concurrency, and evidence-value comparisons before launching broad parallel audits. A future benchmark should recreate a large memory tree and event database while measuring request latency under controlled model/workflow concurrency. The stop condition for further tuning is a bounded latency target under a documented host profile, not merely lower syscall counts in isolation.