Opening
Who decides what a coding agent remembers?
An agent’s context is mostly tool output. When it fills up, a model rewrites the whole conversation as a summary. This film tests one idea: let a model that only makes decisions choose what stays, and build the harness around it.
Of 100 tokens in an average agent turn, 84 are tool output and 16 were written by the model.
That is JetBrains Research’s count for SWE-agent on 50 SWE-bench Lite tasks, published in 2025.
The deciding model is Jev, from TypeSafe AI. The question is whether a harness built on it would beat Claude Code and Codex, and why.
Start with the case against
Four facts the idea has to survive.
0 papers explaining how Jev works, as far as I could find. Its training method is named in the launch post and described nowhere. (TypeSafe launch post, 15 Sep 2026, and the innFactory model page)
24 trials are the best public result for a harness built around Jev: eight tasks, three runs each. The full 66-task run was stopped before any result was posted. (OpenAgents issue tracker, September 2026)
37.5% recall from a Jev-built handoff, in the one test I found on real sessions. Keeping the plain last 24,000 characters scored 48.1%. (kerpopule scorecard, 20 Sep 2026)
2.7 points of task success is all that managing context added at a 128K window. Claude Code now compacts at about 967K. (UMass Amherst and Zoom, 2026)
The idea is worth building only where it gets past these four.
Sources: TypeSafe, 15 Sep 2026; innFactory model page; OpenAgents issue 9535; OpenAgents issue 9558; kerpopule scorecard, 20 Sep 2026; UMass Amherst and Zoom, 2026; Claude Code docs, model config
Three jobs inside every harness
Write: An LLM produces text: plans, code, commands and, today, the compaction summary. Frontier models took 3 to 329 seconds per answer in TypeSafe’s tests.
Decide: Pick one answer from a known set: which model, which tool, allow or block, keep or drop. Jev answers in 70 to 500 milliseconds, with a probability on every option.
Act: Plain code runs the loop, the tools, the sandbox and the hooks. It is the only part that can enforce a rule.
Steve Kinney’s summary of the split: “the LLM writes, Jev decides, and code acts.” The rest of the film asks how much of a harness is the middle job.
Chapter 1: When the window fills
Both harnesses run until the window is nearly full
Claude Code, 1M-token window: compacts near 967K. Adjustable from 100K to 1M with /autocompact.
Codex, 272K-token window: compacts at 244.8K. 90% of the window. It can also compact before, during or after a turn.
Until then, every turn sends the whole history again.
Caching makes that re-read cheap: $0.20 per million tokens on Opus 5.5, against $4 uncached. It does not make the context any shorter.
Anthropic’s engineers call the cost of length context rot: the more tokens in the window, the less reliably a model recalls any one of them.
A long session spends most of its life carrying a very large context.
Sources: Claude Code docs, model config; openai/codex, via a study guide; Anthropic pricing; Anthropic, 29 Sep 2025
In Claude Code a summary replaces the conversation
What comes back, what is rewritten, and what is gone.
Kept or reloaded: The system prompt, the root CLAUDE.md and unscoped rules, auto memory, the plan, and a fresh git status.
Read again: Up to five files, most recently modified first. A file over 5,000 tokens returns as a path with no content. Invoked skills return, cut to 5,000 tokens each and 25,000 in all.
Rewritten as a summary: Your requests, key concepts, files examined with important snippets, errors and their fixes, pending tasks and current work.
Gone: Full tool outputs and intermediate reasoning. The skill listing. Path-scoped rules and nested CLAUDE.md files, until a matching file is read again.
Afterwards the model knows roughly what happened. It no longer holds the text.
Sources: Claude Code docs, context window
Codex writes a handoff note, or asks the server to
90% of the window is the trigger.
20,000 tokens of your messages survive a local compaction.
64,000 tokens survive a remote one, which returns an encrypted item.
10,000 tokens is the cap on every tool result before it enters context.
The local prompt asks for a handoff summary written for another model taking over.
Encrypted reasoning travels with every request, so the chain of thought survives without the server storing the conversation.
The mechanics differ. The move is the same: a model writes a shorter account of the session.
Sources: openai/codex, via a study guide; Codex config reference
The summary call was never the expensive part
$0.19 to re-read a 967K-token history from a warm cache on Opus 5.5, before the summary’s own output. (My arithmetic from list prices)
$3.87 for the same read once the cache has expired. On an API key that takes five idle minutes. (My arithmetic from list prices)
$14 is the average cost of one benchmark trial for Opus 5.5 in Claude Code: $4.7k over the board’s 330 trials. (Terminal-Bench 4.1)
7.2% of a task’s cost was the most that summary calls took in JetBrains’ tests. The least was 0.65%. (JetBrains Research, 2025)
Jev makes the call about a hundred times cheaper. In one test it cost $0.0014 where Anthropic’s compaction API cost $0.145.
That saves a percent or two. The money is in what the session carries afterwards and what it has to redo.
Sources: Anthropic pricing; Claude Code docs, prompt caching; Terminal-Bench 4.1 board; Capital and Compute; JetBrains Research, 2025; jev-compactor benchmark
A summary takes seconds, sometimes minutes
Seconds to compact one synthetic 12.7K-token session
jev-compactor: 0.35 s
Grok Build: 0.5 s
Codex CLI: 1.0 s
LangChain middleware: 10.4 s
Anthropic compaction API: 16.8 s
Gemini CLI: 16.8 s
OpenCode: 17.4 s
Speed is not unique to Jev. Codex’s own compaction took one second here.
In a second author’s test on a 27K-token session, a model-written summary took 24.7 to 117 seconds. Jev pruning took 0.44.
Auto-compaction fires when the window is full, which is usually mid-task. The agent waits.
Sources: jev-compactor benchmark; fast-jev-compaction-pi benchmark; Claude Code docs, prompt caching
A summary paraphrases. Agents need exact strings.
Exact commands kept: Jev alone 0 of 8; Jev tuned 8 of 8; Summary 0 of 8
Exact paths kept: Jev alone 0 of 17; Jev tuned 17 of 17; Summary 17 of 17
Deep error codes kept: Jev alone 0 of 3; Jev tuned 2 of 3; Summary 2 of 3
Memory questions right: Jev alone 3 of 8; Jev tuned 6 of 8; Summary 6 of 8
Tokens left: Jev alone 999; Jev tuned 5,171; Summary about 1,600
Seconds taken: Jev alone 0.44; Jev tuned 0.45; Summary 24.7 to 117
Tuned means the plugin was told to keep every command and the tail of every error. Left alone, Jev pruned all of them.
The tuned run matched the summary on memory questions while keeping three times the tokens. The summary’s error codes came back paraphrased.
The gain is exact text at speed. It is not better recall.
Sources: fast-jev-compaction-pi benchmark
Research finds a simple rule already matches summaries
Points of task success gained by managing context, SWE-bench Verified, four models averaged
32K: 35.7
64K: 15.9
96K: 5.5
128K window: 2.7
UMass Amherst and Zoom ran 176 settings. Neither summarising nor dropping old tool output won consistently, and the gain from either shrank as the window grew.
JetBrains found that hiding old tool output about halved cost against an unmanaged agent. Its solve rate was within a point of summarising in three of five setups and ahead in the other two.
That rule has one setting, and it did not transfer. Ten turns worked in SWE-agent. OpenHands needed 58.
So the bar for a smarter selector is a rule that costs nothing.
Sources: UMass Amherst and Zoom, 2026; JetBrains Research, 2025
Chapter 2: A model that only decides
Jev answers typed questions. It cannot write.
TypeSafe AI calls Jev a System One model, after Kahneman’s fast and slow thinking. It was released on 15 September 2026.
Choice: One option from a list of up to 255. Returns the pick, a probability for every option and a confidence value.
Score: A point on an ordered scale of 2 to 10 levels. Returns the probability-weighted mean and the full distribution.
Noul: A yes-or-no question. Returns one number between 0 and 1.
A request carries a state and a set of questions. Every question sees the same state, and all are answered in one parallel pass. Nothing is generated token by token.
Sources: Maxim AI explainer; TypeSafe, 15 Sep 2026
It is fast and nearly free per call
0.5s or less per request. TypeSafe measured 70 to 500 milliseconds, from laptops near its own servers.
$0.042 per million input tokens. Output is not charged.
28% of the time of an LLM call, in one recorded agent episode. (UIUC, Princeton and Google)
2.6% of that LLM call’s price. (Same study)
Vals AI checked the cost claim. On 400 claim-checking items Jev scored 390, level with three OpenAI models, at about 1/498th of GPT-6 Astra’s cost.
TypeSafe says its headline multiples sit at the high end of what users should expect, and that it cannot prove the price is not subsidised.
Sources: TypeSafe, 15 Sep 2026; UIUC, Princeton and Google, 2026; Vals AI, 6 Oct 2026
Its window is small and its judgement is uneven
Jev’s input limits, drawn against a 1M-token session
State plus the longest question: 32,000 tokens.
One whole request: 64,000.
Long-context option: 262,144. TypeSafe calls it slower, less accurate and not ready for production.
Its documented weak spots are literal reading, counting, dates, multi-step reasoning, and state that holds a lot of irrelevant content.
It can pick a wrong option with high confidence. Calibration holds across many answers, not for any single one.
On Vals AI’s claim-checking task its calibration error was the lowest of twelve systems. On a LegalBench slice it ranked last, with the highest error.
A session near compaction is the input it handles worst: large, and mostly irrelevant.
Sources: innFactory model page; Maxim AI explainer; Vals AI, 6 Oct 2026
What is not published
The method: Its training method, Reinforcement Learning for Calibrated Decisions, is named in the launch post. No paper, architecture or model size has been released.
The founder’s paper: I found none on Jev. Diogo Almeida’s launch essay is the primary source. It mentions coding agents only as work that still needs human oversight, and compaction not at all.
An outside estimate: Two LessWrong authors fitted benchmark scores and put it near 17 billion parameters, with a range from 3 billion to over 100 billion. They call that a rough guess. Across 14 benchmarks it averaged 46.8%, against 58.1% for GPT-5.5 answering without reasoning.
Where it runs: A hosted API on the US West Coast. Closed weights, no self-hosting, no EU region. Every call sends the text being judged to TypeSafe.
The compaction idea came from the community, within days of the launch. It is not in the founder’s writing.
Sources: TypeSafe, 15 Sep 2026; innFactory model page; Gould and Mann, 7 Oct 2026; fast-jev-compaction README; fast-jev-compaction-pi benchmark
Chapter 3: Where a harness decides
One loop, six decisions
Follow one turn through the harness.
Route: Which model takes this request? Today: you choose. Sub-agents can differ.
Equip: Which skill and which tools load? Today: the main model reads short descriptions.
Gate: May this tool call run? Today: rules, then a classifier or a sandbox.
Screen: Is there an injected instruction in this output? Today: the classifier never sees tool output.
Compact: Is it time to shrink the history? Today: a fixed token threshold.
Verify: Is the work checked before the agent stops? Today: the model says so. A Stop hook can refuse.
Each answer comes from a short list known in advance. That is the shape a deciding model is built for.
Sources: Steve Kinney, course notes; openai/codex, via a study guide; Anthropic, 25 Mar 2026
The labs already decide fast, with their own models
Claude Code’s auto mode is a decision in two stages.
8.5% of safe actions were flagged by stage one, a one-token yes or no.
0.4% were still blocked after stage two reasoned about them. (10,000 real actions)
17% of real overeager actions got through both stages. (52 cases)
93% of manual permission prompts are simply accepted by users.
Both stages run on Sonnet 4.6, and stage two runs only on what stage one flags, mostly from cache.
TechCrunch reported Vercel’s command-safety check running 5 to 18 times faster on Jev, and more accurately. The figure is from a Vercel engineer’s post in launch week.
A fast deciding step is not new. Jev changes its price, so it can run on every call.
Handing decisions to Jev works only above a fitted threshold
Points of task success lost when Jev decides for the agent
Whole action, no threshold: −64.3
Fixed threshold of 0.5: −21.5
Every tool choice, no threshold: −19.0
Threshold fitted on the agent’s own runs: 0
Researchers at UIUC, Princeton and Google tested eight decision points. Three passed as cheaper without losing success: tool choice, tool arguments and a trigger for reflection.
With the threshold fitted, cost fell 14.2% with no change in outcome across 500 tasks.
A raw Jev answer is not safe to act on. One above a threshold fitted to your own agent can be.
Sources: UIUC, Princeton and Google, 2026
On coding agents the measured gain is small
10.5% was the largest cost cut from routing steps between a strong and a fast model. The smallest was 7.0%.
+7.9 points of success with the pi harness on Terminal-Bench. The best case.
−6.0 points with SWE-agent. The worst case.
40.7% of an agent’s wrong attempts to stop were accepted when Jev was the only guard.
The authors add that the decision model could hardly tell read-only steps from the rest.
Their best agent did not use Jev. A 4-billion-parameter model trained on the agent’s own states solved 91.5% of shifted tasks, against 82.0% with Jev.
The lasting asset looks like your own trajectories and thresholds, more than any one hosted model.
Sources: UIUC, Princeton and Google, 2026
The safe pattern: code sets the floor, Jev can only raise it
Hard checks: fail closed. Regular expressions, allowlists and the sandbox set the minimum level of risk.
Jev’s judgement: fails open. A timeout or a low-confidence answer changes nothing.
Combined: max(floor, jev). Jev can raise the level above the floor. It can never lower it.
When to ask at all: known answers only. If the options cannot be listed before the call, use a model that writes.
Injected text can steer Jev too, so it cannot be a harness’s only safety layer.
Sources: Steve Kinney, course notes; Maxim AI explainer
Chapter 4: The harness on paper
Code runs the loop. Jev decides. The LLM only writes.
This is Coder One, the open-source harness OpenAgents built around Jev in September 2026.
Code: Checks out the repository, runs each command, and enforces limits on steps, time and output.
Jev: Scores candidate files and chunks of long output. Only the relevant ones reach the prompt.
LLM: Proposes one shell command, or says it has finished.
Jev: Checks each acceptance criterion when the LLM says finished.
Code: Rejects a false finish and names what is missing, or opens the pull request.
There is one Jev request per loop. Jev never runs or authorises anything.
A 6,000-token request costs about $0.00025, so thirty loops cost under a cent.
If Jev is unreachable, the loop falls back to a fixed view: the first search hits, and the head and tail of the output.
Sources: OpenAgents issue 9531
Decide at the door, before text enters the window
This part is my proposal. It follows from Jev’s limits and from how prompt caching works.
A fixed cap, today:
Claude Code returns up to 25,000 tokens from a file read and keeps about 30,000 characters of shell output inline.
Codex cuts every tool result to 10,000 tokens.
Whatever fits under the cap enters the window and is read again on every later turn.
A typed triage, proposed:
Jev scores each result against the goal: keep it whole, keep the head and a pointer, or file it to disk.
One tool result fits inside Jev’s 32,000-token limit. A whole session does not.
Nothing already cached changes, so there is no cache to rebuild.
Coder One already does a version of this. Its own audit found Jev’s main value was better evidence reaching the generator.
Sources: Claude Code tools reference, via a study guide; openai/codex, via a study guide; innFactory model page; Claude Code docs, prompt caching; OpenAgents issue 9531
Prune rarely and deeply. Never a little every turn.
The cache matches an exact prefix. Change one earlier message and everything after it is processed again.
Turns to break even = 24 × tokens after the cut ÷ tokens removed. Opus 5.5 list prices with the five-minute cache: $5.00 to write a million tokens, $0.20 to read them back. My arithmetic.
2.3 turns to pay back one deep prune, from 1M tokens down to 86K.
3,000 turns to pay back removing one 2K-token result that sits 250K tokens from the end.
Claude Code’s docs describe the same effect for old images. It removes them in batches, because each removal reprocesses the conversation from that point on.
With the one-hour cache the multiplier is 39, not 24.
So a fast selector should not be run over old history every turn. Its speed pays at the door, and at the rare deep prune.
Sources: Claude Code docs, prompt caching; Anthropic pricing
At compaction, keep exact text and drop whole tool calls
This is fast-jev-compaction, a Claude Code plugin with about 7,600 GitHub stars. Anthropic’s own guidance calls clearing old tool results one of the safest forms of compaction.
Pair: Every tool call is matched with its result.
Pin: The first message and the newest six are never candidates.
Abridge: The conversation is squeezed to fit 25,000 tokens. Each tool result becomes a short note, so Jev never reads the results it is judging.
Ask: Two questions per call. Should the call stay? Should its result stay word for word?
Apply: At 0.5 or above, keep. If only the call passes, keep the first 300 characters of its result. Otherwise remove both.
Your prompts and the model’s replies are never cut. If Jev fails or saves too little, Claude Code’s own summary runs instead. The README reports no benchmark.
The judge sees the session through a keyhole
967K tokens in a Claude Code session at the point it compacts.
25K is the most the plugin shows Jev, about 1/39 of it. The blue square is drawn to the same scale.
Jev’s own limit is 32,000 tokens for the state plus the longest question. The 262,144-token option is the one TypeSafe says is not ready for production.
So keep or drop is decided from the call, its neighbours and a note where the result was. Large state with irrelevant content is Jev’s documented weak spot.
Its answers also move. In one author’s test the same request gave keep probabilities up to 0.14 apart, and one message near the threshold flipped twice in nine runs.
This is the strongest technical reason to doubt whole-session pruning, and to prefer deciding at the door.
Sources: fast-jev-compaction README; innFactory model page; Maxim AI explainer; jev-compactor benchmark
Drop to disk, with a pointer. Never to nowhere.
Recall on 104 questions about seven real sessions, by what the next session was handed
Jev digest: 37.5%
Last 24,000 characters: 48.1%
No handoff, one search: 56.7%
Whole dialogue to a writer: 58.7%
Jev digest, one search: 68.3%
Last 24,000, one search: 68.3%
Whole dialogue, one search: 75.0%
Jev’s picks beat picks made by recency, 11 questions to 4. Its judgement was real.
The digest still lost, because every turn Jev did not mark as keep was clipped to 400 characters.
One search of the old session added 16 to 33 points to every handoff.
Getting text back mattered more than choosing it well. A pointer helps only if the agent follows it: in the UMass study, 56.3% of settings with a recall tool never used it.
Sources: kerpopule scorecard, 20 Sep 2026; UMass Amherst and Zoom, 2026
The same six points, redrawn
Route: Jev picks a model for a whole sub-task. Never mid-conversation, because each model has its own cache.
Equip: Jev shortlists skills and tools above a fitted threshold. Below it, the main model chooses.
Gate: Rules and the sandbox fail closed. Jev can only raise the level of risk.
Screen: Jev triages each tool result at the door: keep it, keep the head and a pointer, or file it to disk.
Compact: Rarely and deeply. Drop whole tool calls and keep the remaining text exact.
Verify: A Stop hook runs real tests. Jev’s view is a hint and never the gate.
When Jev is down or unsure, each step falls back to what the lab harnesses do now.
Most of this can be added to Claude Code with hooks. That matters in the next chapter.
Sources: Claude Code docs, prompt caching; Claude Code docs, hooks; Steve Kinney, course notes; UIUC, Princeton and Google, 2026; fast-jev-compaction README; OpenAgents issue 9531
Chapter 5: Would it win?
First, what winning would mean
Two frontier points sit on the public board. A challenger has to beat one of them.
Terminal-Bench 4.1: tasks resolved, against the cost of the whole run
Opus 5.5 in Claude Code: 65.8%, $4.7k
Fable 5.2 in Claude Code: 65.8%, $16.5k
Sonnet 5.5 in Claude Code: 63.9%, $7.3k
GPT-6.1 Sol in Codex: 60.9%, $0.6k
GPT-6 Astra in Codex: 60.3%, $3.3k
Kimi K3.1 Pro in Claude Code: 59.7%, $1.1k
Claude Code with Opus 5.5 leads on success: 65.8% of 66 tasks, for $4.7k.
Codex with GPT-6.1 Sol sits five points lower at about an eighth of the cost.
The board’s margins run about three points either way. A harness has to clear that to claim more solved tasks.
Sources: Terminal-Bench 4.1 board
The best direct evidence: same pass rate, lower cost, 24 trials
24 of 24 trials passed with Jev briefing Opus 5.5 at low effort. Eight tasks, three runs each.
63% cheaper than Opus 5.5 alone on four tasks, and 32% faster. The announcement said 61% and 37%.
54% of the Opus arm’s cost was one-hour cache writes. The five-minute cache cut cost 19 to 25%.
0 full 66-task results posted. Full runs were stopped on 23 September, pending a clear gain on single tests.
The winning arm also ran lean and at low effort, so the comparison does not isolate Jev.
Its later notes credit most savings to the lean configuration. Escalation rescued 0 of 12 tasks.
Jev and its host took 1 to 5 seconds. The model doing the writing took 95% of the time.
It shows a cheap front end can make a strong model do less. It does not show that a deciding model made the agent better.
Sources: OpenAgents issue 9535; OpenAgents issue 9558; OpenAgents announcement
On compaction itself: cheap and exact, on two synthetic sessions
jev-compactor: Saved 73%; Cost $0.0004; Facts kept 4 of 4
Anthropic compaction API: Saved 86%; Cost $0.043; Facts kept 3 of 4
Codex CLI: Saved 85%; Cost $0.049; Facts kept 3 of 4
OpenCode: Saved 85%; Cost $0.038; Facts kept 3 of 4
Gemini CLI: Saved 61%; Cost $0.083; Facts kept 4 of 4
Grok Build: Saved 74%; Cost $0.020; Facts kept 4 of 4
LangChain middleware: Saved 66%; Cost $0.013; Facts kept 1 of 4
Vercel pruneMessages: Saved 88%; Cost $0; Facts kept 3 of 4
Oldest-first truncation: Saved 53%; Cost $0; Facts kept 1 of 4
Jev kept all four planted facts at about a hundredth of a summary’s cost. It also saved fewer tokens than four of the six summaries.
A free rule, Vercel’s pruneMessages, saved 88% and kept three facts. Claude Code itself is not in the table.
Confidence that Jev pruning keeps exact text better than a summary: moderate. That it raises task success: unknown. Nobody has measured it.
Sources: jev-compactor benchmark
Where it could really win
Lower cost at the same success. Coder One’s 24 trials, a 14.2% saving from fitted cascades, and decisions that cost 2.6% of an LLM call. Wrong if a full 66-task run saves under 25% against the same model at the same effort and cache setting. Confidence: moderate.
Exact recall after compaction. Tuned verbatim pruning kept 8 of 8 commands and 17 of 17 paths. The summary kept none of the commands. Wrong if agents repeat earlier tool calls just as often after a pruned compaction as after a summary. Confidence: moderate.
Shorter stalls. Pruning took 0.35 to 0.6 seconds where most summaries took 10 to 117. Wrong if the host’s own compaction already takes about a second, as Codex’s did. Confidence: high against Claude Code, low against Codex.
Sources: OpenAgents issue 9535; UIUC, Princeton and Google, 2026; fast-jev-compaction-pi benchmark; jev-compactor benchmark
Where it would not
More tasks solved. Managing context added 2.7 points at 128K, and the gap was still shrinking as windows grew. Routing moved success between −6.0 and +7.9. The board’s margin is about three points. Wrong if a full run beats the same model in Claude Code by more than three points. Confidence: low that it wins here.
A lasting edge over the labs. Compaction, gating and stopping are hook points in both harnesses, and the Claude Code pieces already ship as plugins. Its gate is already a one-token decision. Wrong if the best results need a decision model trained on trajectories the labs do not have. Confidence: low that it wins here.
Sensitive or regulated code. Every decision sends conversation text to a hosted API on the US West Coast. Closed weights, no self-hosting, no EU region. Wrong if TypeSafe ships self-hosting or a regional endpoint. Confidence: low that it wins here.
Sources: UMass Amherst and Zoom, 2026; UIUC, Princeton and Google, 2026; Terminal-Bench 4.1 board; openai/codex, via a study guide; Anthropic, 25 Mar 2026; innFactory model page
My estimate
A well-built harness around a deciding model, against a lab harness running the same writing model, in a full public run.
Probability, as I judge it. A judgement, not a measurement.
Shorter compaction stalls than Claude Code: about 85%. Confidence: moderate to high. Against Codex there is little to gain.
A quarter cheaper than Claude Code: about 40%. At equal success. Confidence: low.
A quarter cheaper than Codex: about 20%. At equal success. Confidence: low.
Three points more tasks solved: about 15%. Than either harness. Confidence: low to moderate.
The cost case rests on slack. On the public board Claude Code with Opus 5.5 used 8.0 billion tokens. The two leading Codex runs used 1.5 billion each.
Much of that slack can be taken without Jev, by lowering effort or changing the cache setting.
Better on cost and waiting time is plausible. Better at solving tasks is not supported by anything published.
Sources: Terminal-Bench 4.1 board; OpenAgents issue 9535; UIUC, Princeton and Google, 2026; UMass Amherst and Zoom, 2026
Chapter 6: How to prove it
Hold the writing model fixed and change one thing
A: The baseline. Claude Code or Codex as shipped. Model, effort and cache setting stay fixed in every arm.
B: The free rule. Hide old tool output behind a tuned window.
C: Jev at compaction. Verbatim pruning in place of the summary.
D: Jev at the door. Typed triage of each tool result.
E: C and D together, with fallbacks on.
Arm B is the control most Jev benchmarks leave out. Without it, beating a summary proves little. Run all 66 Terminal-Bench tasks five times each: the 330 trials behind a row on the public board.
Sources: JetBrains Research, 2025; Terminal-Bench 4.1 board; Capital and Compute
Six numbers to record
Tasks solved: beyond ±3 points. The board’s own margin. Anything inside it is noise.
Cost per solved task: 25% lower. At the baseline’s effort and cache setting.
Rework after compaction: fewer repeats. Count tool calls that repeat one made before the compaction.
Stall time: seconds. Wall time the agent waits at each compaction.
Cache hit rate: no worse. Claude Code already reports cache reads and writes for every turn.
Fallback rate: publish it. How often Jev failed, timed out or saved too little.
Sources: Terminal-Bench 4.1 board; Claude Code docs, prompt caching; fast-jev-compaction README
What would change my mind
For: a full run that wins: A full-suite run where a deciding-model harness beats the same model in Claude Code by more than three points, or by 25% on cost at equal success, with the free-rule arm included.
For: a published method: A published method paper or open weights for Jev, so that its thresholds and failure modes can be studied and not just probed.
Against: the free rule ties: The free rule matches the Jev arms. Then the selector adds nothing that a window setting does not.
Against: the labs ship it: The labs ship typed triage or verbatim pruning as a built-in. Then there is a setting to turn on, and no harness to build.
If you build it
Start as a plugin. Replace the compaction step and add a Stop check inside Claude Code or Codex. The lab harness becomes your baseline for free.
Own the loop only if you must. Hooks reach compaction, tool calls, tool results and stopping. Build a new harness for what they cannot do, and say what that is.
Log every decision. State, question, answer, probability and outcome. Fitted thresholds, and later your own small model, are trained on those rows.
Keep Jev replaceable. Typed questions are an interface. In the one study that tried, a 4-billion-parameter model trained on the agent’s own states beat it.
Test against the free rule. Hiding old tool output behind a tuned window costs nothing. Beat that first.
Publish the full run. Nobody has. It is the cheapest way to become the reference.
Sources: Claude Code docs, hooks; UIUC, Princeton and Google, 2026; JetBrains Research, 2025; OpenAgents issue 9558
The short version
Jev is real, fast and cheap at typed decisions. Its method is unpublished and its judgement is uneven.
A harness has six decision points. Jev fits the ones with small state and a fixed set of answers.
Compaction matters less to task success than it feels. It matters to cost, waiting and exact recall.
The evidence today is the same pass rate at lower cost on 24 trials. Nothing published shows more tasks solved.
Build it as a plugin, test it against a free rule, and publish the full run.
Sources read on 14 November 2026. Estimates and confidence levels are editorial judgement. Every figure is linked under Sources.
Sources: OpenAgents issue 9535; UMass Amherst and Zoom, 2026; TypeSafe, 15 Sep 2026