OpenAI Codex · Session storage

JSONL Is the Truth, SQLite Is the Mirror

You closed the terminal yesterday. Today the list is still there, and you can keep editing the chat. Those two facts look like one store. On disk they ride two tracks. The upper track appends line by line. The lower track only copies the cover. After a mid-run power cut, what you can pick up is always the lines that already cleared the gate.

Course goalAfter this lesson you can explain three things. First, why the session list and session resume don’t read the same disk. Second, why one append must let JSONL clear the gate before it projects into SQLite. Third, what you lose if power dies before the gate, after the gate, or halfway through a compact write.
Try it first · Write as it runs, then pull the plug
A session writes a few things, then you cut power where you choose
When to cut power
The gate is JSONL’s flush. Once it clears, resume can read that line. The projection can arrive late. It cannot jump the queue.
JSONL original0 lines through
SQLite coverEmpty card
flush barrierThe tape hasn’t reached the gate yet.
Resume reads the fileNo cut yet. Watch the tape move first.
The list reads the mirrorThe card copies title, cwd, and path — not the whole chat.
Logic trail · each animation step maps to a stretch of source
  1. Session hands the item to LiveThread; failure is log-onlysession/mod.rs L3753
  2. LiveThread hands the raw slice to the storelive_thread.rs L203
  3. The allowlist drops ephemeral EventMsg; exec markers always staypolicy.rs L9
  4. One JSON line plus a newline; write_all, then flushrecorder.rs L1968
  5. The gate wins first; only then does Paginated project thread_historylive_writer.rs L337
  6. Projection failure is warn-only; next time it resumes from the byte offsetlive_writer.rs L345
  7. Watch the filtered result, then stamp a literal metadata patchlive_thread.rs L212
  8. Resume decodes the file line by line; it does not rebuild history from the threads tablerecorder.rs L1009
  9. If the list has no DB or the DB errors, fall back to scanning the sessions directoryrecorder.rs L547
Hit Play. The session writes as it runs; your chosen spot pulls the plug.
Resume
List
Contract
Teaching demo: Line counts, titles, and cut points are teaching fixtures, so you can compare resume before and after the gate. Line numbers on the logic trail match openai/codex commit 4f39251a01.
Idea 1 · Append-only log as original, derived table as cover
What problem it solves

The list has to be fast. Resume has to be right. One file rarely does both. JSONL is cheap to append and jq can read it. Filter by workspace, pin, or archive, and it stops fitting. SQLite is good at those filters — and should not be where you rebuild model input on resume.

Two accidents that look opposite point at the same rule. Delete state_5.sqlite and the list goes empty, then grows back; the chat is still there. Hand-edit a title and cwd in the DB: after refresh it sometimes follows, sometimes snaps back. The mirror can be rebuilt, and it can be overwritten by the original. Lose the original and the mirror cannot save you.

What the idea is

Codex splits into two tracks. Session never touches the file itself. It hands an already-built item to the current LiveThread. No live handle, or append fails: the turn itself does not abort. The error only goes into the log.

Source:codex-rs/core/src/session/mod.rs lines 3753–3759

LiveThread first filters an observation copy by policy. What it hands the store is still the raw slice. The store runs the allowlist again. Streaming deltas, approvals, warnings — those ephemeral EventMsg never enter JSONL. Compacted, TurnContext, WorldState, SessionMeta always stay.

Write order is fixed. JSONL lands first, then the projection into SQLite. Projection can retry next time. If JSONL fails, SQLite does not get to jump ahead.

Session LiveThread JSONL rollout One item per line; it clears the gate only after flush SQLite mirror A threads row copies title, cwd, and path only Resume, fork, compact replay Read JSONL only; decode line by line, then rebuild List, search, pinned partitions Use the mirror; if the DB is down, fall back to scanning the directory
Teaching diagram: The same item is stamped onto the tape first, then copied onto the card. From here the two read paths split.
Why it lasts

Append writes and random lookups want different physical shapes. Bind them to one format and either the list slows down, or every append rewrites the whole document. Split them, and the write path can lock the original first, then repair the mirror. Rewrite in another language and the contract is still this sentence.

Idea 2 · The mirror may lag; it may not lead
What problem it solves

Write SQLite first and backfill JSONL later, and if the process dies between the two steps the list will show sessions you cannot open. The user sees a title, clicks in, and there is no matching line. That inconsistency is harder to debug than a briefly empty list.

What the idea is

In Paginated mode the comments call SQLite a rebuildable view. The flush barrier must win first. The projection may lag; it may not lead. Only after durable_write returns Ok may materialize_to_sqlite start. Projection errors are warn-only.

Source:codex-rs/thread-store/src/local/live_writer.rs lines 335–347

At the byte step: one JSON line plus a newline, write_all, then flush. That flush is tokio’s file buffer. The source never calls sync_all. Kill the process instantly and the last few lines may still sit in the kernel page cache. Next open pads a newline; a broken half-line counts toward parse_errors.

Source:codex-rs/rollout/src/recorder.rs lines 1968–1974

Resume, fork, and compact replay read JSONL only. The list prefers SQLite. Missing DB, open failure, or unfinished backfill: fall back to scanning ~/.codex/sessions/ and increment the stable metric codex.sqlite.fallback.count.

Source:codex-rs/rollout/src/recorder.rs lines 547–559

Timeline of one append write_all flush gate Project into SQLite Cut before the gate The last line may still be in the page cache; resume cannot see it Cut after the gate Resume can pick up this line; the list cover may lag one beat The barrier sits here; the projection may not cross it
Teaching timeline: Same write, different cut: the cut point decides which line resume can see.
Why it lasts

Anything rebuildable is allowed to drop. It is not allowed to jump the queue. Bind both sides into one transaction and a slow mirror blocks the original too. Lag allowed, lead forbidden — that is the generic contract of a log plus a derived table.

Idea 3 · The compact contract lives in the file
What problem it solves

Compact replaces a stretch of history. If Compacted, WorldState, and TurnContext live only in memory or only in SQLite, deleting the DB drops the window number and the baseline together. The user thinks the model forgot. The loader just never saw those three lines.

What the idea is

Compact edits in-memory history first, then lands Compacted, WorldState, TurnContext in that order. WorldState must follow the replacement history — it is the baseline of that new history.

Source:codex-rs/core/src/session/mod.rs lines 3417–3427

Resume reads it backwards. Scan from the tail. Hit a Compacted with replacement_history and cut the earlier suffix, clearing earlier TurnContext baselines. Then replay WorldState forward: a full snapshot resets the baseline; patches merge on top.

Source:codex-rs/core/src/session/rollout_reconstruction.rs lines 155–188

Those three types are almost useless to the list. apply_rollout_item no-ops on Compacted and WorldState. The mirror is not a full-text index. It is the fields the list and filters need. Titles come from UserMessage, not guessed from the model’s ResponseItem.

Source:codex-rs/state/src/extract.rs lines 14–34

The original lands first. The cover can be rebuilt.
Why it lasts

The resume contract is written into the file. Change the persistence shape and you change the resume contract. The repo-root AGENTS.md puts “resume a session from an existing rollout” on the breaking-change checklist. If the file is still there, the session can replay by contract.

Side-by-side · One history, three landing spots

DSH: one log, two backends

DeepSeek Harness’s persistence unit is the in-memory SessionEvent. JSONL and SQLite implement the same SessionPersistence seam. Swap the backend and you swap storage primitives, not log semantics. The header carries SESSION_FORMAT_VERSION = 0. Wrong version, or an unknown event not marked ignorable: refuse to parse. The error is SessionFormatUnsupportedError.

Silent truncation is harder to debug than an error, so DSH errors. Codex tries to open anyway. Unknown shapes fail in serde and count toward parse_errors; if any items remain it still tries to build a builder.

Source checked on both sides · 2026-08-22 · DSH · Session persistence

Claude Code: one JSONL, no session-mirror DB

The current session path is projects/<project>/<sessionId>.jsonl. Append is synchronous appendFileSync: one JSON line plus a newline, mode 0o600. The list uses getSessionFilesLite, reading file heads and tails, never going through SQLite.

A single-file read has a 50 MB cap. The comments say a session JSONL can grow to several GB, so the caller must bail first or it blows memory. Codex moves that scan cost into startup backfill. Both sides admit JSONL will grow: one refuses to load the whole file past the cap; the other has a backfill worker recopy the cover into the DB.

Source checked on both sides · 2026-08-22
Classroom Exercise
01

Cut power halfway through compact

A session has already written SessionMeta, one user message, and one assistant reply. Compact starts. Compacted has flushed; WorldState has not. Power dies there.

Walk three things: which stretch of history resume can pick up; whether the list-card title changes; and what the missing baseline line becomes on replay.

Takeaway:JSONL blocks lost history. SQLite blocks a slow list. Backfill blocks an empty mirror. Fallback blocks a lying mirror. Any layer can fail. Default: degrade the list, never degrade resume.