MarsDawn

Jess Ou's evals pipeline, and the one step in it that's still yours

In July 2026, LangChain published a new “What is an AI agent?” at the address where Harrison Chase's 2024 post “What is an agent?” used to live — this one written by Jess Ou, with a definition that's almost word for word his. Most of her post is about something his didn't cover: a whole pipeline for evaluating agents automatically. Her post is candid about where that pipeline still needs a person, and where it doesn't.

What the post argues

Ou's definition echoes Chase's closely:

“An AI agent is a system that uses a large language model to decide the control flow of an application.”

Control flow, again, just means which step runs next. From there she describes LangChain's Agent Development Lifecycle — build, test, deploy, monitor — and a layered approach to checking an agent's work without a person reading every run: online evals sample production traces (recorded logs of real runs) for regressions, offline evals run against curated datasets to catch a bad change before it ships, and “LLM-as-a-judge” scores a run's output against criteria a person defined up front, at a scale manual review can't match. She's direct about where a person still belongs in that pipeline:

“For sensitive or irreversible actions, we recommend human-in-the-loop controls that pause the agent for approval, edits, rejection, or clarification.”

She also has a line about judgment that doesn't scale away no matter how good the pipeline gets: “Do not outsource judgment you cannot evaluate. If you wouldn't recognize a correct answer, neither will the agent.” /reviewing-agent-plans/ already builds on that exact sentence — this note doesn't repeat that discussion. Ou doesn't mention MarsDawn anywhere in this post, and doesn't recommend any Markdown tool. She also never mentions Chase by name; what connects the two posts is that LangChain published hers in 2026 at the address his 2024 post used to occupy, with a nearly identical definition — an observation we're making, not one she makes.

Our reading, not Ou's

Ou's human-in-the-loop line is about gating specific actions before they run — pausing a write action for approval, the same idea Chip Huyen's read-only/write-action split points at from a different angle — not about a person reading a finished report after the fact. Read carefully, most of her pipeline is designed to remove a person from routine checking, not add one: online and offline evals and LLM-as-a-judge exist specifically so a team doesn't have to manually review every trace. That's not a criticism of the piece — it's her stated goal, and a reasonable one at production scale. But it means the review you do by hand — reading a plan or a report an agent hands you directly — is exactly the kind of check her pipeline is built to reduce, not replace. Her own judgment line puts a floor under that reduction: wherever you personally can't tell a right answer from a wrong one, you still have to read it for yourself.

Where MarsDawn helps, and where it doesn't

MarsDawn isn't an evals pipeline and has no AI model inside — it won't score a trace, run an LLM-as-a-judge pass, or decide which actions are sensitive enough to pause. It's built for the moment her pipeline still hands to a person: reading the thing directly. The Outline tab (View ▸ Show Sidebar, ⌃⌘S) lists a long report's headings, the source and rendered preview sit side by side (⌘2) with Mermaid and KaTeX drawn out, and Edit ▸ Copy Reference (⌥⌘C) with Copy for AI (⌃⌥⌘C) let you turn a spot check into precise feedback the agent can act on.

Try it

MarsDawn is on the Mac App Store. The free marsdawn command-line tool works today:

brew install redtear1115/tap/marsdawn

It exports Markdown to PDF without the app.

Command Line · Know before you buy: What MarsDawn doesn't do

Next

Sources