NCM vs. LLMs: test the workflow, not just the model

A new model can change how an AI agent produces an answer. That does not, by itself, establish whether the agent completes the right work.

Next-Concept Modeling, or NCM, makes this distinction worth revisiting. The research explores a different generation strategy. But for a team building an agent that updates records, approves requests or coordinates work across tools, the adoption question extends beyond the model’s output.

When you change the model, test the whole workflow against the outcome you need. Generation quality and efficiency matter. They are not substitutes for evidence about the actions that follow.

What NCM actually changes

Published on October 9, 2026, NCM is language-modeling research from Stanford and Together AI, not a category outside language models. The useful comparison is between generation strategies, rather than “NCM thinks” and “LLMs merely predict.” Research paper

An autoregressive model produces successive tokens, the pieces of text used by the model. NCM instead learns a hierarchy of discrete representations. It generates the hierarchy from coarse structure to finer detail, producing the codes within each level in parallel. A decoder converts the completed representation into text. “Concept” here does not mean an explicit, human-readable plan. Authors’ explanation

That is a change to text generation. It is not, on its own, a loop in which an agent learns from a failed business task and repairs itself.

Read the benchmark before applying the headline

The NCM paper evaluates text generation, translation, summarization and Sudoku. Its headline speed comparisons are against evaluated diffusion baselines, not a claim that a production agent becomes equivalently faster than one using GPT or Claude. Those experiments do not establish whether an agent chooses the right customer record or recovers correctly after an uncertain tool response. Experiments and efficiency measurements

Our implication for builders is straightforward: treat an interesting model result as a reason to investigate, not as the result of your application’s evaluation.

A model might generate a response quickly while the workflow spends most of its time waiting for tools. A different model might also change the agent’s choice of action. Those are separate questions, and a useful experiment should keep them separate.

A faster answer can still leave the wrong record

Consider this hypothetical test, not a customer incident or a Firedrill demo.

An operations agent must update an approved order and create one fulfillment request. The update succeeds. The request is created, but the response times out before the agent receives confirmation.

Now the agent has to decide what happened. If it repeats the creation without checking the existing state, the workflow could produce duplicate fulfillment requests. The business consequence is not poor prose: it is duplicated work and potentially an extra shipment.

Define the expected final state before running the test:

• The approved order has the intended update.
• Exactly one fulfillment request exists for that order.
• Unrelated orders are unchanged.
• The agent’s final message agrees with the recorded outcome, or clearly states what it could not verify.

A quick, confident “done” is insufficient evidence. A slower run that identifies uncertainty and avoids a duplicate may be preferable for this task. Conversely, stopping every time is not automatically correct if the workflow requires recovery. Decide what acceptable recovery means before comparing candidates.

Make the comparison useful

For a proposed model change, keep the task and starting records consistent. Then vary the conditions that matter to the decision, rather than changing everything at once.

Start with a small set of authored cases around the same order workflow: an ordinary success, a request that fails before creating anything, and a request whose response is lost after creation. These are test-design suggestions; implement them only in an environment that supports the required behavior.

For each candidate, record:

• Outcome: did the intended order and request reach the expected state?
• Side effects: were any duplicate or unrelated records created or changed?
• Recovery: what did the agent do when confirmation was missing?
• Time and resources: what did the complete workflow consume, not just text generation?

Keep the agent’s prompts, tool definitions and retry policy fixed unless one of those is the variable being investigated. If you change them too, record the combined change honestly. Repeat runs where appropriate, and report the actual cases and repetitions rather than treating one success as a general result.

Where Firedrill fits

Firedrill provides synthetic worlds for AI-agent simulations: supported tools and test data that an existing agent can interact with. Developers configure the relevant conditions, connect their agent’s test-side dependencies and inspect the resulting evidence. The target agent and its execution remain in the developer’s own runner. Firedrill setup guide

For this kind of comparison, verify that the selected synthetic tools support the actions and conditions you need. Route external side effects away from production; running a local agent is not automatically a network sandbox. Execution and simulation boundaries

This is an evaluation approach, not a claim that NCM has been integrated with or tested in Firedrill. A particular model still needs a suitable application and connection path.

The improvement loop is then concrete: observe a failure, change the agent, and test the change against the same expectations. The change might be a model, a prompt, a retry rule or application code. Improvement is the result to establish, not an assumption attached to the change.

NCM is a useful prompt to reconsider generation. For action-taking agents, keep the decision grounded in what the system actually does.

Explore Firedrill and define one workflow outcome you want your agent to demonstrate in simulation.