I’ve been experimenting with a different approach to code generation using a small local LLM.
The initial idea was simple: could a 4B model actually generate a usable Python application?
It could.
Well… sometimes. 
Instead of trying to make the model responsible for everything, I started moving decisions that can be formalized into a deterministic Python kernel.
LLM
- interprets the specification
- makes semantic decisions
- handles the parts that require reasoning
Deterministic kernel
- builds the known structure
- renders predictable code
- wires components together
- validates invariants
The result is Esus, an open-source experimental code-generation system.
The interesting question for me is not:
“How do we make a 4B model code like a much larger model?”
It’s:
“How much do we actually need the model to decide?”
I’ve written up the architecture, experiments and benchmark results here:
Devops agentique, et si le LLM ne devait pas tout décider ?
The project is open source:
GitHub - stengerg-sfeir/esus: Agent Smith · GitHub
I’d be interested in feedback from people working with small/local models:
Which parts of your agents do you think should remain probabilistic, and which parts should be deterministic?
One of my pipeline targets is essentially input sterilization for deterministic events.
Such as; math formula asked, feed it into a guaranteed calculator. Spelling errors likely present, run through guaranteed cleanup, and so on.
Many pipeline elements can be deterministic. So many that many systems can be turned into routed liaison snapped to calculators instead of full solvers. Not all by any stretch of the measure, but there are many. Many tool calls are a combination of determinism and nondeterminism, making them pretty unstable for smaller and weaker models, semi-stable for larger models. Quantization obliterates the stability in either case.
Well, technically your approach is kind of good one. Keep it up.
I think this is a practical way to get more out of smaller local models. Offloading math, validation, and other fixed tasks to deterministic tools can reduce unnecessary model decisions. It also gives the LLM more room to focus on the parts that actually need reasoning. I’d be curious to see how this holds up across more complex agent workflows.
Thanks! That’s actually one of the interesting things I’ve observed so far.
In my current benchmark, Esus makes more model calls than Claude for the same prompts (75 vs 35), but consumes far fewer tokens: about 55k vs 3.9M.
This is partly because Esus doesn’t keep accumulating context in one long conversation. Each model interaction is independent, and the deterministic kernel carries the state between steps instead.
So the idea is not necessarily to make the model do fewer things. It can actually make more, smaller decisions, while keeping the workflow state outside the model.
The complexity of the workflow doesn’t have to translate into an ever-growing model context: the deterministic code manages the state, while the LLM handles the decisions that actually require reasoning.
That’s also why I’m interested in testing more complex agent workflows. What would you consider a “more complex agent workflow” in this context? I’d be particularly interested in concrete examples where you think keeping the workflow state outside the model would become difficult.
Of course, I still need to test how well this generalizes beyond the current code-generation domain.
I’ve been arriving at a similar conclusion from the failure side rather than the model-size side.
One recurring problem is that an LLM can be perfectly reasonable semantically while being wrong about operational state — whether something actually ran, completed, changed state, or survived an interruption.
The useful boundary for me has increasingly looked something like:
LLM → interpret intent, reason, propose
deterministic layer → own state, validate invariants, execute bounded changes, verify
What I’m less certain about is where the boundary should sit for state mutation.
For example, would you allow the model to directly decide that an existing state should be replaced, or have it propose the mutation and make the deterministic layer decide whether that transition is legal?
I’m also curious how you’re treating interrupted execution. If the process dies halfway through a task and restarts, does the kernel have enough durable state to determine what actually happened without asking the model to reconstruct it?
That recovery boundary seems like a useful test of whether something truly belongs in the deterministic side of the system. -SS
This is a very interesting way of framing the boundary. I think I’m converging toward the second option you describe: the LLM should propose a mutation, but the deterministic layer should own the state transition and decide whether it is legal.
In Esus, this is already close to how I’m trying to handle business invariants: the model can interpret a requirement and propose a decision, but once the decision reaches the kernel, the kernel is responsible for applying and validating the deterministic rules.
Your point about interrupted execution is actually something I haven’t explored deeply enough yet. At the moment, Esus is much more focused on code generation than on long-running agent execution, so I don’t want to pretend that I already have a complete answer there.
But I agree with your proposed test: if a process is interrupted, the deterministic layer should be able to establish what actually happened from durable state, rather than asking the LLM to reconstruct it from context.
That suggests an interesting principle for the next iterations:
The LLM can propose a state transition, but it should never be the source of truth about whether that transition actually happened.
I hadn’t explicitly formulated the architecture this way before, so that’s a useful direction to explore.
What kind of interrupted workflow do you have in mind as a concrete test case? I’d be interested in trying to model one in Esus.
The open source small vertical LLM will be the trend in AI Agent era. Maase Inc developed the LingyanMiaoyu MoE LLM for industrial markting.
I think your point that interrupted execution is a different problem from the code-generation problem Esus is primarily solving is an important distinction.
Rather than designing a recovery system first, I’d probably just break Esus deliberately at several points during a real, nontrivial generation run and see what recovery actually costs.
Kill it:
- after an LLM decision
- after deterministic generation
- after validation
- after output has been written
- somewhere awkward between two stages
Then restart it and ask a few simple questions: what survived, what has to be rebuilt, what is ambiguous, and how much work gets thrown away?
If most of the pain disappears by persisting only the last verified stage and its artifacts, then you may only need a very light recovery layer.
If interruptions repeatedly force a large reconstruction of already-expensive work, then you’ve got a pretty concrete argument for building recovery alongside the generation process instead of adding it later.
So maybe the useful test is less “can Esus recover?” and more “how far can it fall before climbing back becomes expensive enough that a few handholds are worth building?” -SS
That’s an interesting experiment, and I think it points to a possible next step for Esus.
But I think I’m slightly earlier in the process at the moment. My main goal with Esus is still to make the generation itself as robust and deterministic as possible, rather than to build a resilient long-running agent.
Today, a generation is basically a bounded process: the goal is to go from a specification to a working application, with the deterministic kernel doing as much of the construction and validation as possible.
The recovery question becomes much more important once I move towards a more persistent agent, especially for brownfield development or iterative debugging.
So I wouldn’t build a recovery mechanism yet. But deliberately interrupting a generation and measuring what is lost could be a useful experiment later — especially to see whether the current pipeline already gives us natural recovery points through its intermediate artifacts.
For now, I’d rather spend that effort on making the generation itself harder to break.
And I think that’s probably the more interesting question for Esus at this stage.
Thanks for sharing. I agree that smaller, domain-specific models could be particularly interesting for agent architectures, especially when combined with deterministic tools and domain-specific validation.
That’s actually quite close to what I’m exploring with Esus, although my focus is currently more on the architecture around the model than on building a vertical model itself.