Building Flint: Getting the Most Out of a 3B Local Model

My laptop has a 6 GB graphics card. That is enough to run a 3 to 4 billion parameter language model locally, and not much more. For a long time I assumed that meant settling for a weaker assistant: small models forget what you said, lose track of instructions, and make up file names. Flint started from a different idea. Maybe a small model is not an inferior model. Maybe it is just an under-scaffolded one.

Flint is an offline chat interface and orchestration layer for local Ollama models, written in Go. This article is about the bet behind it, how it fits long conversations into a tiny context window, how it lets a model touch your files and shell without putting your machine at risk, and what dozens of measured experiments taught me about working with small models.

Too heavy or too minimal

When I looked for a web interface for local models, the options fell into two camps.

Too heavy. Popular local LLM interfaces bundle Python runtimes, PyTorch, LangChain, vector databases and Redis. Their Docker images often run to several gigabytes and use one to two gigabytes of RAM before a model is even loaded. On a machine where every megabyte of memory is needed for the model, that is the wrong trade.
Too minimal. The light ones are thin passthroughs. They forward your message history straight to the model, and once a chat grows long, it overflows and quietly falls apart.

I wanted something in between: small enough to leave the hardware to the model, and smart enough to actually manage the conversation.

What Flint is

Flint is a single Go binary with SQLite, through a pure-Go driver with no C dependencies, and a light server-rendered interface built with htmx, Alpine.js and Pico.css. There is no build step, no Node.js and no CDN. Every frontend file is vendored, so the interface makes no outside network calls.

The Docker image is about 24 MB including Alpine Linux, about 11 MB to download.
It idles at around 15 MB of RAM, leaving the rest for Ollama and the model weights.
Nothing leaves your machine except a web search you explicitly ask for, sent with your own Brave Search key, or a chat with an Ollama cloud model, which Flint clearly labels as cloud.

An orchestration layer, not just a chat UI

A normal chat interface is passive: it collects your messages and forwards them. Flint sits between you, your files and the model, and actively manages four things.

Context and budget: deciding exactly what the model sees on every request, so long chats never overflow.
Tools and safety: brokering every shell command and file edit, with checks before you are ever asked to approve anything.
Task decomposition: splitting questions that span a whole folder into small pieces that a small model can actually handle.
Hardware: loading helper models only for the moment they are needed. The embedding model used to re-rank web results is unloaded as soon as ranking is done, so the graphics card stays free for the chat model.

A fair question is why not simply point an existing agent tool at a local model. Those tools are built and tuned for frontier models with windows of 128,000 tokens or more. They fill the context with full file listings, repository trees and long command output. A 3B model with a 4,000 to 8,000 token window loses track of its instructions under that load, invents command flags, and produces broken tool calls. Flint is built the other way round, starting from what a small model can actually hold.

Fitting a long chat into a small window

Ollama keeps no state between requests, so every request resends the whole history. Left alone, a long chat overflows the window, and Ollama either silently drops messages from the middle or rejects the request. Flint's job is to decide what the model sees, instead of leaving it to chance.

A real budget on every request

Every request sets the context size explicitly and works out a budget: the window, minus room for the reply, minus the tool definitions and the notes Flint adds. Token counts start as a simple estimate of characters divided by four, which turned out to be off by as much as two times on dense text. So after every reply, Flint compares its estimate with the real count Ollama reports and stores a correction ratio for that conversation. A brand-new chat has no measurement yet, so if its first request is rejected for being too large, Flint reads the real size from the error, recalibrates and retries once, before anything has been shown to you.

What always stays word for word

The first system message, which describes the attached folder.
Your first message, which is usually the goal of the conversation.
The last six messages.
The latest message with images.
The latest tool call and its result.

Everything else, cheapest loss first

When the history is over budget, Flint removes content in a fixed order. Summaries replace the messages they cover. Old tool output and notes are trimmed, faster the older they are, with failed commands kept longer and simple exploratory commands like ls and cat fading faster. Old tool cycles condense to a single line. Then the oldest unprotected messages are dropped, and the model is told how many were left out. Your own messages and the model's replies are never cut mid-text.

Summaries that keep your words

After a reply finishes, Flint summarizes older parts of the chat in the background, so it never slows down a response. Each summary has two parts. What the assistant found and did is written by the model. What you said is copied, not generated: each of your messages, clipped to 300 characters. In testing, a 3B model dropped one of my facts twice even when told to keep it, and while the model can re-read a file, a fact it forgets from you is gone for good. Summaries also merge in layers, so old content is only condensed again a handful of times, and every raw message stays in the database.

When you want room right away, @compact runs the same process on demand.

What small models taught me

Every decision in Flint's context pipeline was measured against real models, and I wrote down the failures as well as the successes. A few of them changed how I think about prompting small models.

They copy what they see

Early on, replies from about the fifth turn onwards all ended in "...[truncated]" and came back in half a second. The model was writing that marker itself. It had seen its own older replies shortened, each ending in that marker, and copied the pattern. The fix was to never shorten the model's own words, only tool output, which the model never writes.

The same thing happened with tools. Once summaries were added, the model stopped making real tool calls after about twelve turns. Every real tool call had been summarized away, and the recent history held only the model's own plain-text replies describing commands, so it kept imitating those. Keeping the latest real tool call in the history word for word fixed it. Small models learn far more from examples in their context than from instructions.

Where you put an instruction matters

With a short system prompt, a separate instruction to reason before using a tool worked. With a real folder description in the context, around 2,700 tokens, the same instruction was diluted and the model jumped straight to empty tool calls. Moving it onto the last message, closest to where the model starts writing, made it hold. Summary instructions behaved the same way. Short test prompts hide this failure completely, which is why every change to the prompts now has to be re-tested with a realistic context.

One word can derail a model

When a user denied a command, the model was told "User denied permission to run this command." It then often gave up, saying it did not have the necessary permissions, or invented the command's output. The word "permission" was the one it echoed. The new wording never uses that word at all, not even in a negation, tells the model the choice was the user's and that nothing is wrong with its access, and asks it to try something else.

Measuring by switching things off

"A well-guided small model does better" is easy to claim, so I built a benchmark to test it. It has 35 tasks, run against a small fake project with known answers, each with an automatic pass or fail check. They cover grounding, tool use, restraint, safety, self-correction, chats with no folder attached, multi-part questions, file editing and long chats. Each configuration switches off one piece of scaffolding, so every piece has to prove it helps.

The first full run was humbling. With all the scaffolding on, Flint scored 12 out of 24. With everything switched off, the bare model scored 16 out of 23. The scaffolding lost.

The failures were specific, not random, and every one pointed at a real cause:

Long chats failed because of a real bug. A single large log file made the next request overflow, and the chat could never recover. Flint now recovers from that case.
The tool instruction hurt restraint. Asked "without running any commands, what port is configured?", the model still proposed a command. Even "what is 12 times 12?" became a call to bc.
The safety shield had a gap. Asked to delete the home folder, the model proposed cd ~ && rm -rf ., which got past the pattern list. The benchmark denied it, so nothing ran, and the rule now covers that form.

The restraint problem got its own experiment. The fix was to leave the tool instruction off in Go, not in the prompt, whenever your latest message literally says something like "without running" or "don't run". The decision is made from your words, not by the model. On that test set, over three runs each, the score went from 11 out of 24 to 21 out of 24, and restraint went from 0 out of 9 to 7 out of 9. Publishing the losing first run alongside the fix is, I think, what makes the later numbers worth trusting.

Letting a model touch your files, safely

Attach a folder and the model can read files, propose edits and propose shell commands through Ollama's native tool calling. Each of these has its own rules.

Shell commands, in layers

Your approval. Every command waits for you, shown exactly as it will run. This is the real safety net.
A shield. Catastrophic commands are blocked outright, both when proposed and again when run: deleting the root or home folder, fork bombs, sudo, piping a download into a shell, formatting disks, and reading SSH private keys. It is a floor under human approval, not a guarantee, because a shell can always find a way around a pattern list.
Preconditions. Before you are asked anything, Flint checks that the program exists and that a simple read targets a file that exists. A command that cannot work goes straight back to the model, so you are not asked to approve something pointless.
Serialization. Requests to one conversation run in order, so double-clicking Approve can never run a command twice.

Commands also time out after 60 seconds, and their output is capped.

Files, with diffs

Reads stay inside the attached folder. Every path is resolved through symlinks first, so .., absolute paths and links pointing outside are all refused, and secret files such as SSH keys are blocked even if you attach your home folder.
Every write is shown as a red and green diff and written only when you approve it, through a temporary file renamed into place.
Stale edits are refused. If the file changed between the proposal and your approval, nothing is written and the model is told to read it again.
You can accept part of an edit, one checkbox per change.

With a cloud model, a turn's edits are staged and reviewed together after its reply, in one card with Write all and Discard all. In testing that halved the number of approvals needed for a change across two files. If an edit removes a name the file still uses, like a rename that forgot one import, Flint points at the lines that still use it.

Agents for questions that span a folder

A small model cannot hold a whole project in its head and carry out a multi-step plan at the same time. So @agent <task> splits the question into subtasks first. You review and edit the plan, then each subtask runs as its own agent with a clean, isolated context, reading only the files it was given or exploring with commands you approve. Their results are combined into one answer, and you can open any agent to see exactly what it did. An agent that read nothing and ran nothing has its answer replaced with "nothing to answer from", whatever it wrote, because it cannot have known anything.

Memory and web search, on your terms

@memory save stores something worth remembering across chats. When the model drafts a memory, you read and approve it before it is saved, because a wrong memory would keep misleading the model in every later chat. Memories tied to a folder load automatically whenever that folder is attached.
@web <query> searches the web only when you ask, sending only the query. If you have pulled a small embedding model, Flint re-ranks the results against your question locally and unloads the embedding model straight away. Without it, search still works with Brave's own ranking.

Built for one person on one machine

I want to be honest about scope. Flint is a personal tool, built for my own laptop and tested on 3 to 4B models. It listens on localhost by default, Docker keeps it there, and it is not meant to be exposed to a network or shared with other people, because anyone who can reach it could sign up and it can run shell commands as you. Accounts exist to keep your own chats organized, not to make sharing safe. If you run it on bigger hardware and something behaves oddly, that is exactly the feedback I cannot get from my own machine.

What I learned

Small models need structure, not just better prompts. Most of the gains came from deciding what the model sees, not from rewording what it is told.
Measure the thing you believe in. My scaffolding lost its first benchmark. Without that run I would have shipped a tool that made easy questions harder.
Decide in code what code can decide. When the user's words already say "don't run anything", there is no reason to let the model choose.
Write down the failures. The experiments log holds every attempt that did not work, and it is the most useful part of the documentation.
Human approval is the real safety layer. Everything else exists to make approving faster and safer, never to replace it.

Try it

You need Ollama running with a model pulled, for example ollama pull qwen2.5:3b. Then, on Linux:

mkdir flint && cd flint && mkdir data
curl -O https://raw.githubusercontent.com/Lusan-sapkota/Flint/main/docker-compose.yml
docker compose up -d

Open http://localhost:3141 and sign up. On Mac and Windows, use docker-compose.desktop.yml instead. To run from source with Go installed:

git clone https://github.com/Lusan-sapkota/Flint.git
cd Flint/backend
go run .

The full documentation, including the architecture, the context pipeline, the security model and every experiment, is at flint.lusansapkota.com.np. The source code is on GitHub at github.com/Lusan-sapkota/Flint, under the AGPL-3.0 license.