Running LLM Inference Locally with a C/C++ Engine
A working setup around llama.cpp, the C/C++ inference engine, for running language models on hardware I actually own rather than renting tokens.
Most of the LLM tooling I touch assumes you are calling a remote API and paying per token. Marrow is my working copy of the other path: llama.cpp, the C/C++ engine that runs inference directly on local hardware. No API key, no per-call meter, just a binary and a model file.
Why a C/C++ Engine Matters
The appeal of llama.cpp is that it treats inference as a systems problem, not a cloud problem. The core (ggml) is written in C/C++, so it compiles down to something you can drop onto a laptop, a workstation, or a server without dragging a Python runtime and a stack of dependencies along with it. That portability is the whole point. When the engine is a single compiled artifact, you can reason about what it does, where it runs, and what it costs.
For my fleet, that changes the economics. Running models against subscriptions and owned hardware instead of pay-per-token APIs is the difference between an experiment you run once and infrastructure you leave on.
Keeping Up With a Moving Target
The interesting operational problem with a project like this is that it moves fast. The upstream repo tracks a stream of changes that actually matter to how you run it: models downloaded with -hf now land in the standard Hugging Face cache directory, so they can be shared with other HF tools instead of duplicated. Multimodal support landed in llama-server, which turns the inference binary into something that handles images, not just text. Native support for the gpt-oss model arrived with its MXFP4 format.
Each of those is small on its own, but together they mean the "right" way to run a model this month is not the way you ran it last month. The changelog for the libllama API and the separate changelog for the llama-server REST API are the two documents I check before upgrading, because a quiet API change is what breaks a running setup.
The Takeaway
The lesson I keep relearning is that owning the engine is worth the maintenance tax. You trade the convenience of a managed API for the responsibility of tracking a fast-moving codebase, but you get back predictable cost, local data, and the ability to run whatever model format the community ships next. For anything I plan to run continuously, that trade is easy. The engine that compiles to a binary is the one I can actually depend on.
Have something that needs building, or stabilizing?
These notes are the work log. The paid work runs through Moonshine Labs, my product and engineering studio. Tell us what you're building, or grab a call.
Work with us →