The Server Software Stack — What I Run Locally
The Server Software Stack — What I Run Locally
If you’ve been digging into local LLMs, you’ve probably noticed something: the server landscape is enormous. Everyone has their favorite, things move fast, and the documentation often assumes you already know which path to take. I’ve been running servers on metal — no Docker, no Proxmox — and I want to share what I’ve actually tried, what works, and what I’ve settled on.
One thing to note upfront: I prefer running “on the metal” versus Docker containers or Proxmox. That’s my preference, not a rule. There are benefits and drawbacks to each approach. If it works for you, it’s the right way to do it.
Also, while these are server-side tools (not primarily query clients), the line gets fuzzy. Many packages build on one another. Going over them here should help clear up what each one actually does.
ONNX
I’ll start with ONNX. I’ve been using it — or its proto-server — since before it really existed. My detector RF signal neural net was a classifier system originally written for Microsoft Cognitive Toolkit, which later rolled into ONNX and shares most of the same capabilities.
The key thing to understand is that ONNX is primarily a runtime library hosted by other programs to provide an API interface for a server. It’s more of a foundation than a standalone server, but it’s used enough in the local models space that it’s worth mentioning. You’ll find lots of tools that support it, and there are tools to convert model weights to this packaging format. It can be lossy in terms of capabilities though — ONNX prioritizes stability over aggressively adding new performance features.
Ollama
Ollama is likely the first server tool many people encounter. It can self-host local servers, with the caveat that it really wants that server to be localhost. It’s very commonly what ships with local client software because of the simplicity of packaging and running it.
That simplicity comes with a trade-off: there are limited options to optimize it. As an added layer on top of llama.cpp, performance will be lower simply because of that additional software layer. If you’ve installed any local client packages, you’ve likely had Ollama installed somewhere with a small model (4B-9B parameters normally) already downloaded. It’s a solid starting point, but if you want to really tweak things, you’ll want to move on to one of the next options. For getting started or needing something simple to distribute, though, it works great.
Microsoft AI Foundry Local
Microsoft AI Foundry Local is another one of those fuzzy cases. It’s built to run ONNX models while offering them over a standard interface that matches the Azure AI Foundry endpoints. There’s no real reason you can’t use it locally.
The problem is that if you want to run it on an actual local server and access it over the network, you’ll have to do some network games. It steadfastly insists on being on localhost and won’t listen on your LAN. That was mostly a dealbreaker for me, so I didn’t go much further with it. But if you’re interested in testing locally and then scaling to cloud hosting in Azure — or running most of your work locally but bursting to the cloud — it’s worth a look.
LiteRT-LM
LiteRT-LM is Google’s local platform for models. It’s heavily optimized for working with Google models like the Gemma family and is intended to be a very lightweight host.
It works natively with 0.0.0.0, so that’s a big step up from Foundry in terms of network accessibility. Do note that it is a pretty CLI-heavy tool, but you’ll likely be comfortable with that if you’re already getting into these tools. If you’re standardizing on just Gemma, definitely put this on your short list. Google is working hard on it to enable local resources for AI-driven answers.
llama.cpp
This is where things really started working well for me. llama.cpp is actually one of the foundational environments for running LLMs — you’ll find it underneath a lot of other servers.
You get a lot of control with it. It comes pre-compiled for many environments, and you’re also able to do custom builds without too much trouble. It isn’t super complex, but it is a fairly large project. There are a LOT of configuration options, though, and not all of them are intuitive. That’s also what keeps many people from trying it — there’s a good bit you need to know about your model to use it effectively. But that depth means you can really get the most out of your hardware and control exactly how the model is being used.
The biggest limitation: you’re locked to one model at a time. Which leads me to…
llama-swap
llama-swap isn’t a standalone package — it sits in front of llama.cpp. What it does is let you set up llama.cpp configurations for various models, then your client can call for whatever model it needs. llama-swap will spin up a llama.cpp process with the right model and config.
If you need an LLM, an embedder, and a re-ranker for your pipeline, you can set up llama-swap with all the configs and it’ll swap them around as needed while keeping your memory managed so you’re not trying to run them all at once. It’s pretty lightweight and simple to set up, and one nice feature is that it can monitor your config file and dynamically pick up edits. If you’re using llama.cpp, you should probably default to using llama-swap alongside it.
vLLM
vLLM is the other heavy hitter. It’s built for larger-scale LLM hosting — if you’re doing a private local server it may not be as useful, but it has better multi-user context management than llama.cpp.
The trade-off is that it’s really tuned for hosting a single model for high-volume prompt processing. It’s Linux only (with some experimental macOS support). There are models that are vLLM-specific — I personally use it for Qwen3-VL-Embedding, which seems to really want vLLM. It has a very active and vocal community, gets quick support for new models, and fast bug fixes. Depending on what your workload needs, this could be exactly what you’re looking for.
FastFlowLM
I mention FastFlowLM because it’s one of the few servers that support the AMD NPU — which I have on my Strix Halo server. With FastFlowLM I can run my embedder on the NPU and keep my GPU/CPU free for other tasks.
There are fewer models available right now, but the list is growing quickly. It’s also used for smaller models focused on tool usage. The thing to remember: the NPU is optimized for efficiency, not performance. It’s a great choice for scheduled tasks or anything that isn’t time-sensitive, or as a supporting player for a primary model running on CPU/GPU.
Lemonade Server
This is where I found my sweet spot. Lemonade Server is a project supported by AMD (but not limited to AMD hardware) that wraps the other servers in a management system. It gives you a simple, OpenAI-compatible endpoint that abstracts away which backend you’re using — letting you pick whatever’s appropriate for the task at hand without the complexity.
What I really like is that it’s simple to set up and manage, but you still have almost the full configuration options of running one of the servers raw. It runs on a wide variety of hardware and OSes and handles a growing number of backend servers. One option I haven’t used yet but is opening up new possibilities is cloud passthrough — so you can offload from local to cloud without changing your client configuration, just which model you call.
As of writing, it supports llama.cpp, whisper.cpp, Moonshine, stablediffusion.cpp, Kokoro, FastFlowLM (NPU), AMD Hybrid models (Windows only), vLLM, and the cloud passthrough option. New backends show up regularly, and each backend can be configured independently for which resources it uses. While there are people who push to wring every last instruction out of their performance, Lemonade’s default optimizations are often better than what I’d come up with without spending days or weeks benchmarking and tuning. The dev team is very responsive and the Discord community is active.
whisper.cpp
whisper.cpp isn’t an LLM server, but it’s incredibly useful and supported by Lemonade, so I want to cover it. You’ve probably heard references to talking to your computer — whisper is very often what makes that possible. It works with multiple languages and has various levels of resource usage and accuracy. You get to tune it to how you want it to run. If you want your computer to understand voice input, this should be high on your list.
Moonshine
Moonshine is new to me, but it’s supported in Lemonade so I have it. From what I understand, it’s focused on lighter-weight models and better multi-language support. It has a good following, but I haven’t used it deeply yet.
Kokoro
Kokoro handles the text-to-speech side — letting your computer talk back to you. There are a lot of opinions about which packages are best here. On the cloud side, ElevenLabs seems to be king. The local space hasn’t anointed a single best option yet, but Kokoro is included in my Lemonade setup, so I’m using it.
Voicebox
Voicebox is the other TTS server I use. I haven’t gone deep with it, but in the tests I’ve done I’ve been impressed with what it can produce. It’s worked the best when I want a specific voice for my interactions — like getting Orac from Blake’s 7 to give me announcements.
AnythingLLM
AnythingLLM helps people get going when they want both server and client together. I’ve used it pretty extensively in the past and still have it in my toolbox. It does have some things it really wants to do its own way, though.
The benefit is that it comes out of the box with Obsidian integration and its own knowledge base tools — you can build up your own information repositories without diving deep into RAG setup and all the complexities that come with it. Automations and custom skills are very accessible. If you want a local setup to test or learn with, and don’t want to go all in on hardware breakouts, you can’t really go wrong with AnythingLLM. The author is accessible and keeps an active Discord for discussion and Q&A.
LMServer
LMServer fits into a similar spot as AnythingLLM — server functionality plus client functionality in one package. I haven’t used it myself extensively, but it has a good reputation, especially in the Mac community. It seems solid with a very active user group for questions and advice.
Picking Your Path
There are a lot of options, and a lot of ways to make things work. Here’s how I’d suggest approaching it based on your situation:
Just getting started? Try Ollama or AnythingLLM. They’ll get you running fast without a lot of configuration.
Serious about local? My go-to is Lemonade Server on my Strix Halo + llama.cpp on my Jetson Orin Nano. Lemonade lets me pick the right backend for each task while keeping everything manageable.
Need GPU for high-volume single models? vLLM is your best bet.
Running AMD NPU hardware? FastFlowLM lets you leverage that for efficiency-heavy tasks like embeddings.
Want to test without hardware investments? AnythingLLM gives you a full local setup on a laptop.
Want a mix of the various servers without being tied to any one? Lemonade Server is your go-to here. The ability to have it manage your various back ends and juggle configurations is unmatched in my experience.
As I go through the rest of this series, you’ll see me come back to Lemonade Server and llama.cpp most often. But the nice thing about how I’ve set things up is that I’m never limited to just one approach. Pick what works for your situation, run it, and tweak as you learn.
Now on to the clients — which is a whole world in and of itself.