The escape from Token Tyranny - Local LLMs.
I’ve been doing neural networks for more than 30 years.
I’ve been doing neural networks for more than 30 years. I’ve pushed custom classification models into production devices, poked and prodded every approach I could find to turn data into information — and maybe knowledge. So, when LLMs arrived, I had two questions: is attention really all you need? And can I build a stable baseline to test that — one that isn’t going to change under my feet every week?
Six months ago, when I started this, the question wasn’t which local LLM tools to use—it was why bother at all? The cloud APIs were cheap, releases were constant, and you could just drop in a credit card and go. But the landscape has shifted since then. Subscriptions are giving way to token billing, costs are climbing on capable models, downtimes and throttling are more frequent, and there’s growing uncertainty about what you’re actually getting on each call. The happy time is over, and it’s time to get to work on building a case for ROI, not just exploring what can be done at all.
And so, I’ve been working on all this for a while and discussing my findings and theories with a small group of people. But today LLMs and their associated systems have taken over the tech conversation landscape to the point where almost nothing isn’t touched by it. So, I thought that it would make sense to do some write-ups on what I’ve seen, what I’ve tried, and what I think I’ve learned. And hopefully this series can give you some insights into that and maybe even inspire you to tell me where you see the same, you think I may be misinterpreting what I see, or if I’m just dead wrong on some things. Whatever you see, hopefully it’ll let both of us learn something.
So on to what to expect. I’m going to do a series on what I’m working with - starting from what hardware I use for local work - I don’t use a single system. Why I ended up with each of the components and what functions they serve. Then I’ll go over the server side of things - what parts are being used to support my systems. That’s been a while coming together and I expect it could still evolve as new items get released. And finally, I’ll cover some of the clients/harnesses I’ve tried, and which ones work best for me. And somewhere in all this, I’ll likely touch on what models I’m using, but probably not as much as you think - simply because after a certain level the specific models matter less than how you host and use them. Things change SUPER rapidly there and there are already a lot of people doing model benchmarking, so any “this or that” model commentary already has a lot of coverage. What I will try to cover more is WHEN to use certain types of models and configurations. Why some people will swear by some models, while others just swear at them. Why is there such a divide between what individuals experience when using these things. There are a lot of accusations of “you’re not prompting it right,” but there seems to be a lot more involved than just that.
That’s why I won’t be covering those items as much - benchmarks, quantization comparisons, coding challenges (though I’ll dip into them when needed). There are plenty of people doing that - if there is a need for that, I’ll most likely just link you to what I go by unless there is something I actually want to dive into myself.
So, with that wordy intro out of the way… on to “You can build an AI lab from the things you find at home.”