You can build an AI lab from the things you find at home.
Well, at least at my home...
Well, at least at my home… and with a few strategic purchases. (Hat tip to Bill Sutton - Do It Yourself - Bill Sutton) I’ve worked with lots of various hardware (designed a bit myself), and that’s where I’ll start. Hardware has always inspired some religious wars, and some valid criticism. It’s always tradeoffs and balance. Cost/Features. Capabilities/Battery/Heat. Reliability/Flexibility. All that was in mind as I set up my systems.
So - to start with, I normally work differently than others. I’ve seen all cloud based - you’re basically working on a Chromebook and everything is on a server somewhere. That’s quick and simple to set up. And you normally have little control over it. Easy to set up, easy to manage (just make sure the payment methods have enough on them to cover your usage), away you go. There’s a lot to say for that. But it usually doesn’t teach you a lot about what’s going on inside, and you often are paying more for that convenience factor.
I’ll be dipping into this at points, but it isn’t my focus here.
Then there’s the beefy personal computer guy. I’ve been this guy in the past. You get a nice, top of the line desktop or (these days) laptop and run everything there. You get to watch the processes directly; you can often put your hand on the case and FEEL the processing load by the fan speeds. I have in the past done that to determine data access back when spinning hard drives or floppies were a thing. There’s a lot to be said about this, especially if you’re wanting to learn all the ins and outs. The problem is that LLMs eat hardware for breakfast, lunch, and dinner. And not just any hardware, but the latest and most expensive hardware, in a market where Moore’s Law (or at least the effects - costs going down over time for the same capacity) has been suspended.
I have an aging ROG Strix gaming laptop that fits in at this point. 3070Ti mobile, 8Gb of VRAM and 32Gb of system RAM. Not a slouch, but not even really a mid-weight at this point.
There’s the Homelab person. VMs out the wazoo. Rackmounts. KVM switches. Heck some even have dedicated cooling vents for their server closets. I’ve done data center work - that’s a bit far for me. Don’t get me wrong, I’d love to have all that, but it takes time and attention to set up and maintain, and that’s not the focus of what I’m trying to do here.
In this category, I’m using the house Synology NAS which I’ve dropped some more RAM into. It’ll handle my databases for LLM support and any durable storage I’ll need. At some point this may go over to Unraid, but for now everything’s running well.
And finally, there’s the servers. Big, small, general purpose, dedicated. This is a lot of where I’ve been focusing my efforts on for getting started. You can run the LLM’s on things like my dev laptop above, but you’ll be competing with resources and walking a balancing act constantly. With a dedicated machine, you can avoid things like desktops eating into your VRAM and trim down what you’re running on the systems to limit deadlocks and bottlenecks. You can also wipe and test different things much easier if you need to, or you go down a path that isn’t working out. That’s one of the genuine benefits of the LLM servers being completely stateless.
Here’s where I have a few items.
The first is a Jetson Orin Nano Super. I got this as a follow on to my original Jetson Nano that I did vision work on several years back. 8Gb of VRAM and a moderate GPU, very low power draw, and built to drop into an embedded project. Price is (was?) a great deal for what you get. The only downside is that it needs custom packaging, and that 8Gb is shared memory so anything you do cuts into that. So, you’re going to be doing smaller stuff, but it’s a great way to get into that lower end space.
The second is my Strix Halo computer. Also known as an AMD Ryzen AI Max 395+ (you see why I stick with the code name), I picked up an early unit with 128Gb of shared memory. The nice thing about it is that it has a decent CPU (which I normally underutilize), a decent GPU (Radeon 8060S), and as a nice bonus an NPU which can handle things like Embeddings better than even the GPU. All on that shared memory system that can dynamically move things around as needed. It’s got some decent storage, and mine comes with an occulink interface, which I haven’t done anything with yet.
Third is a couple of old Intel NUCs with N100 processors. These are for trying out things that might need a dedicated always-on system like Hermes or other Agentic-type systems. They don’t need a lot of horsepower, but it is easier for me to work with them as a physical box than a container on the Synology.
And then there are some embedded items - I mention them here as you may see them later on for various monitoring or data gathering. Not directly related to my LLM work, but they sometimes are displays or inputs for those systems. I’m a big fan of the Espressif ESP32 series, but like many people, I’ve got some Raspberry Pis lying around of various generations.
And that should give an overview of what I’m going to be writing up. My dev box is meant to be where I set things up, and often where my test harnesses and UI will go. The Synology is data focused, files and databases - that’s where the RAG/Graph stuff gets stored and queried. Then the Strix and Nano get loaded up with server components to handle the main load and serve up various Generative tasks and other tasks that really can take advantage of their horsepower.
OK, with that out of the way, on to talking about what to run on all these beasts. I’ll move on to covering “The Server Software Stack.”