Welcome back. In our first two primers on getting AI tools working in harmony and how to get started with generative AI, we covered how to talk to these systems and how to orchestrate them for maximum utility. But there is a question I get asked constantly: "Where exactly does my data go and how private is it?"
If you are pasting sensitive IP, financial data, or legal strategy into a public model (think Gemini, ChatGPT, Claude), you are trusting a commercial AI cloud provider. For many purposes, that’s fine. But what if you are prevented by law, client contracts, or just good old-fashioned wariness from uploading anything to the cloud?
There are two ways to solve this (and noting that in this primer I am focusing on solo professionals or very small businesses). This is also a slightly more advanced topic, so with that in mind, let's explore the options: 1. AI as a Managed Service A big player in AI as a Managed Service is Microsoft Copilot. After you sign a business agreement, the AI is "locked" to your business domain and any data you enter is promised to be kept within that domain. Copilot is now increasingly integrated with Word, Excel, and Teams.
The Catch: Your data still flows through their stack and is most likely going offshore. I'm writing this from my home base in Australia and sensitive health data for example cannot, by law, be sent offshore). You need to check your supplier contracts and see if you are happy with where the data is being sent. Bear in mind, your "domain data" may still be processed offshore and that will break many current privacy laws and agreements.
In addition Copilot may not run the latest and most powerful models (the so-called Frontier models) and the result is that at present Copilot has many devotees and detractors, so you need to do your own research on whether it will assist you and your business. There are other AI as a Managed service providers, but the same sort of strengths and weaknesses generally apply. 2. Running your own AI as Infrastructure There is an alternative: running the AI "brain" on your own machine. This provides you with:
- 100% Data Sovereignty: Nothing leaves your machine. Ever. (I have tested this: you can run local models totally offline for example with no problems).
- Control: You choose the model. You control the logs. You see everything.
- Specialisation: You can run models tuned specifically for coding, financial, or specific professional work.
The Catch: There is some complexity and need for very serious hardware.
The Hardware Reality Check
Running a "useful" model locally requires some serious muscle. By "useful," I mean a model that is snappy, smart enough to follow complex instructions, and doesn't hallucinate immediately or choke halfway through its output. Here are the major hardware options for 2026:
Option A: The PC Route
If you are on Windows or Linux, your ability to run AI depends almost entirely on your GPU (Graphics Card).
- PC memory (RAM): 32GB is the absolute minimum. 64GB is where you want to be.
- GPU: You need VRAM (Video Memory - at least 12GB and preferably 48GB).
Option B: The Mac Route
If you are buying a Mac, you have an interesting advantage: Unified Memory. Because Apple shares memory between the CPU and GPU, you don't need a massive dedicated graphics card (since that's already baked into the Apple processor). You just need a lot of RAM and a fast processor.
- Processor: M2 / M3 Pro family or later.
- Memory: 32GB minimum. But if you want to run the big "thinking" models, look for 64GB or more.
- If you are serious about this, hunt for a Mac with an M2/M3 Ultra chip and 128GB of RAM. It will run massive models quietly on a desk.
Getting started: The No-Code Setup
You do not need to be a Python developer or a command-line wizard to do this.
Step 1: Download LM Studio. This is the easiest entry point. It's a clean interface that lets you search for, download, and run models. It handles the heavy lifting.
Step 2: Choose your Model. Models come in sizes (parameters). Think of parameters as "brain cells." More is better, to a point. Going for a big model also slows the processing (so fewer output characters (tokens) per second. There's a trade-off unique to each machine's combination of processor, memory and GPU capability. LM Studio makes it pretty easy to test various models to find what works best for various circumstances.
- The Interns (Small Models): Look for Microsoft Phi or IBM Granite.
- The Experts (Large Models): Look for 70B (70 Billion Parameter) models.
Step 3: Check the Output! Local models are powerful, but they don't have the safety guardrails of ChatGPT nor the same power. They will try and do what you tell them but generally running on a single desktop PC is kind of like running the early releases of ChatGPT. So verify output.
A Personal Learning
I set up my local lab using an older RTX 8000 GPU I snagged on an eBay deal a couple of years back. Although it wasn't designed for AI (it was optimized for photography workstations) it actually performs well. It has enough VRAM to load a very capable model, allowing me to process sensitive strategic documents without a single byte leaving my office.
Models Currently Loaded onto My Machine (Jan 2026)
In my case, you can see I have opted for variety of halfway-house medium power models. I often use Qwen (a 30Billion parameter model from Alibaba) when I want a powerful local reasoning model (i.e. a "thinking" model) which can process image inputs (including PDFs) and also can call tools (a whole other topic). Nvidia nemotron is snappy on my machine and I'm also using it more often now.
Alternatively you can have your AI infrastructure hosted by Amazon or any number of virtual providers, but be aware they are nearly all offshore and cloud-based. I don't take that approach with my data since my machine can run what I currently need. But it's a future option.
The Verdict? If you are a solo professional, a consultant, or running a home lab, "AI as Infrastructure" is no longer a pipe dream. It requires an investment in hardware, yes. But the return is total privacy and total control.
If you have experiences running models locally to share please let me know what you think, and let me know of any inaccuracies in what I have shared here. Stay curious. #ai #aiinfrastructure #lmstudio #nvidia #microsoftcopilot Written by Paul Cooper, grammar tuned by Google Gemini.
Leave a Comment