Can a home PC run useful local AI for software engineering?

2026-07-23T11:30:00.000Z

Can a reasonably powerful home PC run useful local AI for software engineering?

AI is changing software engineering so quickly that I wanted to understand local AI for myself rather than relying on YouTube videos, benchmark results or second-hand opinions.

There was also a practical reason. I burned through my monthly allocation of Codex tokens rather more quickly than expected so I wanted to see how close a local setup could get.

My home PC isn’t especially exotic: a Ryzen 7, 32 GB RAM and an NVIDIA GTX 980 Ti with just 6 GB of VRAM, a graphics card released back in 2015. In other words, this wasn’t a £5,000 AI workstation.

I experimented with 3 things:

• LM Studio to run local language models.
• Continue in VS Code for coding assistance.
• AnythingLLM to build a Retrieval-Augmented Generation (RAG) knowledge base from my own documents.

Here’s what surprised me:

It actually worked:

A ten-year-old GPU can genuinely run modern language models. That’s remarkable in and of itself.

My biggest takeaway was that the software wasn’t really the limiting factor. The architecture is already there; it’s the models that are still catching up on consumer hardware.

LM Studio worked, Continue worked, AnythingLLM worked.

The limitation was the capability of the 7–8 billion parameter models I could realistically run on a GPU with 6GB of VRAM.

RAG (Retrieval-Augmented Generation) was the biggest surprise.

I loaded my own LinkedIn posts into a local knowledge base and asked:

“What have I written about requirements?”

The answer was genuinely impressive. Rather than summarising a single article, it produced an excellent summary of the themes that appear across dozens of articles.

Asking technical questions about an STM32 reference manual was a different story. The retrieval worked well, but the smaller (7B) models often fell back to generic answers rather than synthesising detailed technical responses.

So what did I learn?

Local AI is already useful, and even a surprisingly modest PC can run it. Today’s smaller models still struggle with deeper technical reasoning, but the pace of improvement is extraordinary.

I won’t be replacing Codex or ChatGPT just yet.

But I’m glad I spent the time understanding how local AI actually works. I now have first-hand experience of language models, embeddings, vector databases and RAG, not from a presentation or a benchmark, but from building and using them.

I suspect this is only the beginning. As models become more efficient and consumer hardware becomes more capable, local AI will become increasingly practical. It’ll be interesting to see how quickly that happens, and especially how it changes the way businesses consume AI, particularly where sensitive intellectual property or proprietary information can’t be sent to a public cloud model.

Good engineers don’t just adopt new technology. They take it apart to understand where it works, where it doesn’t, and why. For me, that’s what this experiment was really about.