Biggly Guide to Local AI

1. What “Local AI” Actually Means

Normally, when you use AI (like ChatGPT), your messages go to someone else’s servers on the internet.

In contrast, Local AI means:

• The AI runs on your own computer
• No internet required (after setup)
• Your conversations stay private
• You control everything – but this simple guide is to help you, as there is a lot to understand!

Downside: your computer has to be strong enough.

2. Why Gaming PCs are Ideal (GPUs + VRAM)

AI models need a lot of memory.

That memory lives in the GPU (graphics card), and it’s called VRAM (Video Random Access Memory, like normal RAM but MUCH faster).

Rough guide:

If you have a gaming PC (probably 6-12GB VRAM), you’re in a good position to run AI.

Can I Run Local AI on My Laptop?

If you have a laptop, even a ‘gaming’ laptop, you are probably better off using one of the more private, generally uncensored cloud providers that Biggly Bot can work with, such as Open Router.

3. What an “AI Model” Is

A model is the AI brain, the file that does the thinking and replying bits. For local AI, it’s a big (2-60GB) file you download onto your computer. For online AI it’s the same thing but on a data-center somewhere.

The current generation are called “large language models”, or LLMs. Image-generation models use a ‘diffusion’ technique but this page will stick with LLMs.

Examples:

• ChatGPT, Claude, Grok etc are massive online LLM models you cannot possibly run locally.
• Llama – from Meta. Available from small (7B, you can run it yourself) to “too big for a local PC” (405B – these sizes will be explained soon!)
• Mistral
• Qwen
• Gemma Etc.

Think of it like:

The model = the knowledge + intelligence. All models have their own personality but you mostly control the personality via creating a ‘character’ to talk with.

Different models = different strengths. Some are really good for helping you write a book, some are best for writing software code, some are just fun to talk to.

Model Size:

Current LLMs are rarely worth using below 3B and generally you’ll have a much better time with 7B or bigger.

The B stands for billions of parameters.

At time of writing, some typical sizes (presuming GGUF files, see next section about that):

1-5B – Tiny, can be useful tools but for conversation it’s like talking to a distracted child. Not suitable for most things but OK for an AI puppy or something.

7-12B Sweet spot for local AI with a good GPU, as they’re intelligent enough to have fun with, small enough that your machine can run them at reasonable speed.

14-24B – Now you’re getting seriously intelligent, knows more about the world than you do and still runs fast on a higher-end local GPU

27-35B – Perfect for high-end local GPU. At time of writing, Gemma-3 27B is incredible for its size.

55-70B or bigger – this is about as big as you ever want to go locally, up to maybe 120B, but that will crawl, even with an RTX 3090. Realistically a 3090 will run a Q3 70B at the speed of someone typing rather slowly. I find it OK (it types faster than most of my friends on Whatsapp!) but most people find a 70B to be a bit slow for chatting with a character.

Anything more than a 120B and you really need a special rig with multiple GPUs or a commercial-grade GPU that won’t fit normal motherboards.

4. What a GGUF File Is

A GGUF file is just the model in a format your computer can run.

It’s like:

MP3 for music
MP4 for video
GGUF for AI models

You download these files and tell Biggly Bot which file you want to use, and it’s location. Biggly Bot will load it and then you can chat.

Big Benefit of GGUF files:

The big benefit of GGUF models is that they can run with most of the model using your GPU’s VRAM, while the rest of the model can be handled by your CPU/RAM. This allows you to run models a bit bigger than your GPU’s VRAM size, so a bit slower, but not as slow as purely CPU/RAM.

5. What “Quantization” Means (Q4, Q5, Q8, etc.)

AI models are huge.
Quantization makes them smaller so normal PCs can run them. The smaller they are, the more dumb they get, so I strongly suggest you don’t go below a Q4 if possible (but a Q3 for a large 50-70B can still be very good).

Examples:

If you don’t have much VRAM, you use Q4 or Q5. Note that Q6 seems to be a problematic size, often causing errors that other sizes don’t, and it’s not notably smarter than a Q5, so Q5 is recommended if you want a bit more than Q4, but don’t want to jump to a Q8.

Don’t ask about Q7. We don’t talk about that.

K__M Stuff

That’s the method used to compress it.

Think of it like:

Same photo, different JPEG settings.

They all run the same model, but:

6. What Tokens and “Context” Mean (8K, 16K, 32K or more)

Context = how much the AI can remember in a conversation.

It’s rather like RAM in your PC, in that if you start a new conversation you lose what was in the previous one (but Biggly has a special system to help your characters remember!)

Context is measured in “Tokens”, which is a word or fragment of a word. For example ‘cat’ = 1 token, while ‘catch’ (cat + ch) = 2 tokens.

As a rough guide, 100 tokens is about 75 words.

4K means 4,000 tokens, or roughly 3,000 words, or about 6 pages of normal text.

Some AI models can handle more context than others.

You can adjust how much context to use in Biggly Bot settings. Do not set it higher than the model can handle, or the model will output gibberish and make no sense.

More context = more memory needed.

Note that you adjust the context setting, which is like saying how big a bucket you will use. The bucket is EMPTY when you start but fills up as you go. The fuller it gets, the more it will slow your computer down. That means you might load a big model, set the context very high, start using it and think “Well this is OK, it’s quite speedy?” – but then as you USE that long context and it start filling up, the AI replies will quickly become slower and slower.

Biggly has a self-pruning system that automatically deletes some of the very early messages in a long conversation, to prevent it suddenly stopping with ‘Out of memory’.

So:

Small PC → small context and stick with smaller models
Big PC → big context – and even a big, powerful PC will give slow responses if you set a very high context and actually use all of it, especially if using a larger model, because memory use is both the model and the context.

Wrapping Up

Now you hopefully understand what this means:

Goetia-24B-Q4_K_M.GGUF

Goetia is the name of the model, the 24B means it’s a large, powerful model by local standards, it’s the sweet-spot quant of Q4, and as Q4s go, it’s a mid-level K_M (medium), and it’s a GGUF file, meaning if you have a big enough GPU, you can run it locally on Biggly, Ollama, LM Studio, Kobold etc.

Note this is a big model for local; the file is 18GB in size. Even if you have a 16GB VRAM card, it means at least 2GB would be in in your RAM and running via your CPU, as your GPU’s VRAM would be full. So still fast but not as fast as a smaller model could be. On the other hand a 24GB VRAM card could swallow the whole thing and output much faster than you can read.

7. What OpenRouter Is (Cloud AI)

If your computer is too weak to run local AI properly, you can use OpenRouter, Huggingface or OpenAI, or in Biggly you can enter a custom URL and key for any other provider of your choice.

OpenRouter = a service that lets Biggly Bot talk to a wide variety of online AI models.

You access them with a special URL inside Biggly Bot, and must enter a special key (“API key”) that you can get from them (not from Biggly.com. We have no affiliation with any of the online providers Biggly Bot can use). API stands for Automated Programming Interface and is just the way Biggly connects to different providers.

I recommend Open Router because they have a huge variety of models. Some are very intelligent, some have very long context memory – but the bigger and more powerful the model the more money it costs to use them. Some are entirely free, but you DO need to make a $5 deposit to use the service (to prevent people making hundreds of free accounts and abusing their system)

The good news is they are really quite cheap. I once deposited $5 to test them out, but decided to buy a more powerful PC and run models locally. Later, while building Biggly Bot I deposited another $10 and used it a lot while testing the software. After 3 months I checked my balance – $13.20. I’d still used less than my original $5!

Typical costs: $0.001-0.01 per message depending on model. A casual user spending $5/month gets hundreds of conversations.

So:

• No powerful PC needed
• Works on laptops
• You pay per usage
• Needs internet

Both Open Router and Huggingace are known for less censorship than main providers and only they know who you are – the AI company just gets requests from Open Router. This gives you a degree of privacy but you are still sending your messages online. Talking of privacy…

Privacy

HF/OR don’t store your chat prompts/responses for training; request bodies are processed transiently and logs (user data/tokens) are kept up to 30 days for debugging only. Your IP, device info, and usage metadata get collected as standard, but messages stay private unless you opt into sharing. Data transits encrypted (TLS) – but you might want to double-check their policies haven’t changed since I wrote this!

Open Router is my recommendation, as they also offer ZDR, which means Open Router will only connect you to providers with a Zero Data Retention policy. That means you lose access to some models but does give you more privacy and peace of mind.

8. Can I Run Local AI on My Laptop?

You skipped the opening paragraph, huh?

The answer is maybe, but not really:

• Only small models
• Slower
• Limited memory – and even the ‘same’ make and model of GPU has less VRAM on the laptop version
• Not great for long, immersive chats

Most laptop users are better off using OpenRouter or HuggingFace, as both are pretty private, uncensored and easy to set up with Biggly Bot. Just get an API key from one of those, enter it in Biggly and off you go!

9. Should You Buy a Powerful PC?

Yes – if money is no problem!

But also, no. Unfortunately the massive growth in AI means the price of RAM and VRAM has gone through the roof, especially VRAM (which you can only buy as part of an expensive graphics card, and unlike PC RAM you cannot realistically upgrade your GPU’s VRAM).

If you have:

• A gaming PC
• 8GB VRAM or more
• Can follow some techy directions

Then you can already run remarkable smart and fun local AI models! For example a Q4 of a 12B with 8K context will give you an intelligent, in-depth conversation or role-play session. A smaller 7B or 8B model can also be fun – and will reply much faster than the 12B.

If you do build or buy a PC then the best GPU to buy is an RTX 3090. These have 24GB of VRAM but they are vastly cheaper than a 4090 (also 24GB) or a 5090 (32GB but massive price hike). Prices change a lot and by location, but you can likely find a 2nd hand 3090 for a few hundred dollars.

The GPU is by far the most important part of the PC for local AI, so if you already have 16GB of system RAM or more, upgrading just your GPU to one with 16-24GB GPU VRAM may be all you need – without replacing the rest of your machine.

An RTX 3090 with 24GB of VRAM will happily run 35B models, and run a Q4 quant of a 70B model at the same speed as a low-spec machine can run a 7B – VRAM is king when it comes to local AI!

• The more VRAM the better, but high-VRAM GPUs are very expensive now
• The more RAM the better, but VRAM is the main thing
• The CPU is virtually irrelevant for AI

10. Simple Decision Guide

If you have a gaming PC:
→ Try running local AI models first

If you have a normal PC or a laptop:
→ Use OpenRouter first

If you’re curious but cautious:
→ Start cloud, experiment later

11. Why Biggly Bot Exists

It’s the software I wish I had when I first tried getting into running local AI – and the same can be said for this guide!

Biggly Bot gives you:

• A simple, easy interface, with personal preferences
• Easy character creation, with memory and world lore
• A choice of running your own Local AI or using a variety of online AI providers
• Privacy when you want it (inc password-protecting characters)
• Powerful technical adjustments for those who want to dabble, but tucked behind the ‘Advanced’ button

Bottom line, you shouldn’t need a deep understanding of all the tech, to just use it.