I run a 5 bit quantized Qwen 3.8 27b at home on a gaming PC with an RX 7900 XTX and RAG memory using Gemma MoE (i forget what one)running on an old 1080 ti on my Valheim server.
It takes 3-15 minutes to run depending on how many tool calls the model makes and that’s mostly memory or web scrapes since that runs on the slower hardware. I mostly just use it as a discord bot to look stuff up for me and to fetch the news.
1080 Ti? But that is 11 GB of VRAM? I know because that is my GPU and the reason I have not tried it. They all call for 12 GB VRAM. I have 64 GB RAM, so 50 GB for the model would be fine.
On the 1080ti I run gemma4:e4b to to do RAG and vision jobs. Not sure the quants and I just run it in Ollama that auto adjust the model because I’m lazy. Not 100% sure but I think mixture of experts load the parts that are needed so there easier on memory. Offloading to CPU memory or swap is a big performance hit.
I tried to run a 300b model on the main PC once, but it took half an hour to get a few words out of it. You can go past the VRAM on the card but sacrifice speed.
Probably not, or if it runs it may take hours or days to get a full output. VRAM on the GPU really decides what will run reasonably. Try an 8 billion parameter phone sized model and see if its faster than sending a letter, then keep moving up until your setup can’t handle it.
8B models are a bit too dumb to use any tools reliably but should handle a basic conversation. llama cpp will allow you to manually load a model and tweak settings.
What kind of hardware do you need to run a model like that? And what do you use :-) ?
I run a 5 bit quantized Qwen 3.8 27b at home on a gaming PC with an RX 7900 XTX and RAG memory using Gemma MoE (i forget what one)running on an old 1080 ti on my Valheim server.
It takes 3-15 minutes to run depending on how many tool calls the model makes and that’s mostly memory or web scrapes since that runs on the slower hardware. I mostly just use it as a discord bot to look stuff up for me and to fetch the news.
1080 Ti? But that is 11 GB of VRAM? I know because that is my GPU and the reason I have not tried it. They all call for 12 GB VRAM. I have 64 GB RAM, so 50 GB for the model would be fine.
On the 1080ti I run gemma4:e4b to to do RAG and vision jobs. Not sure the quants and I just run it in Ollama that auto adjust the model because I’m lazy. Not 100% sure but I think mixture of experts load the parts that are needed so there easier on memory. Offloading to CPU memory or swap is a big performance hit.
I tried to run a 300b model on the main PC once, but it took half an hour to get a few words out of it. You can go past the VRAM on the card but sacrifice speed.
Sabata, goddess of computing, can I run it on a 32GB system with a 4GB nVidia GPU?
Probably not, or if it runs it may take hours or days to get a full output. VRAM on the GPU really decides what will run reasonably. Try an 8 billion parameter phone sized model and see if its faster than sending a letter, then keep moving up until your setup can’t handle it.
8B models are a bit too dumb to use any tools reliably but should handle a basic conversation.
llama cpp will allow you to manually load a model and tweak settings.
unsloth will give you a basic setup with some tools, manage model downloads, and settings.
Thank you for all the information 💕 !