I'm curious how your project compares to plain mmap!
Because llama.cpp will already run 26B in 2GB of RAM if you really want to (mmap enabled, repacking disabled).
It seems like the main difference is that your project synchronizes the SSD reads with inference activity, which you've presumably tuned to cause the least latency possible? Whereas the OS wouldn't care about any of that.
My first version used plain `mmap`. On the 8 GB M2, a cold 3.36 MB expert took 10 ms with mmap and 2.8 ms with `pread`. The full simulation was 0.50tok/s for `mmap` vs 4 tok/s for `pread`
With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work
And common weights still use mmap for simplicity
So, I believe llama.cpp might run it under 2gb, but I assume it will be slower
I'm really excited about what's been happening couple last weeks for local inference. I feel like it all started after colibri [1] was released.
Great work !
Anyone got recommendation about what local model to use for what purpose ?
I feel like (as they were saying in moonshot blog post [2]) each llm can be an expert in its own categories and with several small local we might get good coverage for decent usage, granted each one is specialized enough.
I have a project that's almost ready to run DiffusionGemma as well. The two project might potentially work well together. I'm getting ~20tok/s on a 36GB M3 and there's strong possibility we might be able to crib faster kernels from each other.
> It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro.
Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...
The M5 SSD's performance uplift was fairly substantial, even when compared to the prior generation.
> In the Blackmagic Disk Speed Test, the SSD in the M5 MacBook Pro achieved read speeds of up to 6,323 MB/s, compared to just 2,031 MB/s on the M4 MacBook Pro. It's not like the M4 is "slow" in a vacuum, but the M5 SSD is over three times faster, which is a great generation uplift.
My suspicion is that this is simply due to the M5 having more memory, and the OS already having most of the file cached. The M2 has more memory pressure and would cache fewer of the SSD reads
If that's true, inference speed would be even lower if you have only 2GB total, including OS caches
I've run local video generation models on an 8GB graphics card and know firsthand that nothing runs smoothly when memory is insufficient. So seeing 14GB of weights crammed into 2GB of RAM is impressive.
If running continuously for over an hour (like an overnight batch task), will a fanless MacBook Air overheat and throttle? Can the SSD handle the continuous weight reads and sustained output speeds?
This sounds really cool. My intuition was that the selected experts might change heavily for each token, resulting in slow SSD loads for each token. This seems to be wrong. Did you create some statistics on how often the experts need to be changed? What is the longest token run without any expert change? What does such a token run look like? In which cases do experts change frequently?
The full route changes almost every token. The cache works through partial reuse, about 40% of experts repeat on the next token and 57% within two tokens, cutting I/O from 166 to 88 ms/token on M2 Mac.
The longest exact repeat we found was only two tokens. Coding tasks may have higher reuse if code related experts are selected repeatedly
What exact specs do you have? It might be because it's the 256 GB version. afaik, those versions have much slower memory bandwidth than the 512 GB models
My friend tried it on an M4 MacBook Pro and got 25–27 tok/s
This is correct, the 256gb is substantially slower as uses fewer physical memory chips - less ability to read/write in parallel. The 512gb or larger models have substantially higher read/write rates, and typically performs 50-100 percent faster in benchmarks than the 256.
I measured this exact model with a 4k context on the mlx engine. It runs at 75 tok/s on my M5 Mac Pro and using 14 GB of RAM. For my engine the same model uses 2 GB of RAM and produces 31–35 tok/s.
The project is still experimental so performance may vary as it continues to improve. If you want to save around 12 GB of RAM for other tasks and you are ok with 35 tok/s (afaik it is roughly comparable to ChatGPT’s speed for basic responses) my engine may be a good fit.
If you need maximum speed and flexibility just use MLX
Because llama.cpp will already run 26B in 2GB of RAM if you really want to (mmap enabled, repacking disabled).
It seems like the main difference is that your project synchronizes the SSD reads with inference activity, which you've presumably tuned to cause the least latency possible? Whereas the OS wouldn't care about any of that.
With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work
And common weights still use mmap for simplicity
So, I believe llama.cpp might run it under 2gb, but I assume it will be slower
Anyone got recommendation about what local model to use for what purpose ? I feel like (as they were saying in moonshot blog post [2]) each llm can be an expert in its own categories and with several small local we might get good coverage for decent usage, granted each one is specialized enough.
[1] : https://github.com/JustVugg/colibri [2] : https://fireworks.ai/blog/kimik3-fable
Feel free to reach out.
(currently at https://github.com/mmastrac/diffgemma but not in a releasable state yet)
What are your thoughts on this?
Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...
> In the Blackmagic Disk Speed Test, the SSD in the M5 MacBook Pro achieved read speeds of up to 6,323 MB/s, compared to just 2,031 MB/s on the M4 MacBook Pro. It's not like the M4 is "slow" in a vacuum, but the M5 SSD is over three times faster, which is a great generation uplift.
https://www.tomshardware.com/laptops/macbooks/m5-macbook-pro...
If that's true, inference speed would be even lower if you have only 2GB total, including OS caches
Try to leave only a gigabyte or two free, speed likely would drop dramatically.
Edit: or do some calculation / logging of experts read speed, to see if it's faster than SSD spec.
If running continuously for over an hour (like an overnight batch task), will a fanless MacBook Air overheat and throttle? Can the SSD handle the continuous weight reads and sustained output speeds?
Great work, congratulations on the release!
The longest exact repeat we found was only two tokens. Coding tasks may have higher reuse if code related experts are selected repeatedly
My friend tried it on an M4 MacBook Pro and got 25–27 tok/s
What if there is enough RAM to fully load the model? I assume in that case I shouldn’t use your engine.
I measured this exact model with a 4k context on the mlx engine. It runs at 75 tok/s on my M5 Mac Pro and using 14 GB of RAM. For my engine the same model uses 2 GB of RAM and produces 31–35 tok/s.
The project is still experimental so performance may vary as it continues to improve. If you want to save around 12 GB of RAM for other tasks and you are ok with 35 tok/s (afaik it is roughly comparable to ChatGPT’s speed for basic responses) my engine may be a good fit.
If you need maximum speed and flexibility just use MLX
Windows PCs would require a completely different approach
I tried it on my wife's M1 MacBook Air 512GB and it gets 4–5 tok/s
Also, it must be easy to adjust for iPhones and iPads in theory