For the last five months or so, opus and fable have both been able to do this task as a soneshot prompt. An inference engine is almost pure compute and to a first approximation the only operation your hardware needs to support to do llm inference is multiplication [the rest of an inference engine is "some trig"].
So to get any model working on $OLD_HARDWARE, the hardest part of the task is now sourcing the hardware itself and giving the agent ssh access -- a modern agent will be able to source a compiler, write the implementation, optimize it, test in qemu (or any other simulator) and push it onto the device.
Example prompt: "write an inference engine for qwen3 0.6b in pure assembly that runs on this device. Write a script to attach the weights to the resulting binary. Prefix is supplied as a stream of u32 token IDs on stdin. The engine streams u32 token IDs on stdout."
It's technically sweet regardless, so don't take this as even a little bit of negative pushback, but I am curious what the utility of these small models is? Are they smart enough to be useful on practical tasks? What are people doing with them?
Not op but my opinion is that It doesn't matter if it's not currently useful. I think it's important that the community focuses on this line of research because big companies will not. It's in their best interest to make LLMs resource hungry
Recently I started this open source project at my AI company, where I'm trying to make LLMs working
Now Reached 16B models running on iPad Air 1st gen and xiaomi tv box, both have less than 1GB of usable memory, and reached some decent speeds given the hardware.
I'm open sourcing the effort done, kernels developed and results for everyone to re-produce and push more, also the modified models are on hugging face : https://huggingface.co/hyperionaidev
I want a more challenging device, what do you think?
Aw man I would love to be able to use the crank to back up the agent dialogue, maybe make it branch forward in a different direction. Make the dialogue into a game
So to get any model working on $OLD_HARDWARE, the hardest part of the task is now sourcing the hardware itself and giving the agent ssh access -- a modern agent will be able to source a compiler, write the implementation, optimize it, test in qemu (or any other simulator) and push it onto the device.
Example prompt: "write an inference engine for qwen3 0.6b in pure assembly that runs on this device. Write a script to attach the weights to the resulting binary. Prefix is supplied as a stream of u32 token IDs on stdin. The engine streams u32 token IDs on stdout."
0: https://github.com/brysontang/iiama2.c
Now Reached 16B models running on iPad Air 1st gen and xiaomi tv box, both have less than 1GB of usable memory, and reached some decent speeds given the hardware.
I'm open sourcing the effort done, kernels developed and results for everyone to re-produce and push more, also the modified models are on hugging face : https://huggingface.co/hyperionaidev
I want a more challenging device, what do you think?
I'd thought of the ESP32 devices but their RAM is too small
It also runs on MS-DOS 80386 2MB RAM, so...
And iPhone 1. And Digital UNIX.
A Playdate
https://play.date/
One of those vapes with a screen on it.
I'd be interested in seeing some older flag ship mobile devices
Not my site, but same idea: https://www.parkytowers.me.uk/thin/
All this old/obsolete gear is plenty capable when used with the right software.