2) The details say that the performance numbers require 2.7x the overall power, so around 13.5W. Power needed for all the memory is not explicit in the numbers.
3) The compiler is also simulated and it's a prototype, so it's not a drop-in replacement for anything like Nvidia ISAAC
4) 5W seems to refer to sub-4B models, so not the 100B param ones.
80 TOP/s and 150GB/s memory gives us the fundamentals of where this sits performance wise although its not said if the TOP/s is for INT4 or FP16 or what which changes the prefill performance considerably.
A consumer graphics card (say a 9070 XT) will be something more like 750 TOP/s on INT4 (20TOP/s on FP16) and 650 GB/s. So its probably about a tenth the performance for prefill but maybe its only a third if its FP16 and can't handle more than 15GB per token due to its memory bandwidth (at Q4 that would be ~7B active parameters) for decode to achieve 10 token/s and to get its 30 token/s it would be max 2.5B active parameters at Q4. A full FP16 model on this seems very unlikely unless its a lot bigger than we think it is.
This I think is just about enough for a decent home AI, its a little slow and its only going to be of use for MOE (so no qwen 27B) but depending on price I can see it being of interest to a lot of people as some flash models aiming for 100B and 2-7B active parameters could be really quite a lot of home capability in a relatively low power consumption unit.
If it can work with 100B parameter models then stopping with edge devices makes no sense. This kind of compute-in-memory technology being so much more efficient would make most or all existing data centers obsolete. Or so it seems?
Also, they are saying this efficiency is going to become standard. I think this is the type of technology that will allow AI companies to keep increasing the size of AI models. So they will not be bottlenecked by energy needs to make the next qualitative step in intelligence.
It also points towards the next step for local AI after chips like RTX Spark. So say 18 to 36 months from now (or pick your more optimistic timeline) we expect performant local AI to be everywhere.
This mentions targets for inference of 10 to 30tps and I (albeit just skimming through) didn't find any info about ingestion speed. They also mention that the 1st gen chip is used for vision, though I suspect it's only for specific cases and not anything general.
Overall I like power efficient optimizations like this and let's hope that this is adopted broadly. Although the compute in memory sounds like a big increase in memory prices
1) The performance numbers are simulated
2) The details say that the performance numbers require 2.7x the overall power, so around 13.5W. Power needed for all the memory is not explicit in the numbers.
3) The compiler is also simulated and it's a prototype, so it's not a drop-in replacement for anything like Nvidia ISAAC
4) 5W seems to refer to sub-4B models, so not the 100B param ones.
Doesn't seem like 100B dense models are even on the table, only MoE models. Kinda misleading.
A consumer graphics card (say a 9070 XT) will be something more like 750 TOP/s on INT4 (20TOP/s on FP16) and 650 GB/s. So its probably about a tenth the performance for prefill but maybe its only a third if its FP16 and can't handle more than 15GB per token due to its memory bandwidth (at Q4 that would be ~7B active parameters) for decode to achieve 10 token/s and to get its 30 token/s it would be max 2.5B active parameters at Q4. A full FP16 model on this seems very unlikely unless its a lot bigger than we think it is.
This I think is just about enough for a decent home AI, its a little slow and its only going to be of use for MOE (so no qwen 27B) but depending on price I can see it being of interest to a lot of people as some flash models aiming for 100B and 2-7B active parameters could be really quite a lot of home capability in a relatively low power consumption unit.
Also, they are saying this efficiency is going to become standard. I think this is the type of technology that will allow AI companies to keep increasing the size of AI models. So they will not be bottlenecked by energy needs to make the next qualitative step in intelligence.
It also points towards the next step for local AI after chips like RTX Spark. So say 18 to 36 months from now (or pick your more optimistic timeline) we expect performant local AI to be everywhere.
Overall I like power efficient optimizations like this and let's hope that this is adopted broadly. Although the compute in memory sounds like a big increase in memory prices