18 comments

  • lrvick 13 minutes ago
    I run 8x r9700s at home which lets me run up to 2x 1M context sessions of GLM 5.3 flash at ~50tps or 12x sessions of qwen 3.8 flash next at ~100tps.

    I exclusively use this setup for all AI usage from fixing obscure reproducibility bugs in software that once took me weeks, to compiler bugs, to kernel hardening and hardware drivers, to building a complete unprivileged and security hardened replacement for systemd in rust. Most of that was done using Qwen 27b back when I had only 2x r9700s.

    There is literally not a single problem I have faced in 25 years of systems engineering that I cannot solve now 10x+ faster using my local LLM setup.

    I turned down the free OpenAI and Anthropic tokens in my YC batch because I have absolutely no interest in being reliant on third parties to do my job.

    • wez470 6 minutes ago
      How cost effective had it been? I don't want to rely on 3rd parties either, but my experience is it's super expensive to run locally versus using a provider (both in terms of hardware costs and ongoing electricity usage).
    • txdv 4 minutes ago
      whats electricity bill and do you not fear that your hardware will be obsolete while others will have access to better models on faster hardware?

      i just calculated that it should be Per month (30 days) 380euro in my region, which is a lot after an initial 20k.

      I have to say, no limits stopping sounds nice though.

    • random3 8 minutes ago
      Can you share more, or direct us to more resources around local setups? For starters what's the upfront cost, what are the space, weight, energy requirements, etc.
      • vablings 2 minutes ago
        There are tons of resources out there on how to run a local setup. Once you have something that's somewhat functional you can then have the system self optimize your software stack to suit your needs.

        As for cost a 9700 is running at about 2k USD MSRP right now and the rated power is 300w.

  • sachaa 1 minute ago
    Why should users have to choose a model at all? We don't choose a codec before playing a video. An app should be able to request transcription or summarization, and the browser should find and run a suitable local model. That's the layer i am exploring with modelxare.com.
  • paxy 1 hour ago
    I don't see any difference between the conversations around self-hosting local AI and self-hosting regular software, which have been going on for decades.

    Yes, there are a hundred valid reasons to self-host. But 99%+ of consumers aren't going to care.

    People don't even care when the cost of hosting is ~free, like a simple home server. How many are ready to buy thousands of dollars worth of AI hardware, which could very well be obsolete a couple of years from now?

    • wry_xy 59 minutes ago
      The self-hosting crowd, Linux fans, OSS contributors, etc, are our own worst enemies.

      If we care about self-hosting and open source and decentralization because we believe it leads to a better world, and not simply because we want to be control freaks and snobs, then we need to learn how to build shit that is actually convenient and usable and understandable to normal people.

      I think it's possible. I think many people would want this, and care, if we could package it up in a digestible way.

      • lilbigdoot 47 minutes ago
        Linux has been more user friendly and accessible to me for normal use than Windows for years now. I really do not understand this comment
        • guhidalg 21 minutes ago
          Talk to people not in tech.

          Windows never ever ever makes you open a terminal to do anything, that’s scary to non-tech people. Windows “just works” and most people just want to minimize their computer use for work and go watch tiktok on their phone.

      • forshaper 19 minutes ago
        At some point there is a trade-off between those far away controlling "your" stuff and convenience. It is much easier for someone to make something convenient if they have control over it. For example, troubleshooting when you own the stack versus troubleshooting when you don't. However, if someone wants someone far away to help them troubleshoot, then it will be easier if the one far away controls the stuff. There is a trade-off there.
      • pelagicAustral 47 minutes ago
        I'd say one of the major frictions with self-hosting is that for the most part it does always going to take a lot of tinkering around and maintaining things going. A world were we have a lot more people self-hosting would be great, but as it stands, it's an utopia, because normal users will get annoyed the fuck out of the environment the second their infra is busted because of any of the million reasons things break.
        • pydry 12 minutes ago
          Self hosting is in pretty much the same state smartphones were in before the iPhone.

          I remember trying to install apps and it was annoying for all the same reasons self hosting is these days. It was niche, only hobbyists did it, "nobody cared" and the presumption was that for normal non-nerds, phones were exclusively for calls and texting.

          There are some spectacular successes in terms of the actual self hosted software (immich and home assistant) but the overall experience is still N95 not iPhone.

        • japbujan 7 minutes ago
          [flagged]
      • davedx 44 minutes ago
        Ollama and friends are pretty damn easy to use already. Hardware is the real barrier
        • lurking_swe 41 minutes ago
          let's see here, you already make a crucial mistake that a product manager wouldn't. (I say this as a senior software engineer that has worked with product managers a lot)

          If I tell my mom to download and install Ollama on her Mac, do you think she's gonna have any clue which model to download from all of the 5000 models she can choose from? Do you think she understands which ones will even work well on her machine? What if she downloads one that's highly recommended online but it doesn't work on her 64 GB MacBook (hypothetical example)? Because it's too too large/slow...

          and can Ollama, out of the box, do actual work on her computer, similar to the ChatGPT app with its "Work" mode, and dozens of built in connectors (google, etc)?

          you call this easy to use for a normal person? You just made their point. The ChatGPT and Claude apps that my mom can download, set the standard for her. Her standard of good/easy is not what you and I as nerds find easy. you need to really put yourself in the shoes of a tech illiterate person. we haven't even touched on the fact that Claude and ChatGPT ship with easy to use iOS and Android apps that sync up perfectly with her Mac.

          • anonymousDan 27 minutes ago
            This is a bit of a strawman argument. Who cares about your Mom (no offense)? The important part for me is that technical people (e.g. software developers etc) have access to competitive open source and/or local models.
            • guhidalg 20 minutes ago
              That may be personally important to you, but the market for tech-literate people is dwarfed by everyone else.
      • zoeysmithe 26 minutes ago
        Its worth mentioning "normal people" take hours upon hours of classes and trainings to use typical office productivity software and other job-related software, on top of watching and learning from tutorials and family and friends to use personal software and websites and apps. And heaven knows how many support calls and tickets to just do what is supposed to be "easy" in the closed source world.

        Also, none of this is intuitive and a lot of popular closed stuff is harder to use (and vastly disliked by the very same normal users) than some popular open source stuff.

        I'd even mention 'normies' want self hosted Word and Excel and regularly reject the web-based versions of them. Normies don't want streaming games but a gaming PC or console. Normies may buy streaming services like Netflix but get upset when their favorite shows are removed, so they actually don't like netflix. Normies complain about censorship on twitter and reddit and discord and would love an alternative they control, but may be unaware that's an option and seem stuck on those services and hate them.

        Normies want self-hosted all the time too. Their ability to understand and do this is what's lacking. So that's a conversation on its own, but normies aren't actually happy. Form what I can tell most people hate 50% of the stuff they use right now and most of those complaints revolve around the cloud-hosted/lack of control/privacy and other issues self-hosting or federated services or FOSS apps could arguably solve for them.

        This is a big conversation but imho tends to boil down to who has the marketing and sway to get you to install or use something, not that typical end users are savvy judges of software and useability working purely on merit and informed facts. Does your grandma actually like Facebook, or is she just lonely and its the lowest friction way to not be lonely? Do you really like Chrome or Word or is it just easy to get and popular? Is the big mac the best burger ever or is just easy to get and popular?

      • templar_snow 15 minutes ago
        This. The FOSS/etc crowd don't even try to bother with outreach on tools that the majority of people would want if they were marketed correctly instead of kept as niche projects for people to be smug about. Consider the EU rolling out Matrix in a huge way, or people moving to Mastodon. For most normies, "Use this to escape Elon Musk" is more than enough sales pitch.
      • nemomarx 58 minutes ago
        There's a decent opportunity where AI coding agents seem good at creating a very "average" and understandable UI. Maybe not perfect, but probably better than Gimp?
      • goda90 56 minutes ago
        The problem with open source is that it is often written by people who are fixing issues they have or making features they miss. To get to the point of contributing, you have to go through the initial setup process. Once you've done that, it's not longer a problem you care about, so you're less motivated to contribute code to make that process easier.
      • pyrale 27 minutes ago
        > not simply because we want to be control freaks and snobs

        It’s not so much about being snobs as it is about not having time and energy for a second job. The level of quality you demand simply requires a level of attention, investment and involvement that can’t be provided without working full-time on it, for profit.

        Telling a bunch of hobbyists that they should provide service of a quality equivalent to that of a trillion-dollar corporation if they are serious about it is gaslighting.

        • dotancohen 23 minutes ago
          I hate to say this, but Claude Code has turned my home projects and personal scripts from "I tolerate this because I need it" to "This works so nicely I might make it available to others". AI could really help here.

          I'll go purify myself now.

      • betterunix2 46 minutes ago
        > shit that is actually convenient

        I mean...sure...but how convenient was that Office 365 outage? How convenient is it to find that a feature you actually depended on is gone because the company that runs your software decided to get rid of it?

        It is always a back-and-forth. It is not as if centralized computing is a new idea, in fact it was the original way to use computers and people found that issues of control mattered and switched to PCs; the pendulum swung back because people realized that there are advantages to centralization (e.g. not having a failed hard drive destroy high-value documents). The cycle will probably continue for a long time to come because the advantages and disadvantages of both approaches are fundamental.

        • bendergarcia 43 minutes ago
          That inconvenience is still less inconvenient than not being able to figure out your dns or vpn or tail scar settings or why it won’t render and why the machine isn’t reachable. When running your own services is easier than it currently is and lot of tools are built around notifications and debugging that are actually understandable, then you’ll have more adoption of open source software.
    • bee_rider 16 minutes ago
      I think ultimately local LLMs will win but only because models will get good enough that the “cost” of running locally on a 2030 chip will be less than the “cost” of talking to the network. Especially given that building wider/lower precision LLM accelerators is a perfect task for hardware companies (embarrassingly parallel in an era where sequential performance improvements have long dried up).

      In terms of self-hosting vs remote-hosting regular software… I mean, a lot of regular software is fundamentally about communication (social media and office suites) so that goes on the server. LLMs don’t benefit much from being remotely hosted, other than that it makes the flops run in somebody else’s machine. But that will be hard to fund in a universe where the upper-limit to what can be charged is defined by how much additional cost “good AI hardware” adds to the price of your phone. We’ll see I guess.

    • nsbk 42 minutes ago
      Hard disagree. I'm running Qwen3.8-Flash-Next using Strata on a Mac Studio and it covers all my needs. One drag and drop install. Nothing like self-hosting your email.
      • mortalapeman 30 minutes ago
        Seconding this. If you know how to build with a dumber model it scales crazy well. And Qwen 3.8 Flash Next is just wildly good even at low quants. I'm trying to find it's limits and I've been using it for over a week building and haven't hit a wall yet.
        • choxi 27 minutes ago
          Are you using it for coding? I'm looking for a good coding model to use for Xcode and NextJS apps
      • xandrius 24 minutes ago
        How much was your Mac Studio?
    • wmichelin 35 minutes ago
      Self hosting is not free, it's actually pretty hard to setup and keep running sometimes. Time and cognitive overhead are money!
      • Zambyte 28 minutes ago
        > cognitive overhead

        I told a remote model to set up strata on my local machine. With the best options for my hardware. If someone can figure out using remote models, it's trivial to bootstrap local intelligence assuming available hardware. The hardest part was knowing that strata exists.

        Ninja edit: I've even had my agent tell me about suboptimal hardware configurations and told me what to do to fix it (XMP settings, having hardware in suboptimal slots). Cognitive load is simply a non-issue. The only issue is time, but in the grand scheme of things, it's a trivial amount of time to get up and running.

      • mortalapeman 28 minutes ago
        Just get a 5070 and 64 GB of RAM and download Strata. It's the ollama of actually usable local AI.
      • nullsanity 26 minutes ago
        [dead]
    • jbjbjbjb 52 minutes ago
      It’s more about self-hosting the hardware which is more desirable than thin clients.
    • random3 10 minutes ago
      People complaining about how difficult self-hosting is seem to be stuck in the past and missing the point. It's the very AI that we're talking about that removes most if not all friction points related to software configuration and maintenance.

      While this doesn't remove the upfront cost (which could be fixed with basic financing like Apple does) and physical manipulation (e.g. replacement on failure, etc.) it does lower the barrier.

      Finally, there's another path I find interesting and interested to think more of which is sharing hardware locally, from your house to your neighborhood, etc. We've done this in the late 90s - early 00s with Internet and it worked really well. I think it could work great for compute.

      I see a lot of opportunities in this direction, actually and most are actually enabled by the latest improvements in AI, so most of the "old" thinking doesn't apply anymore.

    • redanddead 20 minutes ago
      It comes down to the incentives

      Think of who uses local AI, it’s either

      -power users upset with the limits —-cost of intelligence is going down

      -people and corps that want stabiltiy —-a massive self own by the labs, this is justified

      -American labs seemingly snitching on their users (wtf?) —-huge incentive for people to use Chinese inference, which is cheaper along with other benefits

      Regardless, we’re talking about a subset of a subset of people, 1% of 1%. The big bet is that everyone will become AI power users. I think it’s accurate, forget openAI, fuck Anthropic, let’s look at us, the people. Entire workflows like coding have been buried, there’s no going back to how things were before. Is it reasonable to say that will happen to the other work streams? There’s no indication not, and it’s a total gamble if it’s yes

    • bobajeff 47 minutes ago
      That's like the argument on why people don't care about privacy. Just because most people choose that, all things considered, they are willing to sacrifice sovereignty over their own computer usage doesn't mean they don't care about it.

      By that reasoning you'd also conclude that no one cares about affordable housing or having a good work life balance or having your own lake made out of solid gold.

    • barbazoo 1 hour ago
      It's so sad that we are not empowered to act as more than just consumers as if it was some kind of shared religion. How do we get out of this cycle?
    • jmyeet 11 minutes ago
      Here's one difference: currently lawyers using cloud LLMs violates attorney-client privilege. And you have to be really careful sending confidential medical information to one so as to not violate HIPAA.

      More broadly, those prompts and the responses have value. There's a whole shadow economy in China of aggregating queries across Claude (etc) and then using that as training data. It's like your entire search history is for sale.

      Clould infra has typically been opaque to the provider. You run an MySQL instance on an Amazon VM and Amazon doesn't have easy access to the unencrypted traffic and what you store. Cloud LLMs don't operate that way.

      AI hardware is improving at a rapid pace. By 2030 I believe a significant amount of LLM usage will be local for these and other reasons.

    • jimbokun 12 minutes ago
      Enough that Apple currently build enough Mac Minis and Mac Studios to satisfy the demand.
    • adammarples 17 minutes ago
      I don't know about this whole argument that hardware will become obsolete. If it can run Opus 5.5 today it will always be able to run Opus 5.5 until it burns out, and the quality of the models it can run will continue to increase as the software improves.
    • nullsanity 29 minutes ago
      [dead]
  • giancarlostoro 1 hour ago
    If Nvidia didnt abandon consumer GPUs we all could be running better models locally by now. The moat is companies like Nvidia not releasing better consumer hardware so they can partner with bigger companies so they can sell us overpriced hardware.

    They release a new RTX series every two years, last time was 2022.

    Meanwhile major AI companies are investing in their own chips, if Nvidia wants to survive they really should not abandon consumers.

    Hoping AMD steps up to the plate. Meanwhile the nextXbox and PlayStation consoles are going to have a rough time with their next consoles which will likely cost a thousand dollars or more is my gut feeling.

    • codehorses 1 hour ago
      Not defending Nvidia here, but the RTX 50 series was released in 2025, was it not?

      Regardless, they aren't planning on sticking to the two year release cadence any more which does suck.

      • paxy 1 hour ago
        RTX 5090 - release price $2000, average price today $6500

        RTX 5080 - release price $1000, average price today $2300

        Nvidia also stopped selling its own version of these cards directly to consumers and is happy with letting partners and resellers/scalpers control the price.

        So it is pretty obvious that the company doesn't particularly care about this market segment.

        • giancarlostoro 4 minutes ago
          > So it is pretty obvious that the company doesn't particularly care about this market segment.

          Which I think is foolish, every AI shop is making their own alternatives, it's not going to be a winning bet for them.

        • theaxeonthanksg 1 hour ago
          [dead]
      • Buttons840 1 hour ago
        Ironically, if consumer GPU progress stalls, then slowly PC becomes the best console. Everyone has the same top-end GPU that can run any game.
        • ghostbrainalpha 1 hour ago
          Or game streaming actually becomes a thing. Because the best new games only play in the cloud on computers we could never afford.

          Almost like going back to the Arcade era except we all pay monthly subscriptions instead of 1 quarter at a time.

          • unrented7977 1 hour ago
            Game streaming isn't going to happen. It's been tried several times now and failed every time.
            • Buttons840 30 minutes ago
              I guess gamers are tech savvy enough that they care about ownership on principle? Or why is it that game streaming hasn't taken off?
              • Lord-Jobo 22 minutes ago
                Almost every gamer can immediately tell the difference in fidelity and latency. And only a very small handful of gamers have the internet quality for game streaming at low enough latency and high enough fidelity to be “sort of on par”.

                The quality and internet connection stuff will improve over time. The latency problem essentially never will.

    • Tuna-Fish 22 minutes ago
      GPU-style platforms (lots of compute connected to a big shared pipe that connects to a large pool of fast memory) are inherently more efficient for centralized inference.

      Local inference gets truly interesting when the proper next-gen, natively AI and not just a repurposed GPU, stuff starts to land. Initially things like compute-in-memory and fancy flash, and later possibly the truly exotic stuff like various RMM (read-mostly-ram, for example stacked FeFET on logic) tech.

      IMO there is no chance for ubiquitous local AI to happen if that means people using GPUs. But the industry is chasing all possible solutions for more efficient inference platforms, and if something like pachinko machine architecture with FeFETs ends up winning, suddenly buying a local ai inference solution in a box is no longer as insane.

    • winwang 14 minutes ago
      If consumers were willing to pay more for GPUs like datacenters do, NVIDIA would have sold plenty of consumer GPUs. Or would you press a button to halve your revenue for some reason?

      (I say this as someone who would have liked a 5090...)

    • jimbokun 2 minutes ago
      Yes, ten the home AI market would be similar to the serious PC gaming market used to be when they were the main market for the latest GPUs.
    • scronkfinkle 1 hour ago
      > Hoping AMD steps up to the plate

      AMD's aggressive marketing as an AI-first company leaves me pessimistic about that.

    • jjtheblunt 1 hour ago
      > we all could be running better models locally by now

      Apple silicon says "hold my beer", which I know because i've been doing this on my 3 year old quite beefy mac studio, for example. it's good enough for many tasks.

      • vadansky 1 hour ago
        Can't wait for M7! I hope the rumours are true.
    • Forgeties79 1 hour ago
      No gut feeling necessary. They announced weeks ago that they are going to be $1200-$1800 (Xbox specifically, PS6 somewhere in that range too).
      • lubujackson 1 hour ago
        Not to mention the required monthly subscription... Income inequality like 1929, might as well bring sharecropping back.
  • cj00 20 minutes ago
    Invariably when local AI comes up, people try to see how it can be used for frontier-level AI tasks like asking it to one shot a web app from scratch and then they get disappointed when they discover it's not performing at the same level as Opus.

    But they're ignoring that a local LLM can still be very useful for any kind of repetitive task, things like categorization or generating embeddings, translation or summarization tasks. I'm self-hosting Karakeep and use a local LLM for the AI tasks like embedding.

    And once you've got that local LLM running, you can start thinking about integrations with your own custom built apps.

    • sachaa 15 minutes ago
      Also, they are thinking of one single LLM model at a time. What if we have a network of dynamically loaded micro-models loaded on demand based on the user intent? Think of a global DNS of micro-LLMs. We need to decentralize AI. I started building this here https://modelxare.com
  • brendan_rules_t 45 minutes ago
    I have a 128gb ddr5 unified system, that i have been setting up with linux and local llms and other models and i have to say it feels beyond next level! I still use claude, grok, codex, and neo clouds to offload heavy work loads, but anything i want private or says too much about my system is now done locally. The voice agent experience is incredible
  • montroser 49 minutes ago
    I run Qwen3.8-27B Q4 on a $1500 box in the basement with 32GB VRAM, and a customized OpenWebUI on top. It's pretty great. Occasionally I consult Claude for a second opinion, but I'd say I've moved 95% of my previous conversational Claude usage to be local instead. I use it for leisure coding projects too, but for real work, it's still DeepSeek 4.1 Flash for now.
  • herf 1 hour ago
    I love local AI, but even so, I admit the economics point the other way. Unlike gaming, AI is a uniquely good match for data centers because the applications today aren't latency-sensitive at all (except perhaps DLSS). Most importantly, utilization is massively better with always-on servers doing parallelism (in-flight batching). Also in some areas (like California) our local electricity is quite expensive, so moving the work 100ms away can help. The privacy implications are horrifying, though.
    • younglunaman 54 minutes ago
      I tend to disagree here. Think about how useful it would be to have a nice llm in real-time debugging your applications loaded entirely on your computer. That sounds pretty sweet to me. I don't see how that's not really cool / immediately useful to you?

      Maybe your point is the majority of the time that's not needed, fair, then maybe the approach is both?

    • drdexebtjl 59 minutes ago
      > the applications today aren't latency-sensitive at all

      This is circular reasoning. We don’t have latency-sensitive applications because sufficiently powerful local AI is not profitable to entertain such applications.

    • musha68k 1 hour ago
      The economics point the other way because capital is aligned against you. Should this be a bubble that pops we will hopefully see prices come down again.

      MooresLawIsDead and his spidey web / senses thinks it already popped:

      https://youtu.be/YJeK2gumpEI

  • smath 10 minutes ago
    What is a good place to track the latest capabilities of open source local LLMs vs commercial models? Things like overall intelligence performance, latency/speed, hardware needed, cost.

    Anecdotally, I believe the open weights local models are constantly catching up with commercial ones, but I've honestly lost track. This is a dynamic space.

  • maxdo 1 hour ago
    i'll share a lonely voice on hn, i care very little about local ai.

    I feel like people who consider it as an option thinks of narrow scenarios vs ai is part of our life like internet is. How do you things on mobile for example with your local models?

    AI is very competitive, I use various models, I like to be on cutting edge and to be productive, its the opposite from doing something permanent, and not changing, where local models can be good. I like to do things on the go, and the last thing any normal person would want is to have DC at their basement.

    • cableshaft 1 hour ago
      I'm not willing to feed my personal data (emails, files, etc) to a non-local AI model (well, beyond what I can't avoid, like Google and my Gmail account).

      But I would like to have an AI that does that, like synthesize my journals and notes and provide a local personal knowledgebase I can query, things like that.

      I don't think I have a good enough GPU to manage a local model powerful enough to be worth the effort just yet, though, so I'm waiting a bit longer.

      For coding and bouncing ideas off of though? Yeah sure, online is fine.

      • zbendefy 45 minutes ago
        > well, beyond what I can't avoid, like Google and my Gmail account

        I think this is a real moat google has that will benefit them in the long run.

    • ale42 1 hour ago
      I think that many people want similar things. But (at least personally), not giving my data to BigTech is more important that this.
      • maxdo 1 hour ago
        out of curiosity, is there anything special your data has?
        • desmondl 36 minutes ago
          I’m pretty sure most people on HN have gone down weird wikipedia rabbit holes late at night. All that data is probably stored somewhere. What if 5 years from now an AI classifier by an incompetent private company looks through everyone’s search history and falsely classifies anyone who clicked on "_____" on wikipedia at 3am on a sunday as suspicious?
        • montroser 57 minutes ago
          It's not about anything special -- it's about not wanting to have big tech sharing my personal details that will allow them to influence my world view and how I think, in exchange for political/corporate money.

          Like, you know it's already happening now on Grok that they will sell influence. And it can be so effective if they already know what's inside the head of the person whose mind they're trying to change. All the sudden we have masses of people believing vaping isn't that bad for middle school kids, or that we really should just go ahead and trust Gemini with the most intimate details of our personal lives...

        • cableshaft 1 hour ago
          This sounds dangerously close to saying "If you have nothing to hide, you've got nothing to fear."

          https://en.wikipedia.org/wiki/Nothing_to_hide_argument

        • pluralmonad 9 minutes ago
          If you do or did have them, would you want google training Gemini on pictures of your kids?
    • segmondy 52 minutes ago
      I run local AI, there are tons of local models with no cloud alternatives, the only way to unlock and have these functionality is by being able to run local. So you are not on the cutting edge by using cloud only.
    • musha68k 21 minutes ago
      They will need to realize quantum proof, performant and verifiable zero trust compute to stay competitive with the sovereign front.

      Yet this would subvert one, if not the main pillar of funding - which is total self imposed (of course) surveillance.

    • tencentshill 43 minutes ago
      Local (Free) beats cloud (free) for pretty much any task except live web search (which will be less useful as more cloud AI scrapers are blocked). You run into the free limits in one or two prompts most days if you try to do anything even mildly complex.
    • NoidFonsense 1 hour ago
      I just set up a wireguard tunnel to my home network
    • jakswa 1 hour ago

          How do you things on mobile for example with your local models?
      
      You think you will be doing those things? And if you do keep doing those things forever, won't AI be smart enough to figure it out for you, whatever needs to happy locally?
    • codybontecou 1 hour ago
      Whisper and other stt models work fine on mobile.

      Mobile is far behind frontier coding agents but they are already solving valuable problems with competitive quality and privacy, one-time payments as viable options.

    • xscott 17 minutes ago
      Online models will eventually enshitify to serve you advertisements. They'll give you worse answers and steer you towards commercial products to solve your problem.
    • yieldcrv 1 hour ago
      I disagree, I think local is the supercycle here once it gets out of enthusiast territory

      We have LLMs baked directly into silicon getting 20,000 tokens/sec with low power consumption, Taalas was bought by AMD weeks after that demo. In that world, people get a new LLM only on hardware refreshes, and only after LLMs are good enough to be in stasis for years

      Apple’s M7 Pro/Max if it meets promised will be very expensive, I see some enterprises rolling them out to some staff

      Some language models in the Apple Ecosystem are already baked into iOS and MacOS, and its ubiquity is invisible

      Thats the direction I see things going, but like not anytime soon to alleviate supply chain crunch. 2029, 2032

    • nater5000 1 hour ago
      First, I'll say that I, too, care little about local AI. I do like toying around with local AI, and I do see how it can be a very important option in some contexts. I also loosely agree that it's good for keeping frontier models on their toes, prevent this technology from being locked away behind a paywall, etc. But for me, as of now, they're not very useful, and I use AI quite a bit.

      However, I do think your perspective on local models is a bit narrow. For a lot of people, they want AI tools but do not need cutting edge AI. They'd prefer to just have their own AI which is capable enough and under their control. They'd run it on a small server at home and connect to it as needed for their work (including on their phone). It's worth noting that much of the consumer-grade infrastructure definitely doesn't exist for this, but one could blame the lack of consumer-grade hardware as the source of this rather than just a general lack of demand. You can definitely see people picking up on this by running their various assistants, like OpenClaw, on Mac Minis from their home. It's definitely not accessible or cheap enough for most consumers at this point, but it's also not impossible to imagine us getting to the point where it is more ubiquitous.

      I'll say, personally, that I run Qwen 3.8 27B on an RTX 3090 from my home server, and it operates pretty well for what it is. I'm certainly not giving up my Claude subscription anytime soon, but it is nice to have the option and I can chat with it directly on my phone securely (using Tailscale) and using a nice UI.

      Regardless, I think the whole point of these kinds of discussions is to point out that, right now, local models aren't viable, but not because they can't be viable, but because companies are actively preventing them from being more viable. I can see for myself that they work well enough that proper investment could easily push them into the "good enough" territory for many users, and I suppose it is inevitable that more and more people will end up running local models in one way or another. But I definitely agree with the sentiment that most consumers simply do not care about it. They don't want to deal with hardware, they don't want to think about software, and they're happy to pay $20 per month for a subscription that "just works." I definitely don't blame them.

    • betterunix2 54 minutes ago
      It would be a lot easier to do local AI if we were not using natural language as the interface. In fact there is already plenty of local AI, including on mobile, that most users do not even realize is there -- because nobody has "conversations" with those models.

      Niche? Sure, but let's be honest here, a lot of AI use right now is forced, which is usually a sign that the product is not actually in demand or has questionable value for users. The big, natural-language-interpreting models are also niche -- a minority of people find them extremely useful and a majority find them to be "meh" at best.

    • dofm 1 hour ago
      > How do you things on mobile for example with your local models?

      The same way you can screenshare or forward from your local Mac? Paseo, LM Studio and LM Studio Bionic all support this kind of tailscale-ish access to local models from phone apps. (Haven't tried any of these myself)

      I do, personally, have "narrow scenarios" for AI. I have to use it for work, and want to be informed and engaged on that level, but I want to keep it out of my personal life as much as is practical.

      If you don't, that's fine, but I reframed my mass social media use down to an extremely minimal pattern in 2014 and I am healthier for it; I intend to keep the same distance from AI in my hobbies, for several of the same reasons.

      Paradoxically, for me that has involved taking the control over it that local models offer. I use the smallest, simplest things that get the job done and I try to retain my engagement with my work.

      It's all pretty low-effort and low-tinkering; no more than setting up local VMs for dev work. Old M1 Max macbook, Unsloth, Pi, a model in the Muse Glimmer/Qwen 27B/Nemotron Bolt sort of bracket.

      Good enough to be a rubber duck that talks back, write the minimal kind of AI-generated support code I want, and teach me how it all works.

      I might one day buy an M5 Pro Mac Mini, or an M5 Max Mac Studio, or stick an R9700 in a homebuild PC, or I might just use openrouter for the few things the Macbook won't do.

      I am a cynic: I believe AI has some value, but I also believe it should be a commodified technology rather than a branded SaaS subscription product, and it should demonstrate that value without me letting a couple of large corporations FUD me into pushing it into every corner of my life at once, or simply handing over control of my job to it.

      It is a matter of personal discipline, for me: I am going to resist the endless FOMO of new model benchmarks, learn how it works, keep it in a box, and take it out when I need it.

    • lubujackson 1 hour ago
      I agree. It reminds me a lot of building your own desktop computer from parts - it started as a requirement, then a cost-effective choice, then became for power users only, and now it has little place beyond tinkerers/purpose-driven/privacy tinhats.

      I refused to use Macs for a decade because of how offensive I found it that they locked down their computers from even adding RAM yourself. But this is the direction of the tide and swimming against it will only continue to become more and more arduous. At best, we will have some Tor-like hosted AI, but I am not holding my breath or wasting my money/energy.

    • neutronicus 54 minutes ago
      No, I'm the same way. I started trying to spec out a local AI box, looked at the numbers, and it felt like trying to run my house off a generator.

      Obviously worse than the centralized option, and not the hobby I want. I'd rather just use Cloud tokens help me do what I actually want to do.

  • jkingsman 1 hour ago
    Even dumber local models that are heavily quantized to fit on consumer GPUs can be lovely as a dimwitted household assistant. A lot of people look to offload things they already do to LLMs, which can be great with frontier models, but you've got the highest standards and preferences for tasks you already do, so it's the most likely to fall into a "should have just done that myself" outcome. I've found great success in instead lowering my threshold of things I'd take time to do -- I wouldn't spend time on it, but I'd spend 60s prompting an LLM to take a crack. I'm fortunate to have the hardware to run Qwen 3.8 Flash Next at q8, but most of the tasks I have it do would be totally doable for a mid-quant Qwen 3.8 27b.

    Some recent tasks I've thrown it:

    * Go through my blood test data (from EHR XML dumps from prior years) and give me some over-time charts for consistently-sampled measurements

    * Go through my Good Articles bookmarks, epubify them, and drop one down onto my ereader each morning for my commute, then archive the offline version into cold storage

    * Go through my music library, identify my most-listened low-bitrate tracks, and source or assemble a shopping cart to get high res versions

    * Analyze local tax/foreclosure auctions for vacant land; grab aerial screenshots, tax data, ownership data, and compile a weekly report of things maybe worth bidding on based on some specific selection criteria (fun facts: if you're willing to accept some headaches, 5+ acres can be yours for under $5k, if you're in the right area)

    * Dump my grocery store loyalty data, analyze my purchasing habits, and prepare a weekly instacart for my approval that aims to sync with usage habits. Tweak and learn volume/order adjustments over time.

    * Given a KVM that can mount ISOs onto the host, reimage two old laptops of unknown status with Debian and install Minecraft so my nieces and I can have a LAN party

    * Organize my music (high confidence MusicBrainz Picard matches), movies/TV shows, ebooks, and video game ROM collections with metadata and good folder organization

    * Archive bookmarked fanfiction for my offline library (slowly, and respectfully, and with an actual browser. The browser is anti-bot-countermeasured to the gills, but QFN regularly solves hCaptcha and Cloudflare turnstile challenges. Once I gave it the ability to drag, it even reasonably reliably solves drag-shape captchas. A little while back it made a purchase on eBay which is NOT a bot friendly site.)

    Would I personally spend time on doing most of these? Generally not. Could an LLM screw them up so badly I regret it? Also generally not. I get the luxury of a psuedo-assistant that doesn't judge how menial the tasks I send it are, and I save time. With a local LLM, my privacy concerns are far less than with frontier models, and local abliterated models are less (or just not at all) twitchy around things that frontier classifiers balk at.

    I'm hopeful that big projects like Muse can help to normalize menial task offloading, and that local LLMs provide the bridge for more technically minded folks to use them even more aggressively and limitlessly (as in compute limits/subscription costs) and cheaply.

    • Scrapemist 45 minutes ago
      Trying to get into this too. But my hardware is a bit limited atm. What are you running Q8 on? At what speed?
  • philipodonnell 1 hour ago
    I think one of the more compelling reasons for personal-local AI is that when two services you use add their own AI that only works in that service it creates the need for a third AI to bridge those two. But any service the proposes to sell you that third AI will be blocked by the other two, thus, only people who can create their own local service will be able to bridge them, and that will be the new version of DIY.
  • xnx 49 minutes ago
    "Local" is the wrong distinction. Self-hosting doesn't have to mean running off a box in my house. I don't care about using my own hardware as long as I can run the AI anywhere I please.
    • xandrius 21 minutes ago
      That's basically self-hosted which is basically local in this conversation, no?
  • rockingmeister 31 minutes ago
    [flagged]
  • gnaneshwar 45 minutes ago
    [dead]
  • dahunt33 1 hour ago
    [flagged]
  • Betelbuddy 52 minutes ago
    [dead]
  • routebraid 1 hour ago
    [flagged]