cheap llm inference 3
https://myuberllmglowapipricingnews.talesignal.com/posts/understanding-glow-api-pricing-and-how-it-affects-your-ai-model-choices
Setting up cheap LLM inference feels like finding a bargain on a supercomputer. You're trading raw speed for smart optimization—quantizing models to int8, batching requests, and using CPU offloading during idle times. It's not as snappy as a premium API, but for internal tools or hobby projects, the cost savings are huge. Hosting on a refurbished workstation with vLLM or llama.cpp can run models like Llama 3 8B for pennies an hour, a far cry from cloud prices.