Tag
Every The Latent Daily story, tool, model and resource tagged “Inference”.
Quantization shrinks AI models by storing weights with fewer bits. How 8-bit and 4-bit formats work, GGUF, GPTQ and AWQ, and the quality trade-offs to expect.
AI Desk3 min read
What happens when a server answers thousands of LLM requests: prefill and decode, the KV cache, continuous batching, PagedAttention and the metrics that matter.
AI Desk3 min read
Estimate the GPU memory needed to run or fine-tune a language model: weights, KV cache, activations and overhead, with worked examples for 8B and 70B models.
AI Desk4 min read
A deep dive into AI economics: training compute, chips and data centers, inference costs, falling token prices, business models and who captures the value.
AI Desk6 min read