How LLM inference serving works: batching, KV cache and vLLM
What happens when a server answers thousands of LLM requests: prefill and decode, the KV cache, continuous batching, PagedAttention and the metrics that matter.
AI Desk3 min read
Section E
Developers & Builders — Page 2
AI coding, agents, APIs, RAG, fine-tuning, local AI and MLOps.
31 stories RSS
How much GPU memory does an AI model need? A sizing guide
Estimate the GPU memory needed to run or fine-tune a language model: weights, KV cache, activations and overhead, with worked examples for 8B and 70B models.
AI Desk4 min read
How to build an evaluation set for an LLM application
The most important tool in AI engineering: how to collect test cases, define success criteria, score outputs automatically and by hand, and run evals on every change.
AI Desk3 min read
LLM-as-a-judge explained: using AI models to grade AI output
Language models can score other models' answers at scale. How LLM-as-a-judge works, its known biases, and how to design rubrics and checks that make it reliable.
AI Desk3 min read
10 open-source AI projects every developer should know
From Hugging Face Transformers to llama.cpp, vLLM, Ollama, LangChain and ComfyUI: the open-source projects that form the backbone of today's AI development.
AI Desk3 min read
How to evaluate an open-source AI project before you depend on it
Stars aren't enough. Check maintenance, community, license, security, documentation and governance before building on an open-source AI library or model.
AI Desk3 min read
What is an NPU? AI PCs and on-device AI explained
Neural processing units let laptops and phones run AI locally. What NPUs do, what "40 TOPS" means, which features use them and whether you need an AI PC.
AI Desk3 min read
Prompt injection and LLM security: the risks every AI app must handle
Prompt injection, data leakage, insecure tool use and model supply-chain risks explained with the OWASP Top 10 for LLM applications and practical defenses.
AI Desk3 min read
AI NPCs and procedural worlds: how games are becoming generative
How language models power conversational game characters, how procedural generation creates worlds, and the design, cost and safety challenges studios face.
AI Desk3 min read
How AI is used in telecom networks
Self-optimizing networks, energy savings, predictive maintenance, fraud and spam detection, customer care and AI-RAN: how telecom operators use AI, and what 6G may bring.
AI Desk3 min read
How to contribute to open-source AI projects
A step-by-step guide to contributing to open-source AI: finding a project, starting with docs and issues, making pull requests, and sharing models or datasets.
AI Desk3 min read
Paper of the DayResearch
Paper of the day: Direct Preference Optimization (DPO), explained
DPO showed language models can be aligned to human preferences without a separate reward model or reinforcement learning. How it works and why it spread so fast.
AI Desk3 min read
Quick take: bigger context windows are not always better
Models now accept huge inputs, but stuffing everything into the prompt can reduce accuracy and raise costs. When long context helps, and when retrieval works better.
AI Desk3 min read