Training vs Inference

Concepts Covered

• Training
• Inference
• Model learning
• Model serving
• Offline versus online work
• Latency
• Cost
• Fine-tuning
• Feedback
• Why live AI products care about inference

1. Plain-English Definition

Training is when a model learns patterns from data.

Inference is when a trained model is used to produce an output for a real input.

That is the shortest useful definition.

training: learn patterns
inference: use learned patterns

For a spam filter, training may use many emails labeled "spam" and "not spam." Inference happens when a new email arrives and the model predicts whether it is spam.

For a language model, training may involve huge amounts of text and specialized instruction data. Inference happens when a user sends a prompt and the model generates a response.

These are different moments in the life of a model.

Training creates or improves the model. Inference uses the model.

2. Why This Idea Exists

This distinction exists because AI systems have two very different kinds of work.

One kind of work is learning. The system looks at examples and adjusts the model so it becomes better at producing useful outputs. This can be expensive, slow, and data-heavy.

The other kind of work is serving. The product receives a real user request and needs an answer now. This work must care about latency, reliability, cost, user experience, privacy, and failure handling.

Software engineers already know similar splits.

Building an index is not the same thing as querying an index.

Compiling code is not the same thing as running code.

Generating a search ranking model is not the same thing as serving search results.

Training and inference have the same kind of separation.

If you mix them up, AI products become confusing. You might think the model learns from every prompt, or that using an API means training a model, or that updating product behavior always requires retraining.

Most of the time, live AI features are doing inference.

3. The Beginner Mental Model

Think of training as preparation and inference as usage.

Training is like preparing a model before it enters the product path.

Inference is like calling that prepared model during the product path.

data -> training -> trained model

user input -> trained model -> output

If you ask a chatbot a question, the model is usually not learning from your question in that moment. The product is using a model that was already trained.

Your conversation may be stored, evaluated, used for feedback, or used later depending on the product and settings, but that is not the same thing as the live request immediately training the model.

This mental model keeps the product path clear.

During inference, the product has a request, prepares input, calls the model, receives output, checks it, and responds to the user.

4. What That Mental Model Misses

The preparation-versus-usage model is helpful, but it hides some nuance.

First, there are different kinds of training. A model can be trained from scratch, fine-tuned on narrower data, or adapted with feedback. These are not all the same cost or complexity.

Second, not every improvement requires training. Sometimes the product gets better because the prompt improves, retrieval improves, context selection improves, tools improve, or output validation improves.

Third, inference can still be complex. It is not just "call model." A production inference path may include authentication, prompt construction, retrieval, model routing, streaming, tool calls, safety checks, logging, caching, retries, fallbacks, and evaluation.

Fourth, some systems have online learning or feedback loops where user behavior can influence future model updates. But even then, the live request path and the training pipeline are usually separate enough that engineers reason about them differently.

Training is where the model changes. Inference is where the product uses the current model.

5. A Concrete Example

Imagine you are building a product review classifier.

You want to classify reviews as:

• positive
• neutral
• negative

During training, you collect many examples:

"Great quality, arrived fast" -> positive
"It works, but the packaging was poor" -> neutral
"Battery died after one day" -> negative

The model learns patterns from those examples.

After training, your product receives a new review:

"The screen is beautiful but the charger stopped working."

During inference, the product sends that review to the trained model. The model returns something like:

neutral

The product may then decide what to do:

• show the review normally
• route it to quality monitoring
• summarize product issues
• combine it with other signals

The live user review did not rebuild the model. It used the model.

6. How It Works At A Practical Level

At a practical level, training often happens in a pipeline.

A simplified training pipeline looks like this:

collect data -> clean data -> train model -> evaluate model -> deploy model

This work may happen offline. It may require specialized hardware. It may take minutes, hours, days, or longer depending on the model and dataset.

Inference happens in the product-serving path.

A simplified inference path looks like this:

receive request -> prepare input -> call model -> validate output -> respond

This work may need to happen in milliseconds or seconds.

That difference creates different engineering concerns.

Training cares about:

• data quality
• labels
• compute cost
• evaluation
• model versioning
• reproducibility
• bias and coverage

Inference cares about:

• latency
• uptime
• cost per request
• rate limits
• output quality
• streaming
• fallback behavior
• monitoring

Both matter, but they are not the same job.

7. Where You See This In Real AI Products

In a ChatGPT-style assistant, the visible user request is inference. The user sends a prompt, the model generates tokens, and the product streams them back.

Training happened earlier. It involved large datasets, model architecture choices, optimization, instruction tuning, feedback, and evaluation.

In a Perplexity-style search product, inference may include retrieval plus answer generation. The model is used live, but the search index and the model were prepared earlier.

In a coding assistant, inference happens when the product reads context from your files and asks a model to explain or edit code.

In a fraud detection system, inference happens when a transaction arrives and the model produces a risk score. Training happened earlier on historical transaction data.

In a recommendation system, training may learn from past behavior, while inference ranks items for a user right now.

The product users usually experience inference, not training.

8. Common Confusions

Inference is not the same thing as training.

Calling a model during a user request is usually inference.

Fine-tuning is not the same thing as prompting.

Fine-tuning changes model behavior through additional training. Prompting changes the input you send at inference time.

Feedback is not always immediate training.

A thumbs-up or thumbs-down may be logged for evaluation or future improvement, but it does not necessarily update the model instantly.

Using an AI API is not the same thing as training a model.

Most API usage is inference against a model that already exists.

Retrieval is not training.

Retrieval adds relevant information into the input during inference. It does not by itself change the model.

9. What This Does Not Mean

This does not mean training is always done by your team.

Many teams use models trained by providers or open-source communities.

This does not mean inference is easy.

Serving AI features reliably can be difficult because output quality, latency, cost, and safety all matter at the same time.

This does not mean models never improve from user data.

They can, but that usually happens through controlled data pipelines, evaluation, and later model updates.

This does not mean every bug needs retraining.

Sometimes the fix is better context, better instructions, better retrieval, stricter validation, or clearer product boundaries.

10. What To Learn Next

Next, learn tokens and tokenization.

For language models, inference does not usually happen over "words" exactly the way humans think about words. Text is split into pieces called tokens.

Tokens affect:

• how prompts are measured
• how context windows work
• how generation happens
• why output streams piece by piece
• why long inputs cost more
• why models sometimes split unfamiliar words strangely

Once training and inference are clear, tokens make the live behavior of language models much easier to understand.