How Generative Models Work

The goal of every generative model#

A generative model tries to learn the distribution of its training data — what real examples "look like" — so it can produce new samples from that same distribution. Different model families do this in different ways. Four families matter most.

1. Autoregressive models (how LLMs generate)#

An autoregressive model predicts the next token given all previous tokens, appends it, and repeats.

Prompt:   "The capital of Pakistan is"
Step 1 →  " Islamabad"
Step 2 →  "."
Step 3 →  <end>

A token is a chunk of text — often a word or part of a word. At each step the model outputs a probability for every token in its vocabulary, and a sampling strategy picks one:

• Greedy — always take the most likely token. Deterministic but can be repetitive.
• Sampling with temperature — lower temperature favours likely tokens (focused), higher temperature spreads probability (more varied).
• Top-p (nucleus) — sample only from the smallest set of tokens whose probabilities add up to p.

These settings matter most when you run open-source models yourself. Some hosted model APIs now manage sampling internally and reject temperature/top_p parameters — check your provider's docs before setting them.

Modern LLMs are transformers trained in stages:

1. Pre-training on huge text corpora to predict the next token — this is where knowledge and language skill come from.
2. Instruction tuning on examples of following instructions well.
3. Preference tuning (for example RLHF) so answers are more helpful, honest and safe.

2. Diffusion models (how image generators work)#

A diffusion model learns to reverse a process that gradually adds noise.

• Forward process (training): take a real image and add a little noise many times until it's pure static.
• Reverse process (generation): start from pure noise and repeatedly predict and remove the noise, until a clean image appears.

Text prompts guide each denoising step, so the image moves towards "a watercolour of the Badshahi Mosque at sunset". The next lesson covers diffusion in detail.

3. GANs (Generative Adversarial Networks)#

A GAN trains two networks against each other:

• The generator creates fake images from random noise.
• The discriminator tries to tell real images from fakes.

As the discriminator gets better at spotting fakes, the generator gets better at making convincing ones. GANs produce sharp images quickly, but they're hard to train (unstable, prone to mode collapse where they produce little variety). Diffusion models replaced them for most image generation.

4. VAEs (Variational Autoencoders)#

A VAE has two halves:

• An encoder compresses input into a small latent vector — a compact description.
• A decoder reconstructs the input from that vector.

Because the latent space is smooth, you can sample new points and decode them into new outputs. VAEs alone produce blurrier images, but they're a key component of latent diffusion (e.g. Stable Diffusion), which runs diffusion in the VAE's compact latent space to save compute.

Why transformers and diffusion won#

• They scale. Quality keeps improving with more data, parameters and compute.
• Stable training. Next-token prediction and denoising are simple, well-behaved objectives.
• Flexible conditioning. Both can be guided by text, images or other inputs — the basis of multimodal AI.

Comparing the families#

Common mistakes#

• Thinking LLMs "look up" answers. They generate likely continuations; facts come from patterns learned in training (or from documents you supply).
• Setting temperature high for factual tasks. Use low temperature for accuracy, higher for creative writing.
• Assuming one model family fits every modality. Text and images are usually generated by different architectures, even inside one product.

Key takeaways#

• LLMs are autoregressive transformers that predict the next token repeatedly.
• Diffusion generates by removing noise step by step, guided by a prompt.
• GANs compete generator vs discriminator; VAEs learn a compressed latent space.
• Transformers and diffusion dominate because they scale and train stably.