Preparing a Fine-Tuning Dataset
Collect, clean and format instruction data as chat-style JSONL, split train and validation sets, and avoid common data mistakes.
Updated Oct 30, 2026
Data is the whole game
The most common reason fine-tuning fails isn't the model or the hyperparameters — it's the data. A model fine-tuned on 500 clean, consistent examples usually beats one trained on 5,000 messy ones. This lesson shows how to build a dataset that teaches the behaviour you actually want.
The chat format (JSONL)
Most modern fine-tuning tools expect chat-formatted JSONL: one JSON object per line, each containing a list of messages.
JSON
{"messages": [{"role": "system", "content": "You are a helpful university admissions assistant. Answer in 2-3 sentences."}, {"role": "user", "content": "What documents do I need for the BS Computer Science application?"}, {"role": "assistant", "content": "You need your matric and intermediate transcripts, a copy of your CNIC or B-form, and two passport-size photos. Upload them as PDFs in the application portal before the deadline."}]}
{"messages": [{"role": "system", "content": "You are a helpful university admissions assistant. Answer in 2-3 sentences."}, {"role": "user", "content": "Is there an entry test?"}, {"role": "assistant", "content": "Yes, all BS applicants take the university entry test. It covers mathematics, English and analytical reasoning, and you can register for it on the admissions page."}]}
- system — the same instruction you'll use in production.
- user — a realistic input.
- assistant — the ideal answer you want the model to learn.
The model learns to produce the assistant turns, so every assistant message must be one you'd be proud to ship.
Where training data comes from
- Real logs — past support conversations, tickets or emails (anonymised!) with good answers.
- Expert-written examples — slow, but the highest quality for core cases.
- Synthetic data — use a strong LLM to generate examples, then review and filter them by hand or with automatic checks.
- Existing public datasets — check their licence and quality before using them.
How many examples?
| Goal | Rough starting size |
|---|---|
| Learn a strict output format | 100–500 |
| Consistent style or tone | 300–1,000 |
| A specialised task (classification, extraction) | 500–5,000 |
| Broad new skills | 10,000+ |
Start small, measure, and add data where the model fails.
Quality checklist
- Consistent — the same kind of input always gets the same kind of output (format, length, tone).
- Correct — wrong answers in training become confident wrong answers in production.
- Diverse — cover easy, hard, edge and "I can't help with that" cases. If 95% of examples are one type, the model overfits to it.
- Realistic — inputs should look like what real users type, typos included.
- De-duplicated — repeated examples get over-weighted.
- Safe and private — remove names, phone numbers, CNICs, emails and secrets.
Validate your JSONL with Python
Run this before every training job — it catches the mistakes that crash training or silently teach bad behaviour:
Python
import json
from collections import Counter
def validate_jsonl(path: str) -> None:
errors, roles_seen, seen = [], Counter(), set()
with open(path, encoding="utf-8") as f:
for line_no, line in enumerate(f, start=1):
if not line.strip():
continue
try:
row = json.loads(line)
except json.JSONDecodeError as exc:
errors.append(f"line {line_no}: invalid JSON ({exc.msg})")
continue
messages = row.get("messages")
if not isinstance(messages, list) or not messages:
errors.append(f"line {line_no}: missing 'messages' list")
continue
for m in messages:
roles_seen[m.get("role")] += 1
if m.get("role") not in {"system", "user", "assistant"}:
errors.append(f"line {line_no}: unknown role {m.get('role')!r}")
if not str(m.get("content", "")).strip():
errors.append(f"line {line_no}: empty content for role {m.get('role')!r}")
if messages[-1].get("role") != "assistant":
errors.append(f"line {line_no}: last message must be from the assistant")
key = json.dumps(messages, sort_keys=True)
if key in seen:
errors.append(f"line {line_no}: duplicate example")
seen.add(key)
print(f"{len(seen)} unique examples, roles: {dict(roles_seen)}")
print("\n".join(errors) if errors else "No problems found.")
validate_jsonl("train.jsonl")
Split train, validation and test
| Split | Typical size | Used for |
|---|---|---|
| Train | 80% | Updating the weights |
| Validation | 10% | Watching for overfitting during training |
| Test | 10% | The final, untouched score after training |
Python
import json
import random
with open("all_examples.jsonl", encoding="utf-8") as f:
rows = [line.rstrip("\n") + "\n" for line in f if line.strip()] # every row ends with one newline
random.seed(42)
random.shuffle(rows)
n = len(rows)
splits = {"train": rows[: int(0.8 * n)], "val": rows[int(0.8 * n): int(0.9 * n)], "test": rows[int(0.9 * n):]}
for name, part in splits.items():
with open(f"{name}.jsonl", "w", encoding="utf-8") as f:
f.writelines(part)
print(name, len(part))
Shuffle before splitting, and make sure near-duplicates don't end up in both train and test — otherwise test scores are inflated.
Common mistakes
- Training data that disagrees with itself. If half the examples answer in bullet points and half in paragraphs, the model learns randomness.
- A different system prompt in training and production. Use the exact production prompt in your data.
- Leaving personal data in the dataset. Models can memorise and repeat it.
- Only "happy path" examples. Include refusals, clarifying questions and out-of-scope inputs.
- Evaluating on the training set. Keep the test split locked away until the end.
Key takeaways
- Use chat-formatted JSONL with system, user and ideal assistant messages.
- Quality, consistency and diversity matter more than size.
- Validate every file automatically and remove duplicates and personal data.
- Split into train, validation and test before training — and never peek at test.
Finished this lesson?
Tick it off to track your LLM Fine-Tuning progress.