Problem Types & Approaches

Problem types#

Most AI/ML work reduces to one of a handful of problem shapes. Recognizing which shape you're dealing with is often the single most important step in solving it, because it determines which algorithms are even candidates.

Classification#

Predict a discrete category. "Is this email spam or not spam?" "Is this transaction fraudulent?" "Which of these 10 digits is in this image?" The output is one of a fixed, known set of labels.

Regression#

Predict a continuous number. "What will this house sell for?" "How many minutes will this delivery take?" "What's tomorrow's stock price?" The output is a real number, not a category.

Clustering#

Group similar items together without being told the groups in advance. "Segment these customers into behaviorally similar groups." "Group these news articles by topic." There's no ground-truth label to learn from — the algorithm discovers structure in the data itself.

Generation#

Produce new content rather than a label or number: an image, a paragraph of text, a snippet of code, an audio clip. LLMs and image-generation models fall into this category — instead of classifying or scoring an input, they produce new output that didn't exist before.

Problem type	Output	Example
Classification	A discrete label	Spam / not spam
Regression	A continuous number	Predicted house price
Clustering	Group assignments (no predefined labels)	Customer segments
Generation	New content	A generated paragraph, image, or code snippet

Rule-based vs. learned approaches#

There are two fundamentally different ways to make a system produce the right output:

Rule-based (symbolic) approaches encode a human expert's knowledge directly as explicit logic: if transaction_amount > $10,000 and country != home_country, then flag. The logic is fully transparent and can be read and audited line by line. It works well when the domain is genuinely well-understood and the rules are stable, but it doesn't scale to problems where the "rules" are fuzzy, high-dimensional, or nobody can actually articulate them precisely — nobody can write down an explicit rule for "what does a cat look like in a photo."

Learned (statistical) approaches don't encode the logic directly — instead, an algorithm infers the logic from labeled examples. You show the system thousands of transactions labeled fraud/not-fraud, and it learns which patterns of features correlate with fraud. This scales to problems too fuzzy or high-dimensional for hand-written rules, at the cost of transparency — a trained model's decision boundary is rarely something a human can read and fully explain.

Where classical algorithms still win#

Deep learning gets the headlines, but it is not the default right answer for every problem. Classical, "shallow" algorithms — linear/logistic regression, decision trees, gradient-boosted trees, k-nearest neighbors — routinely beat deep learning in practice, specifically:

• Small, structured (tabular) datasets. Deep learning's advantage comes from learning features automatically from huge amounts of raw data (pixels, audio, text). On a spreadsheet-shaped dataset with a few thousand rows and a few dozen meaningful, already-named columns, there's little raw signal left for a neural network to "discover" — a gradient-boosted tree (for example, XGBoost or LightGBM) will typically match or beat a neural network, train in seconds instead of hours, and need far less tuning.
• Interpretability requirements. In regulated domains (credit scoring, medical decisions, insurance underwriting) you often need to explain why a specific prediction was made, to a regulator or a customer. A shallow decision tree or a linear model's coefficients are directly readable. A deep network's millions of weights are not, and post-hoc explainability tools (SHAP, LIME) are approximations, not the model's actual reasoning.
• Limited data or compute. Deep learning models typically need large datasets and non-trivial compute to train well; a linear model can be fit on a laptop in milliseconds with a few hundred data points and often generalizes better in that regime because it has far fewer parameters to overfit with.

Common mistakes#

• Reaching for a neural network by default, without first trying a much simpler, faster, cheaper classical baseline — you often can't tell if the deep model is actually worth its cost until you've measured against that baseline.
• Confusing "clustering" with "classification" — clustering has no ground-truth labels to check against; if you already have labeled categories, that's a classification problem, not a clustering one.
• Assuming an interpretable model is automatically a worse model — for many tabular business problems, a well-tuned gradient-boosted tree is both more interpretable and more accurate than a deep network.