Aggregation Pipeline

What the aggregation pipeline is#

The aggregation pipeline is MongoDB's tool for transforming and summarizing data — the rough equivalent of SQL's GROUP BY, JOIN, and SELECT combined into a single mechanism. A pipeline is an array of stages, each one taking the previous stage's output as its input and passing its own output to the next — conceptually similar to piping commands together in a shell.

db.orders.aggregate([
  { stage1 },
  { stage2 },
  { stage3 }
]);

Core stages#

$match — filtering#

$match filters documents, using the same query syntax as find(). Placing $match as early as possible in a pipeline is important for performance — it lets MongoDB discard non-matching documents (and use an index, if one exists) before any of the more expensive stages run.

db.orders.aggregate([
  { $match: { status: "shipped" } }
]);

$group — aggregating#

$group collapses documents sharing a common value for _id (the grouping key) into one output document per group, computing aggregates over each group with accumulator operators like $sum, $avg, $min, $max, $push:

db.orders.aggregate([
  { $group: {
      _id: "$userId",
      totalSpent: { $sum: "$amount" },
      orderCount: { $sum: 1 }
  }}
]);

_id: "$userId" groups by each distinct userId value (the $ prefix references a field from the input documents). $sum: 1 is the standard idiom for "count documents in this group," the aggregation-pipeline equivalent of SQL's COUNT(*).

$project — reshaping#

$project selects, renames, or computes fields for the documents flowing through the pipeline — similar to a SELECT column list:

db.orders.aggregate([
  { $project: {
      _id: 0,
      customer: "$userId",
      total: { $multiply: ["$price", "$quantity"] }
  }}
]);

$sort and $limit#

db.orders.aggregate([
  { $group: { _id: "$userId", totalSpent: { $sum: "$amount" } } },
  { $sort: { totalSpent: -1 } },   // -1 descending, 1 ascending
  { $limit: 5 }
]);

$lookup — joining across collections#

$lookup performs a left-outer-join-style lookup against another collection, the aggregation pipeline's answer to SQL's JOIN:

db.orders.aggregate([
  { $lookup: {
      from: "users",
      localField: "userId",
      foreignField: "_id",
      as: "userDetails"
  }}
]);

as: "userDetails" names the new field $lookup adds to each output document — an array of matching documents from the users collection (an array because, unlike a SQL join, $lookup doesn't flatten multiple matches into multiple output rows by default). Since _id is typically unique, that array will usually hold zero or one element, and a $unwind stage (below) turns it into a plain object.

A complete, realistic pipeline#

This pipeline answers "for each user, their total spend and order count, joined with their name and email, sorted by spend, top 5 only" — using the orders/users shape from earlier in this section:

db.orders.aggregate([
  // 1. Only consider completed orders
  { $match: { status: "shipped" } },

  // 2. Group by user, computing totals
  { $group: {
      _id: "$userId",
      totalSpent: { $sum: { $multiply: ["$price", "$quantity"] } },
      orderCount: { $sum: 1 }
  }},

  // 3. Join back to the users collection to get name/email
  { $lookup: {
      from: "users",
      localField: "_id",
      foreignField: "_id",
      as: "user"
  }},

  // 4. $lookup produces an array — flatten it to a single object per document
  { $unwind: "$user" },

  // 5. Shape the final output
  { $project: {
      _id: 0,
      name: "$user.name",
      email: "$user.email",
      totalSpent: 1,
      orderCount: 1
  }},

  // 6. Highest spenders first, top 5
  { $sort: { totalSpent: -1 } },
  { $limit: 5 }
]);

$unwind deserves a callout on its own: it takes an array field and outputs one document per array element, "unwinding" the array into separate documents — the step that turns $lookup's array result into a plain, flattened field, and more generally the tool for turning any embedded array (like the tags or comments arrays from the schema-design page) into one row per element for further aggregation.

Common mistakes#

• Placing $match late in the pipeline (or omitting it) instead of as early as possible — every stage before a filter processes documents that end up discarded anyway, and an early $match can also use an index the way find() would.
• Forgetting $unwind after a $lookup and then trying to access fields on the resulting array as if it were a plain object.
• Confusing the aggregation pipeline's $group accumulators ($sum, $avg, $push) with plain query operators ($gt, $in) — they look similar (both $-prefixed) but belong to entirely different contexts and aren't interchangeable.
• Building an enormous single pipeline instead of testing it stage by stage — since each stage's output feeds the next, running just the first one or two stages during development quickly reveals where a pipeline is producing wrong results.