NOA Labs

Prompt Templates & Versioning

Managing prompts as versioned artifacts instead of inline strings, and a practical rollout/rollback workflow.

Why prompts need the same discipline as code

A prompt that drives real application behavior is, functionally, a piece of the application's logic — changing it changes what the system does, the same way changing a function's implementation does. Treating prompts as disposable strings typed directly into an API call, with no version history, no code review, and no way to tell which prompt version produced a given past output, makes it impossible to reliably debug a regression, roll back a bad change, or even know what changed between two points in time. Managing prompts "as code" means applying the same basic engineering discipline already applied to everything else in the codebase.

Storing prompts as versioned artifacts, not inline strings

The simplest meaningful step: pull prompt text out of scattered inline strings across the codebase and into dedicated, version-controlled files or a prompt registry, referenced by name and version rather than duplicated wherever they're used.

Python
# prompts/support_ticket_summary/v3.txt
"""
Summarize the following support ticket in 2-3 sentences.
Include the customer's specific request and any error messages mentioned.
If the ticket is a duplicate of a known issue, note that explicitly.
"""
Python
def load_prompt(name: str, version: str) -> str:
    return open(f"prompts/{name}/{version}.txt").read()

prompt_text = load_prompt("support_ticket_summary", "v3")

This alone gives you: a diffable history of every change to the prompt (via normal version control), the ability to reference exactly which version produced a given logged output, and a natural place to attach the evaluation dataset and results for that specific version (see this track's evaluating-prompts-systematically page).

Versioning strategy

  • Version prompts independently from application code releases. A prompt often needs to change far more frequently than the surrounding application logic (tuning wording, adding an edge case), and coupling its release cadence to a full app deployment slows down exactly the kind of fast iteration prompting is supposed to enable.
  • Never silently overwrite a prompt in place. Create a new version (v4) rather than editing v3 directly, so any output already generated with v3, and any evaluation results recorded against it, stay meaningfully attributable and reproducible.
  • Tag which version is "live" explicitly, rather than always defaulting to "the latest file" — this makes gradual rollout (a percentage of traffic on a new version while validating it) and instant rollback (point back at the prior version) both simple, safe operations.
  • Record the prompt version alongside every logged interaction. When investigating a bad output days or weeks later, knowing exactly which prompt version generated it is essential — without it, you can't reliably reproduce or diagnose the issue at all.

A minimal versioning workflow

Python
PROMPT_REGISTRY = {
    "support_ticket_summary": {
        "live": "v3",
        "versions": {
            "v2": "Summarize this support ticket in 2-3 sentences.",
            "v3": "Summarize this support ticket in 2-3 sentences. Include "
                  "the customer's specific request and any error messages.",
        },
    }
}

def get_live_prompt(name: str) -> str:
    entry = PROMPT_REGISTRY[name]
    return entry["versions"][entry["live"]]

# Every logged interaction records which version actually ran
log_interaction(
    prompt_name="support_ticket_summary",
    prompt_version=PROMPT_REGISTRY["support_ticket_summary"]["live"],
    input=ticket_text,
    output=summary,
)

Promoting v3 to v4 later, once it's validated against the evaluation suite, is then a one-line change to "live" — with an equally simple one-line rollback available if the new version underperforms once it's live.

Common mistakes

  • Editing a prompt directly in place (overwriting the string used in production) instead of creating a new version — this makes past outputs unreproducible and removes any ability to roll back cleanly if the change turns out worse.
  • Deploying a new prompt version coupled to a full application release cycle — this needlessly slows down prompt iteration, which is usually meant to be one of the fastest, cheapest levers available (see this site's LLM tutorials on fine-tuning vs. RAG vs. prompting).
  • Not logging which prompt version produced a given output — without it, a reported bad response from last week becomes very hard to actually investigate or reproduce.