Skip to main content
An incident game day is a scheduled exercise where a team simulates a failure and runs their real incident response process end to end. The point is not to prove the system breaks; it’s to find out whether the people, tools, and playbooks that are supposed to handle a real incident actually work when you use them under pressure. Game days are how strong teams keep their response sharp between real incidents, and how new responders build the muscle memory to prepare for their first real one. This page covers why game days matter, how to plan one, and how to run them in Rootly using test incidents so your drill stays out of production metrics and off your status page, and how to audit workflows so paging and notifications only fire where you intend.

​
Why game days matter

Most incident response programs look healthy on paper: there’s an on-call schedule, a severity matrix, a runbook folder, a retrospective template. The gap between those artifacts and what actually happens at 3 a.m. is where incidents get long and expensive. Game days close that gap by exercising the whole incident process in a controlled setting. Running them regularly:
  • Validates the process end to end. Paging, role assignment, Slack channel creation, status updates, workflows, and the retrospective flow all get exercised together, not one at a time in isolation.
  • Trains new incident commanders and responders. People learn command by commanding. A game day is a safe place to declare, delegate, and make timeboxed calls before a real SEV1 forces it. Between game days, AI-simulated drills give individual responders extra reps.
  • Surfaces broken runbooks and stale ownership. Docs that no one has opened in a year, dashboards that 404, services with no clear owner: game days find them cheaply.
  • Tests tooling and integrations. Alert routes, escalation policies, workflow automations, status page publishing, and third-party integrations all get real traffic through them.
  • Builds cross-team coordination. Multi-team incidents are the hardest kind. Practicing hand-offs between platform, product, security, and support teams beats discovering the seams live.
  • Reduces mean time to recovery. Teams that drill respond faster, communicate more clearly, and make fewer avoidable mistakes when the real thing happens.

​
Industry best practices

The teams that get value from game days treat them like production incidents: planned, scoped, reviewed, and followed up on. A few patterns hold up across organizations:
  • Start small, then expand. The first game day should exercise the response process against a well-understood failure (a single service degraded, a dependency timing out). Save multi-team, multi-region, or security scenarios until the basics are boring.
  • Write a hypothesis before you start. “If the payments service returns 500s for 10% of requests, on-call is paged within 2 minutes, a SEV2 is declared within 5, and the status page updates within 10.” A game day either confirms the hypothesis or produces a specific finding.
  • Pick one primary objective per drill. Testing a new runbook, onboarding a new commander, validating a status page workflow, and rehearsing a security escalation are four different game days, not one.
  • Announce it, but not everything. Tell responders a game day is happening in a given window so they don’t page a real executive. Don’t tell them the exact scenario, the exact time, or which service will fail. Surprise is where the learning is.
  • Include everyone the real incident would include. On-call engineers, incident commander, communications lead, support, and anyone whose real job appears in your runbooks. A drill that skips the communications lead never tests communications.
  • Have an observer, not just participants. Assign someone to watch the response without joining it. They catch the things responders are too busy to notice, like decisions that stalled or steps that were skipped. AI transcription can help here, but don’t rely on it entirely; human observations are a critical piece of the learning.
  • Run a real retrospective afterwards. Use the same template, the same blameless framing, and the same action-item tracking you use for real incidents. If the game day doesn’t produce follow-through, it was theater.
  • Rotate scenarios. Repeating the same failure teaches the team to solve that failure, not to respond to novelty. Rotate across infrastructure, application, third-party, and security scenarios over time.
  • Keep the scope of impact controlled. Game days can and should touch real systems (that’s what makes them useful), but decide up front what is out of scope, who can call it off, and what “abort” looks like.
  • Measure the same things you measure for real incidents. Time to detect, time to page, time to declare, time to mitigate, time to communicate. Track them across drills to see whether the process is actually getting faster.

​
How to run a game day in Rootly

Rootly is designed to let you exercise the full response flow, including Slack channels, role assignment, workflows, and retrospectives, without polluting incident metrics or publishing to your status page. The key is using test incidents so the drill is clearly separated from production, and checking your workflows before the drill so they only page and notify the people you intend.

​
1. Plan the drill

Before the day of, write down:
  • Scenario — what fails, how, and when. Keep this to the game day leads.
  • Hypothesis — the specific outcomes you expect (detection time, who pages, which workflows run, which runbook applies).
  • Scope — which services, integrations, and teams are in play, and which are explicitly out.
  • Participants and observers — who is responding, who is watching, who is running the drill.
  • Abort criteria — the conditions under which you stop the game day early, and who has authority to call it.
  • Window — a defined start and end so responders know when “this could be a drill” applies and when it stops applying.
  • Workflow audit — which workflows page or notify people, and whether each should fire for the drill. Test incidents trigger workflows just like real ones: a workflow without a Kind run condition will fire for the drill if its other conditions match. See Exclude Test Incidents From a Paging Workflow.
Share the window and the fact that a game day is happening with a broader audience (leadership, adjacent teams, support), so no one confuses the drill for a real event or vice versa.

​
2. Declare a test incident

When you kick off the drill, declare the incident the same way responders would in production, but flag it as a test. In Rootly, run /rootly test in Slack or select the Test Incident checkbox on the New Incident form. Either one marks the incident as a drill so it’s clearly labeled everywhere it appears: in the incident list, in Slack, and in the timeline. Test incidents are excluded from production metrics and status page publication; see Incident Kind for the full behavior. Marking an incident as a test keeps the drill out of metrics and off your status page, but it doesn’t stop workflows from running. Paging and notification behavior depends on the workflow audit you did in planning. Use a test incident for every drill, without exception. See Creating Incidents via Web Interface for the full New Incident form, or Slack Slash Commands for /rootly test.

​
3. Run the response like it’s real

Once the incident is declared, the response should look identical to a production incident:
  • Roles get assigned (commander, technical lead, communications lead, scribe). See What Is an Incident Commander? for role guidance.
  • Responders investigate in the incident Slack channel and update the timeline as they go.
  • The commander drives status updates on the cadence your runbooks require.
  • Workflows fire the same way they would in production.
The goal is to exercise the muscle. Skipping steps because “it’s just a drill” defeats the purpose. If your workflows notify customers or page a real executive on SEV1, decide during the workflow audit whether to scope them to real incidents (Kind is: Incident, or Kind is one of: Incident, Sub Incident if you also page on sub-incidents), route them to test destinations, or leave them live.
Have observers take notes in a separate channel or doc during the drill. In-channel observation tends to turn into participation, which is exactly what you’re trying to avoid.

​
4. Resolve and run the retrospective

Resolve the test incident through the normal lifecycle (mitigated, then resolved) so the timeline captures the full arc. Then run a retrospective using the same template you’d use for a production incident of the same severity. See Incident Retrospectives for the full flow. Focus the retro on the response, not the scenario:
  • Where did detection, paging, or declaration lag?
  • Which roles were unclear or unfilled?
  • Which runbooks were missing, wrong, or hard to find?
  • Which workflows didn’t fire, fired incorrectly, or produced noise?
  • What decisions stalled, and why?
  • What would the observer have done differently?
Every finding should turn into a tracked action item with an owner and a due date. A game day that doesn’t produce assigned follow-through is a rehearsal that never ships.

​
5. Close the loop

After the drill:
  • Confirm the incident is clearly marked as a test everywhere it surfaced (channel and timeline).
  • If the retro surfaced individual skill gaps (communication, coordination, decisions under pressure), assign Rootly Academy drills that target them before the next game day.
  • Schedule the next game day before the current one is out of memory. Cadence matters more than perfection: a rough drill every quarter beats a perfect one every two years.

​
What game days are not

Game days sit in a family of related practices that teams often conflate. Each one exercises a different part of your resilience posture, and each answers a different question.

​
Game days vs. chaos engineering

Chaos engineering is the practice of injecting controlled failure into a running system to test whether it behaves the way you expect: does the circuit breaker trip, does the failover fire, does the retry back off. It’s a systems experiment with a hypothesis about the technology. A game day is an exercise about the people and process. The failure is the setup; the response is the point. The two pair naturally. A common pattern is to run a chaos experiment as the trigger for a game day: inject the failure, let on-call detect and respond as if it were real, and evaluate both the system’s behavior and the team’s response. But you can run either one without the other, and they have different success criteria: chaos engineering succeeds when the system holds up, a game day succeeds when the response does.

​
Game days vs. tabletop exercises

A tabletop exercise is a discussion-based drill. The team gathers (in a room or a call), someone reads out a scenario, and participants talk through what they would do at each step: who gets paged, what gets declared, what the commander decides, what goes on the status page. Nothing actually happens in the system, and no tooling is exercised. Tabletops are cheap, low-risk, and well-suited to:
  • Onboarding new responders or commanders to the process
  • Rehearsing rare scenarios (major security incident, data loss, regulatory event) where a live drill would be too disruptive
  • Cross-functional exercises that involve legal, communications, or executive stakeholders who don’t otherwise sit in a Slack channel
  • Reviewing a runbook or playbook before committing to it in a live drill
The tradeoff: tabletops don’t catch tooling gaps. A runbook that reads well in a conference room can still point to a dashboard that no longer exists. Treat tabletops as a way to pressure-test thinking, and use game days to pressure-test execution.

​
Game days vs. AI-simulated drills

Rootly Academy, built with Uptime Labs, offers AI-simulated incident drills: a self-paced alternative to a full game day. Responders work through a realistic scenario in 15 to 30 minutes, solo or as a team, while AI-powered characters play stakeholders, customers, and dependent teams. Each drill ends with a short debrief from experienced incident commanders. Because nothing needs to be scheduled and no real systems are touched, simulated drills fit well between live game days: onboarding new responders, building communication and coordination skills, and giving individuals reps they can’t get from a quarterly drill. They don’t exercise your own workflows, integrations, or runbooks, so use them alongside game days, not in place of them. Rootly Academy is available to Rootly customers. Contact your account team to get started, or read Introducing the Rootly Academy for more detail.

​
Game days vs. disaster recovery tests

A disaster recovery (DR) test validates that you can restore service after a catastrophic failure: a region goes down, a database is corrupted, a provider has a multi-hour outage. DR tests exercise backups, restore procedures, failover automation, and recovery time and point objectives (RTO/RPO). They are typically scheduled, heavily coordinated, and scoped narrowly to the recovery mechanism itself. Game days and DR tests overlap but aren’t the same:
  • A DR test asks, “can we recover?” Success is measured against RTO and RPO.
  • A game day asks, “can we respond?” Success is measured against detection, coordination, communication, and decision quality.
Many teams combine them: run a DR test as the scenario for a game day, so the recovery procedure and the incident response around it get exercised together. That combination is powerful, but plan it as one exercise with two sets of objectives, not one drill that happens to cover both by accident.

​
Quick comparison

A mature program combines them: tabletops to teach the process, AI-simulated drills to build individual skills between live exercises, chaos engineering to harden the system, game days to exercise the response, and DR tests to prove you can recover. Game days are also not a substitute for real incident practice: real incidents happen at inconvenient hours, involve incomplete information, and carry real consequences. Game days build the reflexes; real incidents keep them honed.

​
Where to go next