How to prepare for an on-call shift, and know what's likely to break
The best on-call shifts start before the first page: you can be reached, you know your services and who backs you up, and you already know which incidents are likely and what fixed them last time. Most teams cover the first two with a checklist. The third, knowing what is likely to break on your shift, usually depends on how good the handover was. Oppex On-Call Prep does that part for you.
The on-call prep checklist
Before the shift starts:
- You can be reached. Your phone number is current, you have consented to calls and texts, and the mobile app is installed and signed in. Send yourself a test page.
- You can get in. Laptop charged, VPN working, and access to the production consoles, dashboards and logs you will need, checked now rather than at 3 a.m.
- You know your services. Which services you cover on this shift, and where their dashboards and runbooks live.
- You know who backs you up. Who is on call at the next escalation level, and who the subject-matter experts are for the services you know least.
- You read the handover. What is still open, what was muted or put in maintenance, and anything deployed in the last day.
And the part that is hardest to do by hand:
- You know what is likely to break, and what fixed it before. Which alerts fire on your services every week, which rarer ones are due, and what the people before you found when they happened.
Why the last item is the hard one
The likely incidents and their causes are scattered across months of incident history: in the comments responders left, in the root cause analyses, and in the heads of whoever was on call last time. A new engineer, or anyone covering a service they rarely touch, has no quick way to find them. So the first time they see a recurring alert is when it pages them, and the fix that took someone else an hour to find gets found again from scratch.
How Oppex On-Call Prep does it
When you are on call, or your next shift starts within 7 days, a banner on the main Oppex pages links to your On-Call Prep page. If you have on-call reminder emails turned on, the reminder lists the likely incidents too.
| On the page | What it shows |
|---|---|
| Your shift | Whether you are on call now or when your next shift starts, the schedule and the window in your time zone |
| Regular alerts | Up to five alerts that fire regularly on your services, or have just started firing |
| Infrequent but likely this shift | Up to five alerts that fire less often, for example monthly, but are due during this shift |
| Key insight | For each incident, up to three likely causes with a confidence of High, Medium or Low |
| Similar incidents | The past incidents behind those causes, and a link to the latest occurrence |
| Readiness | How many of the listed incidents you have reviewed, and the time until your shift |
The lists are worked out every night from the last 90 days of incidents on the services whose escalation policies reach you during the shift.
Where the likely causes come from
On-Call Prep reads what people wrote on similar past incidents of the same service: the comments responders left, and the root cause analysis fields (reason for the alert, action taken, type of fix and notes). It summarizes the causes they point to, and every cause it shows must be backed by at least one of those past incidents.
The confidence levels are deliberately strict:
| Confidence | When |
|---|---|
| High | Two or more past incidents state the cause clearly, at least one of them in an RCA, and nothing contradicts it |
| Medium | One incident states it clearly, several only hint at it, or it comes from comments with no RCA |
| Low | The cause is mentioned vaguely, other evidence contradicts it, or all the incidents behind it are more than 60 days old |
So a High cause is one your team has written down more than once, not a guess. Each key insight also has Was this helpful? feedback, and every RCA your team writes makes the next shift's insights better.
What changes for the person on call
- The first page is not a surprise. You have already read the likely cause of the alerts that fire every week.
- The rare ones are covered too. A monthly batch job that fails is listed when it is due, not discovered at 2 a.m.
- Knowledge stops living in one person's head. What the last three people learned is on the page for whoever is on call next.
- RCAs pay off. Writing a root cause analysis is no longer paperwork; it is what prepares the next person.
On-Call Prep is available to every user for their own shift, including responders, once predictions are switched on for the workspace. How On-Call Prep works.
Frequently asked questions
How should I prepare for my on-call shift?
Before the shift, check that you can be reached and can get in (phone number, app, laptop, VPN and access), know your services and who is on call above and beside you, read the handover and the incidents from the last shift, and review the incidents most likely to happen on your services along with what fixed them last time. Oppex On-Call Prep does that last part for you: it lists the incidents likely during your shift and the causes past incidents point to.
Can a tool predict which incidents will happen during my on-call shift?
Yes, for incidents that recur. Oppex On-Call Prep looks at the last 90 days of incidents on the services you cover and lists up to five regular alerts and up to five less frequent ones that look likely during your shift. For each, it shows the likely causes from similar past incidents, with a confidence level, and links to those incidents. It cannot predict a failure that has never happened before.
How does Oppex decide how confident a likely cause is?
A cause is High only when two or more past incidents state it clearly, at least one of them in a root cause analysis, and nothing contradicts it. A cause backed by one incident, or only by comments without an RCA, is at most Medium, and a cause whose incidents are all more than 60 days old is Low. Every cause must be supported by at least one past incident.