Communication guidelines for incident response for small teams?
Last night we had a small incident where a customer port went down and then another switch went down shortly thereafter. It ended up being a small localized power outage in a building where the CE equipment was located, along with another unrelated switch of ours. Everything is resolved now.
We're a very small company (with <5 technical people) and I just happened to have my work laptop open at night and saw the alerts. I started to try and troubleshoot the problem - trying to understand what happened, how many customers were affected, if anything changed, etc, and realized I was having a very hard time multi-tasking doing that and communicating to the customer (and the rest of the team) that I was looking at the issue.
I eventually managed to post something in our shared teams channel about the incident, what had happened, and what I was doing about it, but I could have handled it a lot better if I had some kind of structure to follow.
I was wondering if anyone has any suggestions for some kind of structured communications for incident response? I've worked at larger enterprises before that had detailed steps of who does what when, but those were teams of hundreds of people, with dedicated 24/7 NOCs and entire teams of specialized engineers. I'm just one guy. An incident response guide will look different if there's a team of people all doing different tasks simultaneously vs 1 guy trying to do everything.
I noticed I had a lot of decision paralysis. The customer's contacting us, I need to respond. My team doesn't know there's an issue yet or that I've seen it and I'm working on it - I need to tell them. I need to understand the scope of the issue before I can meaningfully respond to either. I end up waffling back and forth, typing out a message and then erasing it because I don't want to send incorrect information out - meanwhile I'm trying to gather the correct information.
Does anyone happen to have a list of steps they've found that works for small teams/1 man shows?
I'm assuming it would look something like:
1. Acknowledge incident - cause under investigation - give an update time
Update at previously stated time - explain what has been ascertained and what are the next steps - give time for next update.
rinse and repeat
N. Give post-mortem report.
It's just tricky and stressful to have to stop troubleshooting mid-flow to write out an update about what's happening and what you're doing. It's even harder if you also need to devote brain power to making that update useful by providing correct and specific information instead of "waah, shit's broke! I'm working on it!!!"