The best way to tell if a business is reliable is to observe how they act when everything is falling apart.
And I’m talking end-of-the-world disaster here.
For an airline company like Jet Blue, it’s a huge ice storm hitting the East Coast, grounding well over 1,000 flights.
For a car manufacturer like Toyota or Ford, it’s a manufacturing mistake that forced the companies to recall hundreds of thousands cars.
For a software company like LiveChat, this can be a great many things. But one of the worst that can happen to us is the total interruption of our service. A complete blackout.
Imagine your company completely loses the ability to operate because it is being targeted by a group of criminals in a massive, coordinated attack.
You completely loose the ability to service your customers. They start to get angry with you because they can't get anything done. Some of them will leave. Others won't quickly forget that your service was down and will think twice before using it again. Each minute is a small tragedy, a small let down that shakes the confidence your customers had in you.
It sounds scary, doesn’t it?
But it doesn’t have to be that scary and I will tell you why.
There are crisis management methods that can prepare you for the absolute worst. And without costing you an arm and a leg.
Crisis management is about preparation (and not panicking)
You can’t tell when your next big disaster will happen. You need to prepare and be ready when it does happen.
Just as nobody builds shelters when the bombs start to fall, you can't be thinking about your crisis management strategy only when something already went wrong.
Joe Bush, who is charge of service delivery, quality and consistency in the The Chat Shop, a professional customer service outsourcing company, stresses the importance of preparation when asked about the crisis management methods they use. It’s a rule their agents follow every day:
Make sure you’re set up correctly and prepared. Computer on. Logged into live chat, Knowledge Base, Slack, Email and any other system you might need for the team you’re on ahead of your shift. Then, if you do have any issues there is always someone else on to cover you.
Thanks to careful preparation, we were able get through our last major crisis.
To give you a bit of a background, we were recently hit by a DDoS (Distributed Denial of Service) attack, which is pretty terrible for any online company.
In this kind of attack, the assailants use computer scripts pretending to be normal users that perform million actions a minute in our software.
This unusually heavy load of traffic made the service unavailable for the majority of real users. Most of the 12,000 companies using our service couldn’t do it at that point.
We didn’t act like a deer in headlights though. We knew exactly what to do because we’ve prepared for it.
Crisis management methods
When we get hit by a DDoS attack, we follow a plan. You can use a similar plan to detect and deal with a crisis of your own.
It’s like triggering a carefully designed chemical reaction. First signs of a problem trigger actions that trigger even more actions. And at the end of all of them, you defuse the crisis.
1. You need to monitor the situation
The sooner you notice that something is not right, the more time you have to react to it.
In our case, this is done with monitoring tools that automatically check if our service is OK. Online businesses and software companies can use tools like Hyperspin, Zabbix and Pingdom to monitor their websites and applications.
On top of any automatic monitoring you might do, you should also take notice of any unusual reports from your customers as they will be probably first to let you know that something’s up.
2. Notify everyone involved
Once you spot a potential problem, you need to sound the alarm and fill in other people in your company.
Communication is paramount in situations like this. And if you can’t keep it clean and organized, you risk chaos.
You need a shared communication channel that anyone from your company can reach at any time, with a couple of backups if possible.
In LiveChat, we use Slack for instant team communication. We also use Gmail for our emails.
We have a spreadsheet set up in Google Sheets with phone numbers to every member of the team. This way, if somebody is not available via Slack or Gmail, we can always notify them via a quick call.
The shift management software we use called 7shifts has the option to mass send a text message to all agents. This way one person can notify everyone immediately without having to call each person one by one.
3. Everybody needs to know what to do
Once you spread the news, it’s time for some direct crisis management. Anyone that can help with the problem, should drop what they are doing and get right on it.
This is especially important for B2B companies as your customers businesses may well depend on your service.
For LiveChat, this would be the network administrators who immediately started working on enabling our anti-DDoS defenses and actively tried to locate and separate the attack traffic.
Simultaneously, your other departments should start trying to help too. Even if they can’t directly do anything with the problem, they can help with other things like customer enquiries.
Our customer service department was quick to react, manned the battle stations and started answering questions about the sudden drop of service.
4. Keep your customers in the loop
Another thing other members of your team can do is fill your customers in on the problem.
Apart from answering customer service questions via your support channels, you can have your social media managers spread the news on your Twitter and Facebook feeds.
We have a Twitter account dedicated to communicating technical news and information called LiveChat Status.
People responsible for your website can prepare an announcement that notifies anyone coming to your website.
We also use a dedicated service called StatusPage.io do display the status of our service in real time. If we want to broadcast any information, we can do it via the status page too.
If you’re using live chat on your website, you can modify the welcome message of your chat to immediately notify visitors about the crisis. During the DDoS attack, we added a short message informing customers about it, along with a link to our status page.
5. Calling all hands on deck
If the problem you’re facing is especially messy, don’t hesitate to get everyone involved.
There are things that cannot be simply sped up by throwing more manpower at them. For example, you will see diminishing returns when you add people to solve a programming problem.
However, there are things you can improve by assigning more people to them. One of those things is customer service.
Once the disaster hits, you will get a ton of questions from angry customers. A heck of a lot more than you usually tend to get.
In our case, we got around 46 percent more chats, when compared to the average from three previous weeks. That’s nearly the double to what we are used to, which resulted in clients going to the queue.
Since our shift work is carefully planned, we don’t usually have a big reserve. This is why we mobilized people from other departments. We had to quadruple the usual number of agents and went from 4 agents to 16 people answering chats in the peak.
The involvement of people from other departments helped with the bump in chats and got rid of the queues.
6. Clean up after you resolve the problem
Once the fire dies down a bit and you get your disaster under control, it’s time to switch from crisis management to drawing conclusions. You should plug any newly discovered holes in your service and let everybody know that the air raid is over.
Something obviously went wrong and you need to take steps that will prepare you for the next time it comes.
After the DDoS, we bulked up our network defenses and employed a couple new means of filtering the bad traffic.
You also need to let your customers know what happened. At this point, the majority of your customers is probably still in the dark. You need to not only tell them what happened, but also what you have done to fix it and what are your plans for similar situations in the future.
We make sure that we replied to all customer tickets and emails. If we get a lot of enquiries, we also prepare a post-mortem email in which we tell our customers what hit the service, what actions were taken with a clear timeline and what we have to to make it less likely in the future.
Crisis management is not only about reacting to but also about learning from your mistakes.
Experience-based crisis management
What is the best way to test your crisis management contingency plan? See how it fares against some kind of a disaster.
You won’t know if what you have planned for will be enough. According to Murphy’s law, it probably won’t, but it’s much easier to adapt once you have something to build on.
You can go a really long time without any major crisis, but when it comes, you’ll be glad that you dedicated some time in the past to mitigate it.
Make plans, drill your team and make your company ready for the unknown. With the right plan, the unknown doesn’t seem that scary.
Photos courtesy of Nautilus International, John Lloyd and Johannes Lundberg.