In the past 10 years of working on cyber breach incident responses, I have learned that while no two incident responses are the same there are critical mistakes that many IR firms and in-house security staff make. All too often my teams are called in to replace an existing IR firm that isn’t doing the job properly. Here are four pitfalls that I see unecessarily creating anxiety, increasing costs, and leaving companies without the necessary information to protect their network in the future.
1. Do not jump to eradicate without containment. In several breaches, the client started immediately deleting malware. Like stepping on a mound, the ants scattered and started mounds all over the place. In one case, the actors could not get in from their main backdoor, so they activated another backdoor and planted multiple additional hidden backdoors. The goal should be to ensure as much containment as possible so that when eradication occurs, you get all of the ants in one stomp. Of course, this is easier said than done because you may never know all of the backdoors on your network; especially if the backdoors are custom. The bottom line is never start eradication with partial containment. Don’t lose your element of surprise and let the investigation get the information it needs to secure the network.
That being said, containment and eradication should be coordinated. The goal is to secure the network and disrupt attack activity. The two efforts are dependent upon one another in that a) containment measures such as network monitoring should be used to verify eradication measures, and b) eradication measures should be built upon evidence found during containment measures such as network monitoring and scanning file systems for artifacts.
If an organization performs containment with eradication as a second thought, then the actors will twist and turn around containment measures. In one breach case, the actors used the same backdoors that were dormant on the network for months. While containment monitored the actors actions, the actors were still able to penetrate the network at will. In another breach case, eradication started even before my team could get on scene; the IT staff was deleting malware from the web servers ad hoc. By the time my team arrived, the actors had successfully edited security logs, turned of logging on some servers, deleted their malware, deleted their aggregation files, and successfully wiped their tracks. At this point we were unable to determine the full impact of the breach.
During large breaches coordinating both containment and eradication can take a huge effort, but I have learned that there has to be a single leader in charge of coordinating the activities of both. This person’s sole responsibility is to ensure that containment is progressing at a pace equal to the effectiveness of eradication. The individuals performing both containment and eradication must receive and adhere to orders from this coordinating leader. No unilateral actions should be taken without first reviewing it through the coordinating leader.
The breaches I am using as examples were large in scope. When doing containment and eradication for large breaches, actions cannot typically be made within seconds. Careful planning and thorough action steps should be established prior to pressing any buttons. And therein lays the rub: how do you react quickly while trying to be effective? The answer is efficiency and experience. Organizations should ensure that they are partnering with a cybersecurity firm that has the experience in working large breaches. There are plenty of firms out there that have experience working on many small or medium breaches, but only a very small handful that have consistently worked very large cyber breaches. My organization is one who has worked some of the largest cyber breaches ever.
2. Do not over manage your technical staff. During two large breaches of the past 18 months, I sat as the IR advisor for the company affected and the IR third party hired to do the containment, investigation, and eradication. The two breaches demonstrated two very distinct methods of managing the IR third party and internal security staff. One method was good and one was bad.
The bad method was where the upper management (i.e. Director of IT and Senior Manager of IT Security) would hold daily meetings with all of the IR staff, leverage a project manager to list all IR tasks being performed by anyone and everyone, and request daily status reports from every task group (there were over 8 task groups for the IR). The result was an IR staff that spent 75% of their day either in meetings or preparing for meetings. The productivity of the group plummeted to lows I had never seen before with an administrative document file share with thousands of documents: probably only about 15 documents, but 100 versions of each document or updates to each document.
I realized that the management either a) did not trust their staff or the third party – least likely, or b) felt that because their jobs (right or wrong) were on the line, they should know everything at all times and micromanage the entire process – most likely. During the first hours and days of an IR, time is of the essence. To the detriment of efficiency and ultimately effectiveness of containment, the upper management performed poorly.
The good method was a case where the upper management had a long standing relationship with the IR third party and trusted them to get the job done. The IR third party had a single leader for the engagement who met with the Director of IT once per day for about 30 minutes. During the daily meeting, the third party leader would give the Director of IT an update on what occurred over the past 24 hours, what the teams were doing today, and what we expect over the next 24 hours. The meetings were exceedingly efficient and the Director of IT took notes. These notes would then be put into a report that the Director of IT could take to her superiors on a weekly basis.
Using what I call the good method above, the breach engagement took only a couple months to contain, investigate and eradicate. Under the bad method above, the breach engagement required over 6 months to contain and eradiate. Six months should be unacceptable by anyone’s standards.
3. Find and use a system to evaluate third party vendors. In at least one case over the past two years I have seen where a third party vendor was the point of infiltration into a client’s network. Once my team figured out the point of infiltration within hours of arriving on scene, I asked the client for copies of their third party vendor agreements, MSAs, SLAs, documentation, evaluation criteria, and evaluation results. They had nothing. They were sure they had vendor agreements, but did not have copies of them available. They would have to go to the vendors to get copies of their agrements.
Obviously there are multiple lessons learned from this situation. First, keep all over your vendor documentation in one place should you need it for say… I don’t know… maybe a breach that involves a vendor. Second, perform evaluations based on the type of agreement you have with vendors. If the Target breach taught us anything, then it would have to be that third parties present a credible risk to the security of an organizations cyber infrastructure.
Just as a threat manager or head of IT security would do as a new hire into a new position at a company: identify all computing devices, quantify their risk, and develop a plan to mitigate threats. So too with third parties; all organizations should identify all third party vendors, quantify their risk to the organization, and develop a plan to mitigate any threats from these third parties.
This is not a process that you write on paper, say you have it, and then let it sit in a binder on the shelf. Third party management should be as commonplace at your organization as patch management or change control. For sustainment, third party management should be included during evaluating third parties during acquisition and evaluated on a very periodic basis to address new threats and changes to the organizations infrastructure that may impact the third party.
Incident Response organizations are no different than any other third parties. In almost every large breach case I’ve worked on, my organization has replaced another IR firm that screwed up. In most cases, the outside legal counsel is the one who made the decision to change IR firms as they had the experience to see right from wrong. For many organizations that do not use outside counsel and manage breaches themselves, they are less likely to see the errors of the IR firm for which they are paying a lot of money.
As with traditional third party vendors, third party IR firms should suffer the same scrutiny when it comes to basic expectations for performance. Aspects such as amount of time to do containment, process for eradication, investigation process, escalation measures, communication plans, reporting, and operational security should all be criteria for choosing an IR firm. What you don’t want is an organization that likes to go on the news touting that they know of this bad guy and that bad guy somewhere in the world that is hacking So-and-So’s Retail Chain. IR firms that leak information to the press hurt brands. It makes the IR firm look great, but kills the stock price of the companies they are trying to protect (supposedly trying to protect).
4. Stop “saying” you’re prepared. Everybody is saying it, companies are getting breached, cyber is everywhere, and the bottom line is security is a necessary evil. Organizations should stop saying they are being prepared and actually be prepared. In every single large breach that I have worked over the past decade, all of the organizations were not prepared to respond to the incident. None of the organizations had dedicated incident response staff and all of their security personnel performed incident response as an additional duty.
If incident response is an additional duty, then it will never get done. Everybody has normal daily activities that require 110% of our time. Additional duties consistently get put on the back burner. If incident response is an additional duty at your organization, then incident response will be put on the back burner. When a breach happens, then you won’t be prepared.
During one case last year, I would walk up to a SOC person and ask them if they are seeing traffic on 443 with PDFs going to overseas locations. The SOC person would give me one or more reason as to why he wasn’t looking at those ports. In one instance the SOC person stated “I am working on getting our Blackberry server back online because the CIO can’t get his email.” In another situation the SOC was working on a domain controller that went down because of an erroneous patch that was implemented.