It’s 4 AM and the alarm reminds me that I’ve been awake for an hour or more after grabbing a couple hours of shuteye sometime after midnight. The realization of a 45-minute drive, with a stop off at the office to pick up equipment, breaks my intense contemplation of what it was like to have regular hours. I meet my fellow investigator at the office and we head to the train station. Some eight hours prior, we learned that a company had been hit with a very nasty malware attack sometime in the previous few days. The story was familiar and both I and my compadre could relate the basic details without even hearing them.
A number of employees received a phishing email designed to look like benign financial communications. The emails contained attachments purporting to be financial documents and were designed to look like they arrived from an innocuous sender. The emails were sophisticated and convincing enough to pass a quick skeptical glance and several employees opened the attachments. As it turned out, executed was probably a better description of what was done with the attachments. One of the employees was using a workstation vulnerable to the malware attachments and the malware did its job which resulted in a large number of important documents being rendered inaccessible. Therefore, the client decided to bring in outside consultation which brought me to rolling out of the rack at 0400.
At 6:30 AM, the sun is finally up as the train rolls out of the station. My partner and I luck out and get a decent workspace at the front of a car. We decide to make the most of the ride by going over what we know about the incident. Luckily, my MiFi works relatively well on the train. The client’s description of the incident provides plenty of fodder for research. As we discuss the details, I have to keep the investigator in me at bay. My instinct is to drop into the client site and deep dive into technical details in an attempt to fully understand the malicious activity and associated malware. Instead, the more practical approach is to determine what questions to ask and prepare to listen to what the client describes, and perhaps more importantly, listen for the outcome the client desires. This preparation time is crucial as it is nigh on impossible for my team to anticipate every type of incident and have a checklist or process guide to follow in every case. This particular incident involved crimeware.
The objects of work my team is regularly called on to do run the gamut of nastiness that can occur to an enterprise, from insidious insider theft to national level industrial espionage. The only thing that stays the same is change. Ultimately, my partner and I decide on a set of basic questions to ask, define our individual roles and map out a possible schedule of activity. We operate on the notion that we may have one day or several days to provide the consultation necessary to satisfy the client’s needs as well as to perform any forensic analysis that may be required. The clients, usually businesses, are the ones who experienced an attack. It is the clients’ data and resources under their control that is at risk. Client leadership has to run the business regardless of the need or desire to determine what happened to them and why. They frequently answer to shareholders and, in the grand scheme of things, have real people in the form of customers and employees to be concerned about. Business disruptions caused by incident response activities negatively impact customers. Employees can be negatively affected if the health of the business declines due to the same disruptions. So, when it comes to prioritizing recovery activities, it only makes sense that client needs are considered side by side with accepted and acceptable incident response actions.
At a quarter past 9 and after a long cab ride through extended rush hour traffic, we arrive at the client site. Despite the unusual coolness of the early summer day, I am sweating a bit. This is probably the most stressful part of an engagement for me. Frequently, and understandably, I am the last person a client wants to see. Context, if it’s not obvious, serves to illustrate why this can be. It is not that the client dislikes bald slightly overweight graybeards with warped and raw senses of humor. Nine times out of ten, the client is in the middle of a potentially company altering event. The future of their business may be in question, or at least it seems that way. They are operating on little sleep and enduring endless meetings and calls with everyone from counsel to vendors to customers to the CFO. Not everyone with whom a client engages has their best interest in mind. In many cases, a client has been advised to retain my team by someone not overtly sympathetic to the immediate impact to their business operations. In a client’s mind, I could be the messenger of negativity: “… this is how this happened to you. It’s your fault. You could have prevented this … woulda coulda shoulda.”
Fortunately, while some in the industry serve this critical function, mostly from afar, my team’s role is different. I usually have about ten minutes or less to begin to convince a client that they can rely on my team and that we are, pardon the cliché, there to help. The history lesson, save for leveraging information useful to preventing future incidents, is between my team and the client’s team. In this case, the client was welcoming. My seniors had done an outstanding job of assuaging the client’s concerns in talks prior to our engagement. The client’s IT chief and CTO, despite an obvious lack of sleep and an endless task list, make plenty of time to talk with us and we proceed to learn what happened.
It’s 10:30 and the client providing a quiet private area for the now combined client-consultant team to work and confer is very welcome. As with most clients, this client wants to keep the commotion and negative impact of consultants, analysts and investigators roaming around his workspaces to a minimum. The basics of the incident were as we previously heard: Two days prior, a series of phishing emails arrived with malicious attachments; one successful malicious detonation was confirmed; and the attack rendered several hundred megabytes of important files inaccessible. The client’s primary immediate focus was recovering inaccessible data.
After lengthy discussions, the only way, we and the client could think of, to meet the objective of recovering inaccessible data involved leaving the affected system running. In the very short term, there were a few things the client could do to the enterprise, quickly, to alleviate some of the risk presented by leaving the affected system as is in order to attempt data recovery. While client personnel were calmly and deliberately attacking the problem, it seemed important to convey that yes, something bad happened, and yes, it probably should not have happened, but it just does not matter right now. The team should focus energy and effort toward meeting immediate prudent defensive and reactive goals. It was going to be okay.
It was heartening to see that the client leadership was of the same mind. Even still, at that moment, validation of these thoughts was a good thing. My partner and I now faced a couple immediate considerations with the potential for dilemmas down the road. How much investigation and analysis was necessary to meet the client’s immediate goal of recovering inaccessible data? How much was necessary to illustrate due diligence, if any, was necessary? How much investigation and analysis would result in the best chance of preventing a similar incident in the future? Would adherence to best practices be at odds with client wants or needs? What exactly could we provide that the client did not already have? A quick review of the following runs through my head, as we discussed possibilities:
- Determine as many facts and circumstances as possible
- Identify the primary problem
- Identify the most important outcome
- Identify the client’s concept of the most important outcome
- Consider future events and sustainability of solutions
- Consider all possibilities
- Confer with others if possible
- Balance client needs and resources with those required by possible solutions
- Compromise where possible
- Make a decision
After the client started immediate defensive measures, the whole team rallies again at half past noon. The consensus client desire, at this point, is to do everything possible to recover data first. This presents a decision point with potential tension. Best investigative and defensive practices tend to emphasize data collection, analysis, countermeasures and preventative actions developed via analysis. Data recovery, save for data directly associated with malicious actions, is inherently secondary to this investigator. On top of this, unknowns and doubt permeate the situation. Common sense, given the particular circumstances, says the chances for data recovery are slim. However, the actual chances for data recovery are unknown. Perhaps paradoxically, the extent and nature of the associated malicious activity is not truly known. This is a classic “whatch ya gonna do Vern?” moment.
The irretrievable data is very important. The data is so important that not trying to recover it is akin to a direct assault on the client’s business. This seems to be a case of possibly winning a battle (pressing forward in a safe, completely best practices way) only to lose the war (never knowing if the business data was recoverable). The recovery window is short. We figure we have the rest of the day, at best.
Well, the moment is here, we have an idea of what we do not know (the scope of malicious activity and the chances for successful data recovery). We go for a compromise. The compromise is to finish applying short term protections to the environment and gain some basic network visibility while attempting recovery. After agreeing to concentrate on recovery in a couple hours, we do a quick ready-break and go heads down, irons in the fire. We’ll reassess where we are in 90 minutes.
Two o’clock comes and the appointed time to make an up or down call on continued data recovery or switching focus to defense and investigation arrives, but we’re not ready to make the call. Network visibility is not as robust as we would like. A closing window of opportunity based on reduction of impact to the client’s current operations and lack of visibility dictate a delay in potentially destructive actions. Further research into what others experienced when dealing with similar attacks and consults with the mother ship (other investigators and leadership at our laboratory) do nothing to increase confidence in recovery. However, the path to data recovery is chosen. It would be counter-productive to be anything but positive. At this point, I am not sure if I am channeling Admiral Farragut or Captain Bligh or both.
At 5 PM, active data recovery attempts arrive at a watershed. We are actively attempting to interact with involved malware while watching for signs of undesirable system and network activity and, ultimately, for signs we are recovering data. The next 30 minutes or so should produce recovered data, or not.
An hour later it is becoming more and more clear that we will not recover the desired data today, if ever. The attackers’ destructiveness was robust and Lady Fate is throwing running smoking chainsaws at us. The devil gnawing at my mind is gleefully reminding me that the client’s network is exposed more than it has to be. My partner and I start to float the notion that the team’s focus should shift to more defensive efforts. Of course, there is always one more thing to try.
At 6:30, the peak of the watershed is banging on the door. It is time to evaluate the situation and determine a way forward. Given the lack of success in recovering the data we wanted, the time of day, the clear need for the team to take a break, and consideration of available resources, it is time to lock things down to an acceptable level. Essentially, we want to identify and remove any affected systems from the network and concentrate on protection going forward. Investigative data will be collected as practically as possible. However, the focus is shifted to defense. This is one of those moments where we definitively earn our keep. While the client, at this point, is in agreement with the lockdown, we provide an outside view of acceptable risk that most times tempers a client’s, perfectly understandable, desire to meet goals that may be at odds with security measures.
At 8:30 PM, the lockdown is done. My thoughts drift toward more creature comfort matters. We had just contacted our Program Manager and asked her to acquire some lodgings with the thought that we might be needed for a bit more at the client site the next day. However, there is a train in less than an hour, there really is not a whole lot more we can do at this phase, and the client is good either way. The lyrics from a classic Clash tune reverberate a bit: “Should I stay or should I go?”
A quick consult with my partner indicates it might make sense to head back to base. We take another long cab ride through seemingly chaotic traffic that moves in fits and starts, make a few calls to cancel our reservations, and get clearance to return to the mother ship. Our clumsy navigation of the train station maze with our hastily packed gear in tow leads us to our train, which is much rougher than the morning accommodations, but functional.
While on the train ride back, my partner and I review the very short engagement and proceed to solve all the world’s intrusion problems in a way that’s only possible after way too many hours without sleep and on-the-road sustenance can provide.
Bleary eyed and by this time completely exhausted, we arrive at our train stop at half past midnight. It is funny how long a ride can seem on a return trip. After a short detour to the office to switch vehicles, I am on my way to what I hope is at least a few hours of rack time.
At 2 in the morning and after one more adrenaline rush as I witnessed two deer scampering across the highway with one getting clipped by a ridiculously large truck a few hundred yards ahead of me, I arrive at my abode. It is funny how quiet the world can be in the wee hours. I didn’t quite manage a 24 hour day, but I got close to one. One of many pangs of Monday morning quarterbacking hits me. Did we do the best we could? Did we provide good advice? Did we leave the client in better shape than when we arrived? At this point I remind myself, as I do many times, that many may find fault in our approach and our decisions, but they weren’t there when the client needed help and we were. Rock throwing is easy. Doing is hard. At some point you have to stop talking about it and just do it. Tomorrow, or is that today? ... is another day.