Security vulnerabilities are the symptom, lack of IT maturity is the disease; information security is not the cure to security vulnerabilities, IT maturity is.
It’s not unusual to see a company with hundreds if not thousands of known security defects, commonly called vulnerabilities, presents in their network, on servers and in applications. The tools to detect these defects are easy to purchase and run, the tools to deploy patches are readily available. Yet these well understood defects, these publicly documented issues that sometimes even have step-by-step instructions for how to use them to gain control of the affected computing system, remain unattended.
The IT security team will point in frustration and mutter cryptically about 0 days (pronounced “oh-daysâ€), exploit kits and malware as they tear there hair out in frustration; after all most of these defects can be fixed with the quick application of a vendor supplied and tested patch or the a minor configuration change. So why aren’t they fixed?
The IT teams will rightfully raise concerns about pushing a patch into production without adequate testing or that making a minor configuration change could have unknown impacts downstream. Worse, they may be resource constrained and choose to spend their time on activities they deem to be business critical (righteous shorthand for that which makes us money). The root cause of this always some form of resource constraint; not enough budget or not enough staff (which is really not enough money).
Some will argue that it’s all about helping management truly understand risk and then somehow all these problems will go away. Similar tropes will be trotted out about aligning with the business needs, driving awareness or implementing metrics. These are all important things to do but they’re not the solution. They may help get security defects patched, but they won’t drive the IT revolution overnight that makes it anything less than death by a thousand cuts for any security team.
——————
John the security guy visits infinite world …
For a moment let’s imagine a world where one has a supply of system administrators, network engineers and other technology operators that is always larger than the demand. Deploying security updates would be easy and perhaps even happen without prompting by the security team. Need to test a patch across various server builds or test a configuration change against multiple technology environments? No problem, just throw more people at the problem.
The utopia of infinite resources doesn’t exist, utopia’s never do.
Now let’s look at a slightly less utopian world, where we have more administrators and engineers than we have today but not a supply greater than demand. While not horribly overloaded let’s assume that there is just a bit more work than there are working hours in the day. A backlog will begin to form immediately, growing over time and eventually tasks will fall off the bottom of the list, forgotten and never to be done.
——————
If you’ve read The Phoenix Project by Gene Kim , the above should be familiar to you. If you have not read this amazing book, you should go do that now (the first half of the book is available for free).
——————
Producing long lists of risks and vulnerabilities is all good and well, but you probably have a list that is for all intents and purposes infinitely long. There are probably entries on that list that predate your tenure as at the company and possibly even your career as a security professional. This list of gaping security issues that keep you up at night is never going to disappear unless you fix the root issue. Your security defects are no more important than usability defects, support issues, failing servers or anything else that goes wrong in the world of IT. So if the system administrators are ignoring you it’s probably for one of two reasons: either they’re overloaded or you’re not integrated into their process. Why should you integrate into their process? You want them to do something for you so best to speak their language and use their packaging (when in Rome…).
Chances are, the “get something fixed†process is not well defined or not defined at all. So before you get on your soap box about these unresponsive and uncooperative BOFH s that don’t seem to care about security, go find out what the process is and if it doesn’t exist then interview some folks and get it mapped out. Process engineering probably isn’t in your job description but sometimes you been to engage in a little guerilla organizational improvement. With the processes mapped out, you can go about instrumenting, optimizing and finally automating them. Figure out how long it takes to apply a fix or any other sort of change then, see if you can make it go faster by optimizing the process and eventually automating it (seriously, who wants to do manual IT labour?). Now go use your risk ratings to figure out which ones to put in this efficient queue first.
Properly documented and optimized processes that leverage automation are the sign of a mature organization, anything less and you cannot secure any reasonably complex environment.
——————
In which John visits infinite world again …
Let’s go back to that utopian world of an infinite number of system administrators all hammering away at keyboards, not producing Shakespeare, deploying patches and closing security defects. In this world, there are also thousands of happy users making arbitrary changes to their workstations and applications introducing all sorts of security problems thanks to outdated and badly configured software. Fortunately the infinitely large team of system administrators dives right in and starts updating, reconfiguring and cleaning up this user introduced mess.
Again, the utopia does not exist, but let’s pretend that you have a team of system administrators just large enough to deal with open tickets and planned work; quickly the mess introduced by users (or less well behaved system administrators) begins to pile up and never goes away compounding into yet another form of technical debt .
——————
As a security professional, how do you tell the difference between a breach and an unauthorized change? How do you stop things from getting worse again once you’ve cleaned everything up? The answer is obviously change management, not just at the server level but everywhere across your environment. Not only will change management stop users (and system administrators) from reversing the cleanup but also help you determine isolate changes from security breaches. Without robust change management every unauthorized change could just as easily be a breach; you not only want change management in place to maintain your recently achieved healthy environment, you also want to enforce it as a mechanism to filter out the acts of intruders installing backdoors and keyboard sniffers from legitimate users installing approved software.
Mature environments have change management, not for processes sake, but for maintaining known good states. If your environment does not have change management there is literally no way you can be secure. As the security officer it’s your job to maintain system/data integrity in the face of bad actors regardless if they’re the unwitting user or the evil hacker. If changement management doesn’t exist, go build it and then use the same tools you’d use to find intruders to identify unauthorized changes.
——————
John’s last visit to infinite world …
One final return to the world of infinite system administrators that are consuming defects and following change management at near infinite speed. Some of the changes or patches to fix security defects are simple, already well tested by the vendor and completely safe to deploy without a seconds thought. There are some changes required that are not as obviously safe and their implementation may cause unplanned outages. Fortunately the infinitely large team of system administrators just piles into every unplanned outage and normal functionality is resumed before the users are even aware.
In the less than utopian world, the team initially keeps up with the torrent of patches and changes but every time a change goes sideways, a backlog appears and eventually you’re back to a mountain of unfixed problems as you struggle to fix the outages.
——————
The refrain of “we need to test all these security changes first†often feels like the passive-aggressive response from your IT operations team, a way to sideline you as they work on other stuff. There’s probably not even a place to test security related changes because there’s no non-production environments for half your systems; they were probably deployed and never touched again so where’s the need for test systems. The cynical among us will respond that the IT operations team who asked you to test first knew that there were no test systems to start with and they really are playing silly games. Let us however treat this as a cry for help and as good security professionals help push for those test systems; let us help make the environment more mature.
An environment without the ability to test cannot be kept in a good state; outages can look suspiciously like bad changes introduced by a clumsy intruder. Without a place to test changes, how do you know the change will actually produce the security outcome you want? You may just be introducing pointless work. Without a place to test changes, system administrators will aim for the easy target, the most recent security related change, as the cause of a current outage and roll back your hard work making the environment less safe. Without a test environment, security cannot iterate on changes to evaluate the impact of enhanced security against the user experience, operability and maintainability thereby resulting in technically good changes that annoy users and administrators to the point of rebellious bypass.
Once you have the test environment, implement automated testing scripts so that you can quickly determine if a security change (or any change) is safe based on objectives tests. If you can’t automate the tests then you don’t understand what a stable system looks like and you’re not solving the problem because you’re adding more work to an already overloaded system in the form of manual testing.
A mature IT environment can be a secure environment, an immature IT environment cannot be made (or kept) secure. If you find yourself in an environment with poor security make sure you invest in maturity before you spend any money on risk management or awareness training.
<FIN>
TL;DR: Create a mature IT environment that can be made secure by:
- Map all the processes security relies on in IT operations then (help) measure, optimize and automate them;
- Help enforce change management so you can keep an environment safe and stable;
- Build and automate testing of changes to confirm they are safe.
For a more though out and much better articulated approach to building a mature IT environment go read The Phoenix Project by Gene Kim , the above should be familiar to you. The first half of the book is available for free but the book itself is worth every penny so just go buy it.
(Originally posted on Medium.com)