New business processes (and technologies) too often throw auditors into a tailspin. It’s no one’s fault, really: the decision making on what IT and process controls that need to be in place moves slow – and it is the tail to the rapidly moving technology dog. In fact, while there is no way a standard or regulatory body can predict technological change from year to year, there are ways to help ease the relationship between auditors and organizations embracing DevOps and continuous deployment practices.
“DevOps audit is an interesting problem,” says David Mortman, distinguished engineer at Dell (formerly Entratius). “Audit is something you just can’t avoid, especially in any industry that’s regulatory controlled. And auditors are a lot like security people, in my experience, which is to say that they have a lot of responsibility and they are justifiably concerned about things they don’t understand,” Mortman says.
And concern their certainly is when it comes to any organization that must deal with acronyms like HIPAA, PCI DSS, SSAE 16, SOX, FedRAMP, among others means few things will raise auditors’ eyebrows more than hearing that developers are performing deploys on their own code.
That’s why it’s important to know how to manage these audits, and continuously improve the audit process.
Rapid technological change, glacial control placement
One of the first things to note is that these standards and management programs move slow. “If you are familiar with ISO 27001, which is a security management program, they just released a new version at the end of September [2013]. From 2005 to now, the standard hadn’t changed,” said James J. DeLuccia, a senior manager in the advisory services practice of Ernst & Young LLP in his Flowcon presentation, Successfully Establishing and Representing DevOps in an Audit. But technology certainly has changed in that time: increased use of virtualization, the rise of cloud computing, ubiquitous mobile, as well as changes in development and operational processes such as continuous deployment, continuous integration, and DevOps.
One of the biggest issues that come up with disciplines such as DevOps and continuous deployment is separation, or segregation, of duties – that development, testing and operations should be separated duties. Not only do many government and industry mandates require it, so does many enterprise security policies. That’s why DevOps, when not explained or approached properly, can create bug-eyed expressions in unfamiliar auditors.
The goal of separation of duties comes down, essentially, to proper change management. If the developer who wrote the code can deploy the code into production, and they are given too much access (such as root) or shared access, there’s a lost control check, and potentially no log of the event. But it’s a concern that is overstated, experts say.
“In a proper DevOps environment, the person deploying the code doesn’t need to have root access. There’s no reason why, when properly setup, developers can’t logon, push the deploy button, and create an auditable log that says at 2:32 on Friday the 18th, David deployed a version of the code. And you have it down to the hundredth of second who did what,” says Mortman.
As DeLuccia explained in his talk, when auditors are asking to see change control logs they are really trying to see that these controls are in place. “They’re really asking for help to understand this linkage,” he said. Depending on the policy and requirements, the auditor needs to see reports that demonstrate, in a statistically relevant way, that everything in IT is working the way it needs to and is said to be working. “Because remember, when we’re putting together these reports, we’re trying to say, everything is working the way you say it’s working,” he said.
Auditing DevOps: An education process
When done right, Mortman says DevOps and continuous deployments create more accurate logging and change management. “You have a log of every change that’s ever happened and it’s all generated automatically. You don’t have to worry about someone changing something. When setup properly, they don’t have the access to change anything other than the very limited roles of what that application they can deploy to,” Mortman says. “So in a lot of ways, in fact you’re actually maintaining better control over your environment.
The challenge there, as DeLuccia explained, is that audit programs are built a year in advance. “When they come to see you, they’re already have it planned out based off last year’s risk. That’s already too late [to keep up with rapidly changing technology and processes]. You really want to plan this a year in advance,” he said. The goal, he explained is to become part of the process and show how the procedures they are asking the IT team to execute against are in line with reality of the processes and the technology on the ground.
“Much like any other industry, you have an education problem,” says Mortman. “Security people run into this all the time. The business will say, “Hey, we want to do this new thing, and the security people reply, “Whoa, hold on a second. You can’t do that. That’s not secure. Or, we can’t audit that.” So a lot of the challenge with DevOps is an educational challenge. And you need to educate auditors that the control processes they are looking for are in place and that there are adequate logs that can be audited and evaluated,” says Mortman.
