devopsy.kinsta.cloud

Tag: audit

Don’t Let Developer Toil Affect the Business Value of Your Apps

It is no surprise that the world today is driven by apps. Organizations all over the world are basing their business goals on their capacity to integrate new apps quickly, and they need to constantly reimagine how they interact with and communicate with their consumers and employees to fulfill changing expectations. To do this effectively, businesses need to operate at a high level of speed and agility. But in many cases, this isn’t happening. The phrase ‘move fast and break things’ could not be further from the truth when it comes to how enterprises approach innovation today. In fact, according to Forrester Consulting, 48% of executives say they haven’t changed their applications in a year or more. One reason for this may be that organizations are dealing with a high amount of developer toil and technical debt. Developer toil is the repetitive, predictable, constant stream of tasks that support the creation of software and how it’s run, but don’t actually affect the features in the software that people use. Or in other words “any activities that don’t directly create business value.”

Typically, developers spend too much time on processes that could be automated or even eliminated. Ironically, this toil builds up over time as their organizations prioritize shipping features rather than addressing these problems in their overall software process. While developers can change their software quickly at first, just like debt in real life, if this toil and tech debt are neglected, it takes over and consumes the organization’s ability to grow. This results in long, infrequent release cycles. A feature that seemed simple and once took just 15 minutes now takes weeks, even months, to get in front of users.

Conducting a Developer Toil Audit

Anything you can do to speed up your software release cycle will improve the quality, resilience and business value of your software. Continuously addressing developer toil provides a significant boost to your organization’s software capabilities. For one of VMware’s customers, addressing developer toil reduced the time taken to introduce a new feature significantly–from 400 hours to just 24 by doing a developer toil audit and addressing key factors contributing to this technical debt.

So how can organizations overcome developer toil? One way to do this is through a developer toil audit. This is a systematic process for finding, valuing and prioritizing fixing waste in your software process. First, the process finds wasted time and process debt in how you build and release software. Second, the process then helps you justify delaying working on features in favor of eliminating and automating your software creation process. The result is speeding up your software release cycle. The more frequently you can change and release software, even with just small changes, the more opportunities you must learn what works and put those features in front of customers, employees, and other users.

The process of conducting a Developer Toil audit includes:

1. Asking the right questions: A great way to uncover developer toil is to ask about people’s struggles, frustrations, and even boredom in the form of a tailored question set. We advise using questions that are specific to your organization in addition to general questions like “how long does it take to perform a full build?” For example, highly regulated organizations should ask about tasks involving compliance. Once the survey is completed, you can do some quick analysis to locate and prioritize developer toil.

2. Combine into usable metrics: You now have a single metric for each type of developer toil. And by combining all the questions you create a single metric to track overall developer toil. Extra points for creating dashboards with visualizations! As with all such aggregate metrics, these are more directional than exacting—their job is to point you to problems that should be solved.

3. Link these metrics to business value: A simple ranking of developer toil is better than nothing. However, before deciding which developer toil to address, we recommend linking each type of developer toil to the business value created by fixing the toil. Put briefly, the value you get will be related to the time saved, and, thus, the ability to ship more features in the future.

The less time developers spend on toil, the more time they can spend on tasks that directly benefit the business. Another way of looking at this is plain old productivity: Developers can now do more with the same amount of time.

Another important outcome is increased staff morale. The less time developers spend on boring, repetitive work—toil—the happier they’ll be. Stronger employee hiring ability and retention are (or should be) a strategic imperative for any organization that depends on software. Morale also increases software quality: happier people make better software.

Slow Down to Speed Up

With the survey results in hand, you should have a pretty good idea of which developer toil to fix. The last step is to do the usual product management prioritizing to weight fixing these items with other tasks you could do. These are strategic decisions that product managers need to make. To fix toil, you need to stop shipping features to free up time. You need to slow down the business. At the start, before product managers have been empowered to make these kinds of decisions, higher-level management should be involved to calibrate how much slow down you’re willing to put up with for future agility. The calibration you’re doing is this: If I fix one item of toil and ship just one feature (instead of two) this release cycle, then in the next release cycle I can ship four features. Humans are not great at that kind of bird-in-the-hand versus birds-in-the-bush thinking, so you’ll need to figure out what works in your organization.

Through our experience as consultants working with product teams in different industries, we’ve used Developer Toil Audits to focus and motivate product managers and developers to fix toil and speed up their release cycles and, thus, improve how their organizations build software. This directly improves the business by adding more capabilities and capacity to deliver more features, even in the short term. Each time you fix any technical debt, you gain more capacity and capabilities to create new features and improve your software. This is how you use a modern software culture to increase business agility. Or, put more simply: Less developer toil leads to better software, and better software leads to better business.

August 8, 2022
Is DevOps Transformation a Holy Grail?
Both development and operations teams are popping up in enterprises all over the world, and even though DevOps programs are still in the early stages, our ultra digital world calls for even more software applications. This is why many analyst and consultants alike believe that, in the near future, DevOps will be even more powerful than legacy IT. DevOps will be something businesses of all sizes will not be able to live without and the sooner you initiate the DevOps transformation, the better.

Why DevOps Transformation?

One of the most exciting aspects of DevOps is enacting technology practices and architecture that help organizations create an agile and continuous product flow, from development to testing, operations and all the way to deployment while increasing stability, reliability and security at the same time.

More importantly, adopting DevOps can enact changes in the culture of your business, such as allowing teams to work more quickly with more independence. Back in the days before DevOps, deploying even a minor update, upgrade or bug fix would involve literally hundreds of developers doing weeks worth of work.

Today’s top-performing businesses and DevOps adopters are deploying much faster, and the products they release generally have greater reliability. Still not convinced that DevOps is the holy grail? Consider the following.

Focus on Speed, Not Cost

Do you want to achieve the same capabilities of Amazon and Netflix? Of course! Who doesn’t? It’s easy to make an excuse that they have a lot of money to spend, but, when we look deeper, companies that achieve DevOps success are more focused on optimizing speed, not cost. Organizing and optimizing for cost creates functional silos for networking and database people, server admins and many others.

When you optimize for speed, you are willing to cultivate talent from within, increase engineers’ salary and send them to conferences. You would not necessarily do these things if you were simply optimizing for cost. There are a whole bunch of techniques we can use to optimize for speed, such as using small, cross-functional teams that can deploy value to the customer independently without having to rely on hundreds or, in some cases, thousands of people.

Best of all, you don’t have to be a big conglomerate to make this transition. Even small and medium-sized businesses can choose to optimize for speed, break down silos and get their product to market faster. After all, this is what DevOps is all about.

Software Speed vs. Quality

Businesses that rely on developing and deploying software to support or supply their customers often are faced with choosing speed or quality. If you go to market before your competitor(s), you could be cutting corners in terms of testing and quality assurance, which can ruin your launch. However, if you wait until your code is perfect, you could end up lagging behind in the race for the next big breakthrough. Fortunately, the DevOps approach can resolve this dilemma while improving the bottom line.

Way back in 2015, the “State of DevOps Report” collected the DevOps health and habits of more than 20,000 tech professionals and found that top performers using DevOps show more agility and reliability in their releases and they were twice more likely than non-DevOps users to exceed productivity goals. In the roughly 800 companies that had a stock ticker symbol, the top performers had an almost 50 percent higher market capitalization growth over three years. This indicates that DevOps is great for IT performance as well as organizational performance.

What everything boils down to is that all modern businesses are increasingly more reliant on technology when it comes to acquiring new customers and bringing value to those customers. It is becoming increasingly apparent that agility and speed are key attributes for companies who want to win in today’s marketplace.

DevOps Transformation Guide

With all of the benefits listed above, you are probably eager to get started with DevOps. Here is a simple step-by-step guide to assist you in your DevOps transformation.
1. Conduct an Enterprise Readiness Audit – The highest probability of a successful DevOps transformation is when it is championed from the very top. The participation of business leaders is just as vital as operations, security and development teams. To get everyone on the same page and to ensure that your DevOps transformation gets off to a good start, seek assistance from an experienced, external consultant or coach. The purpose of this first step is to get executives to buy in, create mutual goals and get an understanding of what an Agile-DevOps program looks like.
2. Create a DevOps Center of Excellence (CoE) – One of the most important things you must do here is designate an Executive DevOps lead with enterprise authority. There is no use in creating this Center of Excellence at the incorrect organizational level. It must be spearheaded by someone who has the support and trust from all branches of the organization including delivery and vendors.
3. Create a Program Governance – Such a program will include creating a communications plan, an enablement program as well as establishing program KPIs. In terms of making a cultural impact on your organization, this is perhaps the most important step. The roles and responsibilities of practitioners will change with Agile and DevOps and it is vital that they understand collaborations between structures where historically animosity could have existed so they can break down these organizational silos. The KPIs will also need to be transitioned from individual metrics to holistic customer business results.
4. Create a Project Intake Process – Project intake really sets the tone for how successful each new sprint that comes into the program will be. What matters here is not the tools and process, but rather the proper sizing of the tools and process. There is a need to fit for purpose when selecting accelerators, toolchain and development methods. Once it is created, the communication of the intake process needs to be disseminated across the organization. Keep gathering feedback and and continuously evolve and mature based on this feedback.
5. Identify and Initiate Pilots – To achieve the highest probability of success, apply this step to a specific portfolio of applications or a domain solution. The purpose here is to create a value-stream map of a specific application with company wide participation on all levels. This needs to be done with the necessary amount of detail to identify manual and automated processes, end-to-end process tooling and the people and skills that are involved.
  The next aspect of the intake project is the collaboration between the subject matter experts from the Center of Excellence with the project team. The goal is to advise on “to-be” processes with proper tooling, reusable assets and automation. As the application goes through the development and test cycle, be sure to identify and capture the KPIs for the program in order to compare with existing KPIs.
6. Scale Your DevOps Program – This final step involves taking feedback from the pilot metrics and scaling them by taking different application portfolios and running multiple release trains. It is not enough to merely do automated deployments and daily builds. The last, but by no means least, piece of the DevOps puzzle is constant feedback and optimization.
Key Takeaways
- To keep up with top-performing businesses and stay alive in today’s ultra-competitive, fast-paced environment, implementing DevOps is an absolute must. DevOps is not solely reserved for giant enterprises such as Amazon and Netflix and can be implemented in any-size organizations.
- In the not too distant future, DevOps will become something businesses of all sizes will not be able to live without. To reap all the benefits DevOps has to offer, the sooner you make the transition to DevOps, the better.
- With DevOps, there is no longer a reason to choose between speed and quality when deploying your software. When implemented properly, DevOps will allow you to deploy software with both agility and reliability increasing your customer base and the bottom line.
- The step-by-step approach mentioned above can and should be implemented in any flavor and variety that best suits each individual organization’s needs and constraints. Pay particularly close attention to large gaps or excessive deviations as they are red flags which will need to be corrected.
About the Author / Stanislav Ivaschenko

Stanislav Ivaschenko is a certified AWS solutions architect at Squadex with more than 10 years of professional experience in building, delivering, supporting and optimizing a broad range of software applications. From bare metal to the cloud, from monolith architecture to microservices his expertise and experience help to meet business needs for startups, mid-sized companies and large enterprises. He has experienced a large number of DevOps and infrastructure related tools, services and management practices.
December 11, 2017
The Automation of Compliance

Consider how much money your enterprise spends on audits. Smaller firms not subject to external regulation should consider how much money is spent on quality assurance—or the lack of it. A business that suffers from a customer-facing quality incident will, at minimum tarnish the brand of its services or at the high end, lose market share altogether. Larger firms that fail audits can suffer from brand tarnishing as well as high fees, not to mention further scrutiny by regulatory agencies.

Failure to perform or failure to follow proscribed processes inevitably leads to a black eye in front of customers. The entire discipline of governance is premised on avoiding both. Whether your governance practices are efficient or not—and whether your “process” as it sits today leads to quality or not—is not the subject of this review; it is how DevOps and automation can shine a spotlight on both and radically reduce the costs you spend today to achieve them.

The Spotlight of DevOps

The spotlight of DevOps begins with consistency that leads to predictability. Automating any given kind of process or technology the same way every time begins to give insight on any “holes” from things that do not work (or work well). If all that changes in your deployment process, for example, are pointers inside a configuration file and the kind of testing executed against this version of the code, you have achieved a build-once, deploy-many state of deployment automation. Operations staff who support the deployment of change or maintain the environments in the software development life cycle (SDLC) begin to isolate any issues faster because they become familiar with “how” a given kind of technology is automated to deploy. This reduces the total number of deployment errors and, in tandem, reduces the length of time it takes to fix them. Any “process” problems or weaknesses in your governance model become apparent quickly and can be fixed quickly. Limiting variation is the medicine to keep them away in future.

The spotlight of DevOps next turns to transparency. Offering your entire enterprise visibility or “read only” access to your entire DevOps tooling suite has a profound impact on the transparency of change. As an example, consider that I am a humble accountant in your firm who has submitted an agile story to improve our billing system. I may or may not, given the nature of my job, be co-located near the programming team who will fulfill this request. If our firm specializes in accounting software then perhaps I will co-locate, but if I am only requesting a change to an ancillary or support system, likely not. Without direct access to the programmers through traditional agile co-location, my visibility into the progress of change is limited to the tracking system I submitted the request within (assuming you have one, and don’t use email to do this).

If I know a programmer in MIS, perhaps I ask him at lunch how my change is coming along. But if I have read-only access to the testing systems, or to the release orchestration (RO) tooling, I may get a much better up-to-the-minute view of exactly where my change is, how it is doing from a quality assurance (QA) perspective, and when it is scheduled for release into production. Based on this news, I may make decisions about how we bill our customers in what time frame to increase overall revenue to the company. I would notify my boss of same (who also has access, but pays less attention), and of any changes that impact the new timeline I create. The marketing team (who also has access) begins to work with public relations (PR) and the advertising guys on customer announcements, etc. etc.

The net impact of having universal transparency is that audiences who are not traditionally notified or interested in the inner workings of change movement now make decisions based on it. Consider that prior to this systemic access, and the reliability of DevOps to ensure delivery, the only news you got before was from “the IT guy,” who was often wrong. It does not take long before IT loses credibility from being wrong about delivery timelines, and these other audiences begin to pay less attention (if at all). At that point, an “I’ll believe it when I see it,” attitude emerges. And this same attitude infects internal auditors because of what they hear and see from disaffected internal customers who have been misled repeatedly by the failed manual methods of the past (pre-DevOps).

Processes that are disinfected over time by the consistency of DevOps begin to change the internal company thinking about the delivery of innovation—namely that it can be counted on as accurate. Transparency offered to all audiences of a company increases the visibility into all aspects of innovation delivery and better ideas emerge, both about what to do with completed efforts and how to improve the innovation process itself. And before you know it, your governance and QA processes become efficient, reliable and predictable. Variation is eliminated or greatly reduced, and the costs of operation and innovation begin to decline.

The Forgotten Stepchild of Automation

What is rarely a matter of focus in most companies is the secondary ability to automate the compliance history of a given application. Most firms are content (translate, only able to) work with their testing tools to produce logs about a given version of an application now residing in production. A test log does show an auditor the majority of the quality related information about a given version of an application. But the log is a highly limited view of what has taken place. What is needed is the full context of innovation. Enter compliance automation.

Consider for a moment, that requirements drive the train. Why are we doing the innovation we are doing? Is this a bug-fix, a new feature, a client request, a regulatory requirement? Being able to trace the code itself back to the requirements system that drove the need adds significantly more context than is possible from mere test logs. Next, think about the number of attempts that finally resulted in this version of the application. An automated build tool (part of our DevOps suite) has this information. At this point you could evaluate how many builds and deployments took place in development before a worthwhile candidate emerged to traverse the remainder of the SDLC. An automated deployment tool (part of our DevOps suite) can trace the number of environments this version of the app was deployed into for testing and how long the overall process took to get something moved from development to production.

And finally, testing is often done from a variety of tools, with results compiled in a variety of result repositories. This information may be available from a solid RO tool that tracks the gated progress of this version of the application, but if it’s not, it still should be sitting in each of the testing tools used throughout the process. Depending on the level of inherent DevOps tool integration (i.e., using a vendor suite) much of this information may “flow” from one tool to the next. But if not, the information does still exist in the logs produced by each of the DevOps tools.

It may be well worth the application of internal programming resources to put together a tool that gathers all this information (automates the collection and presentation of) for audit purposes. Imagine the reduction in cost of supporting an audit that can gather a simple push-button report (or dashboard) of the entire life cycle of a given version of change. At that point, you have the entire context for a piece of innovation. You know what drove it (requirements). You know what it took to create it (build and deploy). You know the entire history of testing it (testing tools). And, finally, you have the entire governance process used in its deployment to production (RO tooling). But instead of examining multiple tools to collect and present it, you have created a push-button system to do so.

Impacts to the Bottom Line

The investment in adding automation to the compliance history of innovation pays off in more than one way. It is not just the ease with which history can be recited. It follows the same principles as DevOps itself. Automation introduces consistency in how audit information is gathered and presented (reducing variability from one audit to the next). Automation introduces transparency if the entire company is given “read only” access to this new system. Automation shows auditors we are not afraid to learn from our mistakes and proud of our accomplishments. And in effect, it reduces the need for audit altogether.

Typically, regulatory bodies investigate and audit firms where they are uncertain of how the results were obtained. An automated system like this one fully eliminates that need. Transparency as could be offered here, may convince auditors that attention is not warranted here. Keeping in mind the spotlight of DevOps has already improved quality and streamlined process, add to that full transparency of audit materials and the confidence of auditors regarding your company may increase significantly. Routine becomes the only reason to audit, rather than suspicion.

Invariably the “new” will sometimes result in an error that your current systems and processes (automated or not) will not catch. But this truism is recognized, and accounted for, in how DevOps enables innovation. And if you have this kind of automation in place, it can be demonstrated in how we fixed what we fixed (whether governance or QA). Your strategies in DevOps should help you achieve this level of confidence. (If you need help, feel free to contact me, I am currently looking.)

To continue the conversation, feel free to contact me.

May 26, 2016
Security Should Be the Top Driver for DevOps

I’ve often said that the driving factor for many companies in adopting a comprehensive information security program are the dreaded “F” and “A” words – FUD and Audit. Technically FUD is an acronym for fear, uncertainty and doubt. And it might be better said that audit is the action used to hopefully demonstrate compliance and trust.

Just a few years ago, the predominant drivers for security spending were regulatory and compliance requirements. While compliance remains a driver today, the primary moving force for many is the concern of cybercrime. Fear of being the next Sony, Target, Chase or countless other victims is driving information security budgets to be readdressed.

In a recent CA survey the top obstacle (28%) to DevOps in their organization were security or compliance concerns. Yet, in the same study, a huge percentage (88%) already have or plan to adopt DevOps in the next 5 years. The dichotomy of the situation has not escaped me. Organizations are in a situation where they are actively spending money on DevOps and information security, but at the same time view these two initiatives as counter.

The answer is simple, information security teams need to adopt DevOps principles.

While a widely agreed upon definition of DevOps is still up for grabs, its safe to say that DevOps is generally inclusive of a few core tools and principles. First, DevOps promotes a culture of cooperation and sharing among different groups in an organization. Second, DevOps promotes the use of heavy automation, decreasing time to market and agile development.

Information security’s goals are the CIA triad – confidentiality, integrity and availability. Security teams argue that DevOps is the antithesis of good security. Constant change, open culture and automation smack directly in the face of security’s tactics of compartmentalization and tight process control.

Here is a little secret – DevOps is winning, information security is losing.

DevOps teams are deploying code faster and faster. They are reducing time to market and increasing revenue. Information security teams, well, lets just say that they continue to be in the news for the wrong reasons.

The second little secret – the companies which are really winning are using DevOps tools and principles everywhere in their organization. Even in information security. Imagine overseeing infrastructure configurations with code that ensures compliance 24×7 across an entire organization. Or picture being able to instrument thousands of data points to benchmark security performance over time. Conceptualize being in charge of an automatic closed-loop security system which automatically takes action to mitigate attacks based on shared threat intelligence. These are possible with DevOps.

For all that DevOps provides, security should not be a hinderance to DevOps adoption. Instead, security should be a top driver for adopting DevOps. Using DevOps to create the next generation information security program might just be your only hope in combating the next cyber threat.

January 22, 2015
Q&A with Gene Kim: Bringing the auditors to DevOps

As more enterprises embrace DevOps, organizational disconnects often are created between what controls DevOps teams have in place and what IT controls auditors believe need to be in place. And if the right controls are, in fact, in place they absolutely need to be communicated to the auditing teams. Bridging these gaps is often painful for both IT teams and the auditors.

Helping enterprises to better understand what auditors need from them to do their work is the top-line goal of the DevOps Audit Defense Toolkit, the aim of which is to “define the authoritative guidance of how management and auditors should conduct audits where DevOps practices are in place,” according to the homepage for the effort. “By doing this, the DevOps Audit Defense Toolkit will elevate the state of the management practice, defining how to understand risks to business objectives, correctly scope and substantiate the of effectiveness of controls, which reduces the costs of audits and increases effectiveness of audits.”

The review draft of the DevOps Audit Defense Toolkit was release earlier this month for review and comment. The authors, James DeLuccia, Jeff Gallimore, Gene Kim, and Byron Miller, expect the final version to be available in coming months.

To get a deeper understanding of the Toolkit, we grabbed a few minutes of time with Gene Kim, author of The Phoenix Project.

Why do you think something like the DevOps Audit Defense Toolkit is necessary?
I think at its heart DevOps is transformative. It’s the way to increase the flow of work through the DevOps value stream. DevOps increases reliability, stability, and security. And it also helps the organization win in the marketplace.

Yet, when organizations actually embark upon this journey, probably the number one obstacle they encounter is that their “compliance guys will never let us do this.”

I think there’s some truth to this. Because of so much that’s being proposed when you embark upon the DevOps journey, you actually take away some of the key controls that traditional IT organizations have used – like separation of duty, like change approval processes.

So when they hear, “Hey, I’m going to be letting our developers deploys whenever they want, without necessarily getting a change approval order from the change advisory board,” auditors, compliance managers, and security people often freak out.

And you have first-hand experience of seeing this in action, right?
It’s why I think it’s a very important effort. The more I talk to people, the more I hear about the friction between the audit teams and audit groups in organizations trying to move this way.

How do you see the DevOps Audit Defense Toolkit achieving this?

The DevOps Audit Defense Toolkit was designed to train. Since we can’t bring DevOps to the auditors, we must bring auditors to DevOps. What I mean is the goal is to train practitioners in DevOps how to think like an auditor and actually walk through what an auditor will do when examining a process and systems. That’s starting from the very highest level of the top-down risk assessment into the details, just as an auditor would go through the list asking what can go wrong.

The DevOps Audit Defense Toolkit shows how an enterprise has the environment that can prevent bad things from happening, and if we can’t prevent it, we can detect and correct for it. Then, it shows how we evidenced it. I think by doing all of that, one is doing exactly what the best auditors who examine DevOps organizations do themselves.

By training people in in the DevOps organization to think like auditors, we can actually bring the auditors along, show our work, connect the dots, and hopefully have the auditor become DevOps’ best friend along the way.

So, this is training IT how to think like an auditor and view the systems and processes the way that auditor would?

In many ways, yes. This is kind of a structured thinking process that any world-class auditor would do to actually create an audit plan and then audit fieldwork.

The toolkit asks questions like: Why is this being audited? Describe the environment Walk through the process of how code moves to production, and where the development environment comes from. It also asks: How do you prevent developers from introducing backdoor code into production; how do we ensure that untended code doesn’t make it into production?

I’ve learned a lot just working with the other team members, in terms of showing how we have a basis to say that we know what we’re doing, and that we’re really thinking through how to have an effective control environment.

That takes it from sort of the high-level platitudes and claims down to well, no, really, this is how we enforce peer reviews of code. We ensure that all deploys are put into the build systems. We run all the automated tests. Every time someone commits code, we run all the security tests before it can ever get into the staging environment. We can confirm that what’s deployed actually passed all the tests.

We communicate most effectively when we write down and show our work. And we’re exposing all that is required for an auditor to be able to look at it and say, “You know what? This is actually better than my best auditors would’ve done in the field, and you know what you’re doing.”

I would imagine that simply learning the language of auditors and how they speak is very valuable itself.

Yes, we’re now using a language that auditors use among themselves. Even just exposing what that language looks like, I think, is helpful to connect our own dots to be able to say, “Oh, I know why it’s being phrased this way. Tons of value, just by providing that.

Did you learn anything along the way as you created the DevOps Audit Defense Toolkit that surprised you?
It’s funny how much I learned by working on this. You would think the person who co-wrote the Phoenix Project would know what this would look like? But I learned quite a bit when I was looking at what were the actual business flows, and where that regulated data could reside, and what specifically are the applications in-scope for the compliance domains.

The second thing was going through the management risk analysis – to walk through the structured thinking process and look at how a company is really doing the diligence that is required, deciding what are its top business risks and if it really has a control environment that can prevent, detect, and correct.

Of course, we all know that DevOps makes everything better, but we need to know why and how to prove it, By what standards are you holding developers accountable? You talk about peer reviewing, but where is that documented? What is the standard? How is it enforced?

You want to be in compliance, and that’s important. But you also don’t want people to insert backdoors. You don’t want systems to crash when changes are deployed. So you really need to name all the dots, connect them, and then have that processed challenged. I have learned so much in this project.

June 30, 2014
DevOps: Getting Past Audit

New business processes (and technologies) too often throw auditors into a tailspin. It’s no one’s fault, really: the decision making on what IT and process controls that need to be in place moves slow – and it is the tail to the rapidly moving technology dog. In fact, while there is no way a standard or regulatory body can predict technological change from year to year, there are ways to help ease the relationship between auditors and organizations embracing DevOps and continuous deployment practices.

“DevOps audit is an interesting problem,” says David Mortman, distinguished engineer at Dell (formerly Entratius). “Audit is something you just can’t avoid, especially in any industry that’s regulatory controlled. And auditors are a lot like security people, in my experience, which is to say that they have a lot of responsibility and they are justifiably concerned about things they don’t understand,” Mortman says.

And concern their certainly is when it comes to any organization that must deal with acronyms like HIPAA, PCI DSS, SSAE 16, SOX, FedRAMP, among others means few things will raise auditors’ eyebrows more than hearing that developers are performing deploys on their own code.

That’s why it’s important to know how to manage these audits, and continuously improve the audit process.

Rapid technological change, glacial control placement
One of the first things to note is that these standards and management programs move slow. “If you are familiar with ISO 27001, which is a security management program, they just released a new version at the end of September [2013]. From 2005 to now, the standard hadn’t changed,” said James J. DeLuccia, a senior manager in the advisory services practice of Ernst & Young LLP in his Flowcon presentation, Successfully Establishing and Representing DevOps in an Audit. But technology certainly has changed in that time: increased use of virtualization, the rise of cloud computing, ubiquitous mobile, as well as changes in development and operational processes such as continuous deployment, continuous integration, and DevOps.

One of the biggest issues that come up with disciplines such as DevOps and continuous deployment is separation, or segregation, of duties – that development, testing and operations should be separated duties. Not only do many government and industry mandates require it, so does many enterprise security policies. That’s why DevOps, when not explained or approached properly, can create bug-eyed expressions in unfamiliar auditors.

The goal of separation of duties comes down, essentially, to proper change management. If the developer who wrote the code can deploy the code into production, and they are given too much access (such as root) or shared access, there’s a lost control check, and potentially no log of the event. But it’s a concern that is overstated, experts say.

“In a proper DevOps environment, the person deploying the code doesn’t need to have root access. There’s no reason why, when properly setup, developers can’t logon, push the deploy button, and create an auditable log that says at 2:32 on Friday the 18th, David deployed a version of the code. And you have it down to the hundredth of second who did what,” says Mortman.

As DeLuccia explained in his talk, when auditors are asking to see change control logs they are really trying to see that these controls are in place. “They’re really asking for help to understand this linkage,” he said. Depending on the policy and requirements, the auditor needs to see reports that demonstrate, in a statistically relevant way, that everything in IT is working the way it needs to and is said to be working. “Because remember, when we’re putting together these reports, we’re trying to say, everything is working the way you say it’s working,” he said.

Auditing DevOps: An education process
When done right, Mortman says DevOps and continuous deployments create more accurate logging and change management. “You have a log of every change that’s ever happened and it’s all generated automatically. You don’t have to worry about someone changing something. When setup properly, they don’t have the access to change anything other than the very limited roles of what that application they can deploy to,” Mortman says. “So in a lot of ways, in fact you’re actually maintaining better control over your environment.

The challenge there, as DeLuccia explained, is that audit programs are built a year in advance. “When they come to see you, they’re already have it planned out based off last year’s risk. That’s already too late [to keep up with rapidly changing technology and processes]. You really want to plan this a year in advance,” he said. The goal, he explained is to become part of the process and show how the procedures they are asking the IT team to execute against are in line with reality of the processes and the technology on the ground.

“Much like any other industry, you have an education problem,” says Mortman. “Security people run into this all the time. The business will say, “Hey, we want to do this new thing, and the security people reply, “Whoa, hold on a second. You can’t do that. That’s not secure. Or, we can’t audit that.” So a lot of the challenge with DevOps is an educational challenge. And you need to educate auditors that the control processes they are looking for are in place and that there are adequate logs that can be audited and evaluated,” says Mortman.

May 14, 2014
You have to crawl before you walk…

In previous posts I have explained what Security Policy Orchestration is, why DevOps folks should care, and how it can help facilitate the cultural change necessary for organizations to reap the long-term benefits of the cloud and virtualization. In this post, I’ll provide a few examples of how Security Policy Orchestration can create some “quick wins” – wins that create common ground between Security, Dev and Ops, and set the stage for higher profile wins with significant business impact.

As infrastructure becomes increasingly complex it gets harder to decipher. In software we call this Readability – it determines our ability to support the code over time and to ensure security. The same concept applies to networks – the more complex they are, the harder it is to use them, maintain them and ensure security.

The primary value of Security Policy Orchestration is providing a holistic view of the infrastructure. It “sees” the entire network and creates an index, which enables network administrators to perform quick queries across their environment providing “needle-in-a-haystack” search capabilities. While it might seem obvious, gaining visibility into the environment is a very powerful tool that can deliver huge benefits, especially when the environment is complex and subject to constant changes.

Let me give you a few use cases:

Audits: As my security brethren know, preparing for an audit can take weeks (if not months).   However, once the network infrastructure is indexed, meaning you have structured records of the network elements (devices, interfaces, routes, security rules…), the kind of documentation required for security audits for PCI, SOX, NERC, and other regulations is literally a few mouse clicks away. Imagine finding all applications flows involved in a PCI flow in one click, or, even better, through a REST API call.

Many of Tufin’s large enterprise customers report that their internal audit requirements are more stringent and time consuming than external compliance mandates, and require customized reporting, which they often have to script themselves. Policy Orchestration solutions enable them to be much more agile – to the point where security no longer becomes a “showstopper.“

A successful and effective audit (one that actually improves security) requires a collaborative approach across Dev and infrastructure – having the needed data at hand ensures agility, which is crucial for embedding this into the continuous process.

Server Migration: Many enterprises and service providers are currently tasked with server migrations to virtual data centers or cloud environments. The challenge they face is identifying the talkers, connections and protocols that are needed to maintain connectivity and business continuity. Firewall policies are often the best place to find this information because their important role in “attaching applications to the network”. However, it’s a labor intense task. Even if you have only 10 firewalls, with a few hundred rules and a thousand objects, identifying the flows is going to take a while. Now multiply that by 100 migrations per month – it’s virtually impossible.

However, once you automate the process, it becomes quite easy. With change automation in place, all you have to do is click “Find”, or, even better, call an API, and you will know the flows immediately.

Application Deployment: As most devopsy.kinsta.cloud readers are (likely) well aware, when an Application team needs a connection opened to say, an LDAP database, they open a ticket with a business request: “I need to connect server A to LDAP database B.”

It’s the network guy (or gals) job to translate that request into a set of IP addresses that plot a secure, efficient network path that connects the database to the application. A week or so later, the requisite changes are made, and voilà, the application can leverage the LDAP database. Or not, and then the network team needs to re-trace its steps and figure out what the problem is.

The above process, in many organizations, is a loose one, usually conducted via email, and between people who don’t often traverse the same orbit.

Add human error into the equation – a mistyped or incorrectly documented IP address, and you end up with re-dos, which everyone hates because they are time consuming and in complex network environments can be impossible to trouble shoot manually.

Ask any network manager – reducing (if not eliminating) re-do’s is an impressive feat that offers substantial efficiency gains.   It’s also worth noting that without automation it is really, really hard to properly risk assess proposed changes. In worst-case scenarios, an insecure network connection becomes apparent after something bad has happened. But why wait for something bad to happen before innovating? Bring security into your next network virtualization initiative at its onset – I promise you, you won’t be sorry.

DevOps folks are likely familiar with the idea of abstraction, but it is still a relatively new concept in network security circles.   When applied to network change management, Abstraction separates business logic from the technology underpinnings of network infrastructure, and allows Operations teams to translate higher-level application language into terms they can use to enable underlying network services.   In scenarios like the above, the ability to automate this process can not only deliver huge efficiencies and create a much more collaborative dynamic between application, network and security teams.

Hopefully, these examples build the case for how Security Policy Orchestration can be leveraged to deliver some quick wins. Of course, organizations that commit to deeper and wider change reap greater benefits. However, as my Co-founder, Tufin CEO Ruvi Kitov likes to say, you have to crawl before you walk, and walk before you run. So in my next post, I’m going to explore why security people are so weary of automation, and how to get around that, in hopes that Security, Dev and Ops teams can all walk together.

April 29, 2014