devopsy.kinsta.cloud

Tag: risk

How to Manage Risk, Regulation and Compliance Differently —and Better
During my career as a technology leader I have worked on PCI security, lawsuits, federal and state compliance, foreign market launches and acquisitions. All of these have regulatory, compliance and legal aspects to them, and over time I’ve come to realize we are bad at dealing with such things—really bad. At the heart of being so bad at this is something I call “binary risk management.” Binary risk management can, and often does, paralyze an organization.

Binary Risk Management is a failure to recognize the probability associated with a risk and treat all risks as if they are destined to happen. If there is no risk at all, proceed. If there is any risk, no matter how small the probability of its occurrence, halt!

We have all heard something like this: “Legal says we have to replace all the images that have any text in them across our entire French site because you’re not allowed to have any English on a site in France.” I actually heard that. It turned out not to be the case, but that did not stop me from wasting quite a bit of time working on this project. I was scared. The fines were said to be “hefty” and legal said it, so it must be true. Right?

I have wasted months on work that never had to be done. I have seen a CEO need to ask a compliance department’s permission for lower-level employees to have an informal social get-together. I’ve seen an acquisition almost fall apart because of a non-existent risk (the risk was similar to your chances of catching AIDS from a toilet seat). I have many stories and I’d love to hear some of yours in the comments section.

In all these cases, the outcome of the risk materializing instead of the probability of it actually happening was the focal point of conversation. In some of these cases the probability of occurrence was, in fact, zero because the whole thing was misinterpreted to begin with, so paralyzing are specters of lawsuits and fines.

Here are a couple of rules of thumb—tenets, if you will—for dealing with regulatory, legal and compliance issues. These tenets safeguard against wasting time and money that could be better spent helping our customers and making my company more secure, compliant and safe.
1. Examine the probability, speak of the probability: Examine the probability of each risk that is factored into a business decision. When speaking about a risk don’t just speak about outcomes, as if occurrence is presupposed. Speaking in probabilistic terms can help us reason more effectively about risk.
2. Examine the outcome: Is the worst case really the worst case? Oftentimes things can be done in the event of a failure that soften the blow. If such mitigations are available, then the ultimate risk that should be considered is with this softened outcome, not the worst case. The wind may blow and the tightrope walker may fall, and then he will die. But wait—what if we put an air mattress at the bottom? Now the walk is not so risky. In business, we often assume and focus on the unmitigated worst case.
3. Know the wording: Don’t just take for granted that “it is decreed that” you must do something. Read the law and regulation in question and know exactly what it says. You will be shocked how frequently the experts don’t do this because they are busy and overawed by perceived negative outcomes. Knowing the actual regulation can aid in finding an effective solution, not a broad CYA solution that hurts your company and customers.
Legal and compliance teams often are not compensated on business outcomes, so, for them, choices on preventing risk can be either A: Take a risk, or B: Take no risk. They often choose B. Forcing a critical examination of the wording of regulations in question, an exact understanding of the consequences, how they can be softened and a real analysis of the probability of the outcome of an event can save your company a lot of money. The money saved can be directed to achieving better outcomes for your customers.

I’d love to hear some of your examples of binary risk management. Please share in the comments.

I’d also love to hear from those legal and compliance minds that do get it (I have met a few) on other techniques for being safe and effective while avoiding paralysis that comes of binary risk management.

— Martin J. Logan
April 28, 2017
Lean Security: How Better Development Can Protect Your Business

If companies are to reach their strategic goals—reducing time to market, boosting sales, improving product market fit and brand image, and cutting cybersecurity costs—then it’s time for a new outlook on software security.

Today’s business leaders must learn to see security for what it is: A differentiating factor. Companies with reputations for secure developmental processes and infrastructure will rise above those known for data breaches.

Security as a Business Strategy

The road to a more secure company—and realizing the perks associated with it—begins with lean security. This is an approach to information security similar to the Toyota principles of management and production that calls for environmentally aware engineering, simplified coding, automation of security checks and constant incorporation of feedback.

Although some business leaders believe organizational security is inherently expensive, lean security doesn’t mean massive costs. Much like DevOps, lean security champions process improvements and cultural changes above purchasing new tools.

Lean security does, however, require engineers to keep its principles in mind throughout the development process. In this way, security is much like sales: A sales representatives’ best strategy is to consider his goals from the beginning. Just as the salesperson isn’t trying to make any sale, the engineer isn’t trying to create just any product. For the salesperson, this means tweaking the sales process—perhaps by vetting leads before engaging them—to make profitable sales. For the software engineer, it means structuring the development process through lean security to create secure products.

Done right, lean security results in a self-defending, simple system created by people who are aware of a product’s real-world security risks and the company’s business goals. Lean security recognizes your company’s data is always at risk, no matter the size of the company or team, and it works to create value through systemic protection. Here’s how to start:

Improve Environmental Awareness

Lean security approaches risk management as a team effort. Think about a hospital. Everybody working there — from the cooks to the nurses to the executives — has a responsibility to take care of patients.

Software firms must function the same way: Everyone from engineers to salespeople to the CIO must keep data privacy, software security and business needs in mind throughout product development and a product’s life cycle.

Environmental awareness means addressing security with the same sensibilities that you would your own life. Just as you don’t hand your credit card to random people, neither should you hand the keys to a software engineer who doesn’t know the risks of insecure development.

More secure coding begins with a change of mindset. Once team members understand a product’s security risks, they start thinking about how to create a more secure environment. If a developer is creating a mobile app for a car, for instance, then they must recognize the app will be used in conjunction with a device that carries human life.

Simplify Engineering

Complexity is the enemy of security. Simplification, however, cannot just be limited to lines of code: It means organizing your teams—and thus, your entire system—in a way that improves time to market and tightens feedback loops.

DevOps is a great way that business leaders can improve security through sensible reorganization. While the integration of development and operations teams has been lauded as a security boosting measure by software leaders at companies including Dell, CA Technologies recently studied DevOps’ business benefits. It found companies that have implemented DevOps techniques are 2.5 times more likely to improve customer retention, twice as likely to grow their revenues and 3.4 times more likely to improve their market share.

In short, keep teams compact and agile and design with the end in mind. The more lines of code you add, the more complex teams you create and the more times you reinvent the wheel—and, ultimately, the less secure and less profitable your company’s products become.

Automate or Die

Automation is an essential component of lean security, and DevOps engineers are no strangers to automation. Threats are constantly increasing, and no engineer—or team of engineers—can discover and remedy them all manually.

Look to Netflix for a great example of automated security. Once upon a time, Netflix addressed malware alerts by manually creating a help desk ticket and assigning an engineer to investigate the problem. The time from alert generation to eventual resolution, according to Netflix, often spanned more than a week. As Netflix’s security challenges increased in both diversity and complexity, it found itself spending exponentially more time and money combating these threats.

Then, Robert Fry, Netflix’s senior information security architect, came up with a brilliant solution: He engineered FIDO, an orchestration layer that automatically evaluates, assesses and responds to security threats. FIDO detects threats, then scores them based on the attack’s intended target and other factors. FIDO then attempts to mitigate the event by closing a network port, ending a VPN session or disabling the account.

Tools such as Netflix’s FIDO automatically detect risks in seconds, while it could take a human months to stumble across them. Think of security automation like a home security system: ongoing, vigilant security without the need for continued manpower.

Measure Everything

Although its origins are disputed, many credit English mathematician Karl Pearson with quipping, “That which is measured improves. That which is measured and reported improves exponentially.”

Development teams should take Pearson’s wisdom to heart, and measurement must begin with identification of goals. Is your goal to reduce application downtime or decrease event response time? Then it’s essential to measure mean time to repair. Do you want to reduce code defect density? Begin by measuring the number of issues per thousand (or million) lines of code.

Once you’ve identified goals, use software to track progress over time. Etsy’s engineers built a tool they call StatsD. The software helps Etsy monitor everything from login failures to coffee availability. The data is then displayed in handy graphs to help the team make sense of the information.

Once you’ve taken baseline measurements, be sure to review your metrics on a regular basis. For instance, you might realize an application’s deployment times aren’t where you’d like them to be. Keep measuring results—and keep trying new things—to nudge those values into better territory. Just remember: You cannot improve that which you do not measure.

The Business of Lean Security

The value of a secure, simple system cannot be understated: It means getting products to market faster; it means better public perception and press; it means more satisfied customers who feel secure using your product; and it means fewer disruptions in engineers’ work schedules. At its core, lean security lowers a company’s overall exposure to risk and reduces its expenses.

Companies are already staking their futures—and consumers’ safety—on code that facilitates transactions, drives vehicles and manages power plants. The Internet of Things will soon run everything from toasters to jet engines, and security will grow even more important. Protect your company from within through lean security.

June 3, 2016
Evolving to Continuous Testing
As agile development practices mature and DevOps principles infiltrate our corporate cultures, organizations are realizing the distinct opportunity to accelerate software delivery. However, when you speed up any process, immature practice areas such as testing and roadblocks become much more pronounced. It’s the difference between driving over a speed bump at 5 mph vs. 50 mph … at 50 mph, that speed bump is going to be quite jarring.

Accelerating any business process will expose systemic constraints that shackle the entire organization to its slowest moving component. In the case of the accelerated software development life cycle (SDLC), testing has become the most significant barrier to taking full advantage of more iterative approaches to software development. For organizations to leverage these transformative development strategies, they must shift from test automation to continuous testing.

Drawing a distinction between test automation and continuous testing may seem like an exercise in semantics, but the gap between automating functional tests and executing a continuous testing process is substantial. This gap will be bridged over time as the process of delivering software matures. Both internal and external influences will drive the evolution of continuous testing. Internally, agile, DevOps and lean process initiatives will be the main drivers that generate the demand for change. Externally, the expense and overhead of auditing government and industry-based compliance programs will be the primary impetus for change.

A Roadmap for Testing

Any true change initiative requires the alignment of people, process and technology—with technology being an enabler and not the silver bullet. Yet there are some basic technology themes we must explore as we migrate to a true quality assurance process. In general, we must shift from a sole focus on test automation to automating the process of measuring risk. To begin this journey, we must consider the following:

Driven by business objectives, organizations must shift to more automated methods of quality assurance and away from the tactical task of testing software from the bottom up.

From Causal Observations to Probabilistic Risk Assessment

With quality assurance (QA) traditionally executing manual or automated tests, the feedback from the testing effort is focused on the event of a test passing or failing—this is not enough. Tests are causal, meaning that tests are constructed to validate a very specific scope of functionality and are evaluated as isolated data points. Although these standalone data points are critical, we must also use them as inputs to an expanded equation for statistically identifying application hot spots.

The SDLC produces a significant amount of data that is rather simple to correlate. Monitoring process patterns can produce very actionable results. For example, a code review should be triggered if an application component experiences all of the following issues in a given continuous integration build:
- Regression failures greater than the average
- Static analysis defect density greater than the average
- Cyclomatic complexity greater than a prescribed threshold
From Defect Documentation to Simulated Replay

The ping-pong between testers and developers over the reproducibility of a reported defect has become legendary. It’s harder to return a defect to development than it is to send back an entrée from a world-renowned chef. Given the aggressive goal to accelerate software release cycles, most organizations will save a significant amount of time by just eliminating this back and forth.

By leveraging Service Virtualization for simulating a test environment and/or virtual machine record and playback technologies for observing how a program executed, testers should be able to ship development a very specific test and environment instance in a simple containerized package. This package should isolate a defect by encapsulating it with a test, as well as give developers the framework required to verify the fix.

From Structured Data to Structured and Unstructured

The current tools and infrastructure systems used to manage the SDLC have made significant improvements in the generation and integration of structured data (e.g., how CI engines import and present test results). This data is valuable and must be leveraged much more effectively (as we stated above in the “From Causal Observations to Probabilistic” section.

The wealth of unstructured quality data scattered across both internal and publicly-accessible applications often holds the secrets that make the difference between happy end users and unhappy prospects using a competitor’s product. For example, developers of a mobile application would want constant feedback on trends from end user comments on:
- iTunes app store
- Android app store
- Stackoverflow
- Twitter
- Facebook
- The company’s release announcements
- Competitors’ release announcements
This data is considered unstructured since the critical findings are not presented in a canonical format: parsing and secondary analysis are required to extract the valuable information. Although these inputs might be monitored by product marketers or managers, providing these data points directly to development and testing teams—in terms that practitioners can take action on—is imperative.

From Dashboards to Business Policies

In a Continuous Everything world, quality gates will enable a release candidate to be promoted through the delivery pipeline. Anything that requires human validation clogs the pipeline. Dashboards require human interpretation—delaying the process.

Dashboards are very convenient for aggregating data, providing historical perspectives on repetitive data, and visualizing information. However, they are too cumbersome for real-time decision making because they do not offer actionable intelligence.

Business policies help organizations evolve from dashboards to automated decision making. By defining and automatically monitoring policies that determine whether the release candidate is satisfying business expectations, quality gates will stop high-risk candidates from reaching the end user. This is key for mitigating the risks inherent in rapid and fully-automated delivery processes such as Continuous Delivery.

From Tool Dependent to SDLC Sensors

Let’s face it—it’s cheap to run tools. And with the availability of process intelligence engines, the more data observations we can collect across the SDLC, the more opportunities will emerge to discover defect prevention patterns.

Given the benefit of a large and diverse tool set, we need to shift focus from depending on a single “suite” of tools from a specific vendor (with a specific set of strengths and weaknesses) to having a broad array of SDLC sensors scattered across the software development life cycle. And to optimize both the accuracy and value of these sensors, it’s critical to stop allowing tools to be applied in the ad hoc manner that is still extremely common today. Rather, we need to ensure they are applied consistently and that their observations are funneled into a process intelligence engine, where they can be correlated with other observations across tools, across test runs and over time. This will not only increase the likelihood of identifying application hot spots, but will also decrease the risk of false negatives.

About the Author/Wayne Ariola

Wayne Ariola, Chief Strategy Officer, leads the development and execution of Parasoft’s long-term strategy. He leverages customer input and fosters partnerships with industry leaders to ensure that Parasoft solutions continuously evolve to support the ever-changing complexities of real-world business processes and systems. Ariola has contributed to the design of core Parasoft technologies and has been awarded several patents for his inventions. A recognized leader on topics such as service virtualization, SOA and API quality, quality policy governance, and application security, Ariola is a frequent contributor to publications including software test & performance, SOA World, and ebizQ—as well as a sought-after speaker at key industry events. Ariola brings more than 20 years strategic consulting experience within the technology and software development industries.
March 29, 2016
Rugged DevOps: Less Capture the Flag, More Teamwork

At the recent DEVNEXUS conference in Atlanta, I caught up with Chris Corriere to talk about his experiences in the realm of Rugged DevOps. Chris is a DevOps Engineer at AutoTrader and a contributor here at devopsy.kinsta.cloud. During our conversation, we discussed automation, culture and collaboration, and which thought leaders he is following. Chris also shared insights on the upcoming DevOps Days Atlanta conference where he serves on the organizing committee.

Derek Weeks: What’s your definition of rugged DevOps?

Corriere: Rugged DevOps to me is definitely manifested in the security sector but it’s a shifting quality and awareness left in doing the best we can with the tools we have, the constraints we’re given, and in the environment we’re in. It’s about being more experimental, understanding that we’re going to encounter failure, and learning to adapt accordingly. We’re very much having to make tough decisions in real time. If you’re running anything in production these days, you need to do that as safely and as consistently as you can.

Weeks: Is rugged DevOps only about security?

Corriere: No. Security is really about mitigating risks. I think a lot of this ties into situational awareness about understanding your company’s risk. Is it user data? Is it credit card transactions or do you have a Russian botnet running in your infrastructure that you’re not aware of? What’s the asset you need to protect? Are you trying to protect an asset that is not there?

You can’t have security come in and become suffocating where it’s not needed. In other situations you can’t leave things wide open that really do need to be protected because you’re more concerned with moving quickly than moving safely.

https://youtu.be/fgMPKkM1NdY

At the end of the day when you’re mitigating risk, it doesn’t matter if it’s customer data or a broken build or a bad user experience, these are all risks we need to mitigate. That’s really what rugged DevOps is. It is about is tying that value back to the user. It’s knowing what problems we trying to solve and understanding what we doing effectively (or not). It can be locked down as tight as you please, but if you cannot use the tool to solve the problem you picked it up for, it might as well not be running.

Weeks: Security teams have been one of the last parts of the organization to come into DevOps practices. What’s been your experience with engaging security teams at Auto Trader?

Corriere: That’s a good question. I’ve had experience with more than just security teams at Auto Trader. I’ve been involved in security a few places and they’ve approached me first when they found out I was into the DevOps stuff. They often come looking for assistance. They want to know: How do we get this baked in? How do we make this a priority?

It really ties back into multi-factor authentication. It’s a good way to frame it up where you want developers and operations (those people who aren’t permanently rooted in security) to be aware of these things, to have automated scans, to be aware of cross-site scripting, and to understand how to remediate those kinds of issues.

We’ve been implementing open source tools earlier in the development lifecycle to give us quicker feedback loops in order to be able to triage these things. Security applauds that but they can’t trust us too much with it. They’ve got to verify that we’re doing a good job with this, right? It’s a give and take where I’ve seen security come and offer automated solutions to DevOps teams and those solutions get adopted. The DevOps teams then take that and really start to run with it. Then security wanted to slow down a little bit because they were concerned about the quality of the vulnerability remediation. There is some back and forth there that has to be anticipated.

________________________________

“We’ve been implementing open source tools earlier in the development lifecycle to give us quicker feedback loops.”

________________________________

For security to trust you you’ve got to trust security to a large extent. If you’re putting more automation into the left side of your process and doing things before they hit production should be making your job easier. One of the side effects of that security may start going through some systems with a finer tooth comb because they’re not doing as much fire fighting. That may result in another ask from security. Understanding that this isn’t a declaration of war—we are simply continuing with an incremental progression toward further improvements. Interactions are cyclic and will happen again.

We need to help security understand our pipeline, determine what’s going to be a good fit, and keep that communication level high so the trust is there. In these symbiotic relationships, cooperation is dependent on that trust. If that trust goes away you end up in something more competitive.

There’s always room to drop back to where it’s more commensal and knowing when the kitchen’s gotten too hot. At those times, everybody needs to take a cool down lap and come back to it at the beginning of the next month. It goes back to that situation awareness, knowing where you’re at right now before you try to get to where you’re going.

Weeks: We talked earlier today about red teams, blue teams, purple teams. What are they and can you describe some of the practices you’ve used to end up with more purple teams?

Corriere: Yeah. Red teams take an attacking, reactive posture. Red security teams are worried about internal compromise, and when they find gray areas—for example, things that haven’t been fully signed off and authorized—it can result in unplanned work cropping up for the DevOps team. Generally that kind of unplanned work is introduced because the “Red” team felt they were not able to accomplish their job another way.

Then the blue team side of the equation is really trying to stomp all that out. Blue teams keep things standard and locked down. They make sure everything’s approved in triplicate before it gets implemented, which can be rigid and very frustrating for developers.

Purple teams allow that experimentation in earlier in the development lifecycle where they’ve got a little bit more leash. They’re allowed to experiment and stand up new things and make those things work before they getting it approved. Again, it’s about being transparent. Teams first want to understand what change was introduced and what the advantages of it are. They may also acknowledge that you need more than one solution to some problems, and that one tool might not fit everybody. If they’re using tools to automate that’s good. We can’t box them all into one necessarily. That’s where you see this go from blue and red to more of a purple. There’s more communication. There’s more conversation. There’s less capture the flag, more teamwork.

________________________________

“There’s more communication. There’s more conversation. There’s less capture the flag, more teamwork.”

________________________________

Weeks: You’ve been involved in the conversations around rugged DevOps for awhile. Who are some of the other thought leaders you learn from?

Corriere: Jamesha Fisher out of San Francisco is one. She’s done some neat security stuff and is starting to look more into the vulnerabilities around some of the automation tools. Where are those vulnerable and where can they be leveraged against us? Automation is great until it automatically starts doing things we didn’t want. Another aspect of that is really trying to think more like an attacker; we need to think about: how would I break into this thing instead of just thinking how do I protect it.

Georgia Weidman is popular in a lot of security circles and I’ve missed her workshop once. I’d like to catch it. She’s got a book out on pen testing which covers a lot of operations and concepts from an attack perspective. I would say that can be a fault of some rugged DevOps practices; you’re so quick to get the thing working and provide business value that you don’t realize where you’ve left something vulnerable. Sometimes you’ve got questions about why that thing you work with frequently is always locked down. Perhaps you find you can’t use it the way you want to. At those moments, it’s time to read through Georgia’s book. You’ll get some information as to why that thing is not open the way you want and where someone might use it for reasons might not expect.

________________________________

“You’re so quick to get the thing working to provide business value that you don’t realize where you’ve left something vulnerable.”

________________________________

Weeks: As we wrap up I know you’re on the organizing committee for DevOpsDays Atlanta. Do you want to give us an insight on the dates of when that’s coming up and call for papers, et cetera or we can help you share that?

Corriere: Our CFP is open through the first of March. Our event is going to be in April. It’s the 26th and 27th which is a Tuesday and Wednesday. Monday, the 25th of April, we have Jeff Sussna coming in to do a full day workshop on continuous design. He’s going to be giving the keynote on our first day. We’re excited to have Jeff in town and excited to have him as a keynote. John Willis is going to be giving our keynote on the second day.

We’re looking for proposals from anything in progress, people learning in the field, or trying to drop that barrier of entry. We’ve got a lot of bright people in the space. We’ve got a lot of people learning as they go. We need to be more vocal about that learning process and how we find our way through it. There’s not always this instantaneous vision that strikes us. We understand how to convert a shop into continuous delivery every night. There are definitely steps in between that. We need to have more conversations about those step in our industry. Yeah, anywhere in the stack, any interesting projects going on, we’d love to hear from you if you’ve got something.

If you are interested in learning more about this subject, I invite you to download Amy DeMartine’s Forrester research paper, “The 7 Habits of Rugged DevOps.”

As Amy notes, “DevOps practices can only increase speed and quality up to a point without security and risk (S&R) pros’ expertise. Old application security practices hinder speedy releases, and security vulnerabilities represent defects that can leave a company open to cyberattacks. But DevOps practitioners can leap forward with both increased speed and quality by including S&R pros in DevOps feedback loops and including security practices in the automated life cycle. These new practices are called Rugged DevOps.”

February 24, 2016
How to lower risk in agile development
The increasing complexity of application development and subsequent rise of Agile development is changing the risk appetite in many software development organizations. Unless you’ve got an application which is highly regulated or mission-critical, it’s rarely feasible or sensible to manage Quality Assurance within the zero-bug environment often associated to Waterfall. Software development teams can become handcuffed by the lack of flexibility and struggle to maintain a focus on delivering business value. Agile methodologies paired with DevOps emphasize speed, rapid iteration and a “fix as you go” mentality over driving toward a perfect product out of the gates. So how can we lower risk in agile development?

There are other contributing factors to the growth in software risk, notably the continuing adoption of cloud computing. Given the appealing economics and flexibility for updating and repairing code in the cloud, many teams will choose to host even the most high-risk data on the cloud. Conversely, teams must focus even more attention on security, as the cloud exposes more data outside of the company’s network.

Software interdependencies are another risk; many business processes today rely upon the use of outside services from APIs to OAuth authentication. Teams must be on top of changes in these external systems to ensure that critical functionality can still operate. This is especially true in a mobile context, where even the operating system and hardware components are changing at a rapid pace.

There doesn’t have to be a trade-off between speed and quality. It takes a change in mindset and processes, however: managing software risk in the age of Agile and cloud is a whole new game. Here are a few things to consider:
1. Balance risk with business needs: A shining reason behind Agile’s popularity is the unceasing demand for speed and business alignment. Software and IT organizations have realized that in order for their employers to be competitive, they must operate in ways that are highly business-responsive and adaptive. Agile also helps organizations address the complexity of software development today, given the impact of mobile devices, the proliferation of new languages and tools for Web apps, growth in cloud-based computing and the overall importance of digital processes to a company’s success. QA teams need a more educated and informed approach to testing. That means taking on a measured approach to risk, wherein risk management tasks are implemented with business priorities in mind rather than as a one-size-fits-all strategy.
1. Risk-based testing: In order to take on more risk as part of Agile and DevOps thinking, there must be proper checks and balances integrated within the testing process. Instead of testing every feature and function, and shifting timelines, prioritize testing and quality management measures to fit business priorities. Risk-based testing commonly assigns a risk rating score for individual tests, which gives developers and QA an idea of the level of risk involved in delivering the code covered by that test if the test is skipped. As a result, the product team might spend more time testing versions of the application for devices which comprise the most important or largest sector of the customer base. However, common device types and operating systems are not the only variables that can factor into risk scores. For example, if one particularly large customer relies on a key feature or function, that should mandate additional testing around that area of the application. By linking business drivers to development and testing, organizations can still deliver quickly yet with the assurance of superior results in the areas that matter most to revenues and customer satisfaction. Ongoing testing of high priority application features also results in the avoidance of costly, wholesale changes down the road.
1. Integrate development and testing: As mentioned earlier, integration between development and testing is critical for many projects, yet it is frequently overlooked when managing Agile teams. To maximize success, there needs to be a deeper understanding between coders and testers on potential ripple effects of introducing new features as they are planned. This can occur during the everyday processes of Agile collaboration and sprint planning. Testers should know early on which are the top risk areas affected by a piece of new development; testers and developers both should incorporate development best practices which focus on understanding and planning for interdependencies, such as designing code that limits the downstream impact of integrations.
1. Best of Breed tools: The market for tools has expanded vastly along with the onset of cloud, mobile, social and big data technologies and the DevOps and Agile movements. The average IT shop is using dozens of tools for application lifecycle management (ALM), build, automation, project management, change management and testing. The newer tools, designed for Agile and Web development, allow for rapid iteration yet still provide controls and collaboration features to minimize risk. Beyond functionality, look for tools that will integrate with your existing toolsets. This will allow for faster, real-time visibility on application deliverables, new features and any quality issues that have been discovered.
Software development is more complicated today than ever. The good news is, it’s much less expensive, faster and easier to make a change in the code compared with even a few years ago. Incorporating some risk into a product is good, because it means that innovation and time-to-market are top of mind. However, introducing too much risk in high-priority customer areas or critical periods in a company’s growth can sink a business. Understanding the difference is what can allow you to keep both your customers and executives happy, with a brisk yet manageable pace of innovation.

About the Author/Kevin Dunne

Kevin Dunne is the Director of Product Strategy at QASymphony, ensuring their continued commitment to innovation and delivering tools to create better software. With a deep interest in the emerging trends in software development and testing, Kevin is dedicated to collaborating with thought leaders in this space.

Kevin comes to QASymphony from Deloitte, where he managed testing on large government and Fortune 500 engagements delivering ERP implementations and custom software development. As one of the first employees at QASymphony, Kevin has seen many facets of the business working in sales, customer support, marketing, and product management.

Kevin holds a Bachelor of Science degree from Vanderbilt University. Connect with Kevin on twitter.
August 20, 2015
Ensuring security and managing risk in enterprise DevOps

Today’s question in this recurring series comes from a developer in a typical large enterprise – risk averse, tightly controlled, highly secured – a major challenge to any DevOps transformation:

Q. I work in a large government department that is tight on security. We have strict policies on separation of duties, can’t share users IDs or passwords, struggle to share code, can’t use AWS or Azure, audit everything, and so on. How can DevOps work in my environment?

If DevOps is about collaboration and removing bottlenecks, then it must include security, from planning to coding to testing to operations; building, delivering, and running applications that are secure by design. Unfortunately, for many enterprises security is a major roadblock in the application delivery lifecycle.

Yet security does not need to be a DevOps bottleneck. Indeed, these specific issues are solved problems. While I do not boast of a deep security expertise, I can address the solutions based on my experience (and some further reading – see below):

Maintain separation of duties

Large enterprises must maintain a separation of duties, to prevent (for example) a rogue developer from adding a back door to new code and putting it into production themselves, with no checks and balances. But staff can still have the access they need when they need it – and no more. When roles change, such as developers accessing the operations ‘flight deck,’ or operators being seconded to development, identity management and access controls can provide the right access automatically, as needed, audited and policy-compliant.

Protect privileged user IDs and passwords

For large enterprises it is never acceptable to share passwords, especially to privileged accounts like cloud control panel admins. However, privileged identity management allows staff to ‘check-out’ privileged IDs from a secure pool, use them as needed (e.g. for break-fix), then check them back in when they are done. As policy dictates, access can be subject to additional approvals, time limits, access limits, keystroke logging, session capture, etc. This allows on-demand access to sensitive IDs while maintaining stringent audit and control.

Provide secure cross-team interfaces

Part of DevOps is sharing – processes, skills, knowledge … but also reusing code and services (or microservices) through shared APIs, whether public or proprietary. This sharing accelerates delivery and reduces costs, but can also introduce risk. To prevent unintended consequences when sharing code and services, API security can enable sharing both internally and externally, but with controls like user authentication (e.g. OAuth), granular access controls, volume/rate limiting, blocking and more.

Enable secure on-demand cloud access

Most enterprise staff cannot just fire up an AWS or Azure instance on the corporate card. Even applying for ‘approved’ access can be a multi-week process. Instead, security pros can set up automated self-service provisioning to ‘known good’ or ‘enterprise-grade’ services, pre-approved by security, HR, finance, etc. This helps to maintain necessary audit and control while enabling DevOps staff to quickly and easily adopt cloud and other services.

Use automation to maintain security and governance

Automation is a powerful way to reduce risk, ensure compliance, and maintain governance. For example, release automation allows developers to deploy new code without having personal access to production systems (the tool gets the access, not the individual). It can also restrict access to such systems through ‘known-good processes,’ to ensure deployment complies with policy. Tracking who initiates deployment maintains that essential audit trail for governance and reporting. Test automation, data management, process automation, configuration management and other automation tools all provide similar security benefits throughout the application delivery lifecycle.

Have a security culture

Of course, there is more to secure DevOps than technology solutions. Just as DevOps requires a new culture of collaboration, good governance ultimately requires a culture of security. Security is everyone’s responsibility at every step of the service delivery lifecycle. Security is still a specialist role, so security pros should have a place in agile planning, standup and triage meetings, but good governance is everyone’s role. Dev, ops, test, QA and others all need to step to the security challenge; and security pros should share their requirements, knowledge, best practices and tools with the rest of the DevOps team.

Further Reading

In compiling this post, I leaned on many others who have helped to establish the concepts of DevOpsSec / SecDevOps and Rugged DevOps. A great starting point for more specialist detail on this topic is the SecDevOps content here on devopsy.kinsta.cloud, particularly the work of Rich Mogull (@rmogull) and Andrew Storms (@st0rmz). For an early intro, check the Gartner note, DevOpsSec: Creating the Agile Triangle and the seminal work of James Wickett (@wickett) on Rugged DevOps. For a more recent thought, try the Threatstack blog, and Edward Haletky (@texiwill) for a good end-to-end view. Finally, for a great resource specific to DevOps auditability, check the Gene Kim (@RealGeneKim) project, the DevOps Audit Defense Toolkit.

September 22, 2014