devopsy.kinsta.cloud

Author: Dwayne Melancon

  • Security in the operational relay race

    Security in the operational relay race

    If you’ve ever seen people run a relay race, you’ll notice that most of the risk happens when the baton is handed from one runner to the next.  In IT processes, there is also dramatically increased risk during handoffs.  Whether you’ve moving from one process phase to another, one “owner” to another, or one environment to another the risk is greatest during the phase transition.

    Security can play a strong role during these handoffs, whether you’re using DevOps or not.  Here are some principles that can help safeguard the handoffs and reduce the risk in the operational relay race.

    Reduce access

    They say too many cooks spoil the broth and that can apply to IT processes, as well.  Limit the number of people who can directly make changes in production and you have a shot at increasing security and defending availability by reducing random acts of deployment, as well as change collisions and mystery changes that just show up in production.  This is a key anchor for creating predictability and reducing change collisions.

    Create defensible “choke points”

    Reducing the number of ways things can be deployed into production will help you defend the integrity of the production environment.  This doesn’t necessarily mean preventing changes, it is more about controlling how changes enter the environment.  Focus on creating trustworthy repositories, deployment sources, automation tools, and service accounts that reduce the options for how deploys happen.  When you define clear swim lanes and you know what the expected path of deployment looks like, it is easier to detect unexpected activities and actions.

    Use choke points to enforce acceptance criteria to ensure that you are ready to take responsibility for the code, systems, and applications that being handed to you.

    This approach allows great freedom in how work gets done in autonomous teams, while enforcing more consistency through the release management and transition to Ops phases of the life cycle.

    Think like an air traffic controller

    When dealing with multiple streams of work that need to come together and interoperate in production, I think a mental model of an air traffic controller works well.  Airports allow a bunch of independent operators to come together in a shared environment in an orderly fashion, with minimal collisions and predictable throughput, and the air traffic controller plays a key role in making that happen.  In the IT context, this means leaning on people who have a broader perspective than any individual team so they can look at the big picture and coordinate activities holistically.

    This “zoomed out” perspective is critical in identifying risky activities and gaps, understanding dependencies that can break your security model, and stepping in to help mitigate dangerous conditions.  This approach also makes it easier to work with third-party providers, since you can focus on their results (outputs and handoffs) rather than getting caught up in the minutiae of their day-to-day activities.

    Monitor continuously to identify “trust gaps” and outliers

    DevOps is no stranger to continuous monitoring and instrumentation.  From a security perspective, continuous monitoring is about establishing a set of controls that allows you to quickly identify conditions that increase risk, and the elements above help you do that job more effectively.  For example:

    • Access limits and choke points enable you to quickly detect when someone has gone around your process and made changes directly in the production environment.
    • These elements also allow you to keep tabs on third-party contractors and providers to ensure that they play by your rules, and you know what they are up to all the time.
    • Defensible choke points enable you to sound the alarm if someone is tampering with your source repositories, automation scripts, or security monitoring components (such as disabling logging).
    • Study your indicators to find dark spots in your telemetry around handoffs, and ask how an attacker (internally or externally) could exploit any weaknesses in your handoffs.

    These are just a few examples of how these concepts can be applied.  When you recognize that handoffs are risky, you can quickly see how security can play a role in mitigating those risks and helping defend the reliability, safety, and trust of your production environment.

  • Continuous integration for better security

    Continuous integration for better security

    One of the big advantages to smaller deploys and continuous integration is that it can make it easier to provide more proactive security. In short, continuous integration (and the associated automated testing it enables) makes it easier to focus your security team on analyzing areas where the security risk is higher.

    If the notion of rapid development, rapid deployment, and continuous integration make your security team nervous you may be able to get them engaged by sharing proven ideas from companies who’ve made this approach work well in the real world.   For example, I was recently flipping through a presentation from Etsy on SlideShare, and discovered a clear explanation of how continuous integration enables better security.

    Slide34At right, you see slide 34 from this (very long but clear) deck. this is the first in a series of slides that talks about how you can identify high-risk areas of code, then use those designations to drive  automated checks for changes to those areas of code.  The power comes in when you automatically alert the security team that someone has just check in code which changes one of the high-risk areas.

    This seems like a commonsense way to focus high attention on high risk.

    Another positive benefit of this approach is that it forces the conversation upfront about what constitutes a high-risk area code. This is an opportunity to get security, ops, and the business on the same page about where risk is likely to occur. When you incorporate this approach with periodic reviews of where things stand, it allows you to refine your approach to automated analysis based on your experience. In other words, during post-incident review if you discover one of those”Gee I wish I’d known that I had of time” sorts of situations, you can add a check into the automation library that looks for those conditions in the future. This sort of continuous improvement model benefits everyone.

    Oh, by the way – one of the scenarios this approach can check for is the introduction of new third-party libraries. In today’s climate of elevated concern over third-party risk (as well as fears of security flaws in open-source components) having that capability can be a very good thing.

    What about you? How are you integrating continuous security checks into your continuous integration?

  • Protect and Defend: Repositories

    Protect and Defend: Repositories

    When you’re creating work that gets deployed into production, there is typically a repository involved. It could be a code base, a deployment hub, a definitive software library, a library of VM or application templates, etc.

    Often, you may not think twice about what’s in these repositories or whether you can trust their content – after all, you and your teams built these repositories, right? That may be true, but what if your trust is misplaced?

    The recent Target breach illustrated (among other things) that you may be incorrect in assuming the “inbox” of your deployment process is trustworthy. In the Target example, there was a queue of patches ready for deployment but someone added additional, malicious code to that queue without being noticed. The automated deployment process grabbed both the authorized and malicious code and distributed it to every point of sale device in the network. Not a good outcome.

    Their workflow included a trustworthy process to test code that before it was checked into the deployment queue, but there was also a flaw in the process: there were no controls in place to detect when anyone went around the process and made arbitrary changes to the repository. In other words, even if you put trusted packages into the queue, the packages could be modified and additional code could be added without those alterations being noticed.

    What we can learn from this kind of situation is the value of the tried & true principle of “trust but verify:”  don’t assume you can trust your repositories without validating what’s in there.  With that notion in mind, think about your environment – are there repositories that people could alter without being noticed? If so, that could present a problem both for security and reliability in your organization.

    To correct this, we need to treat our repositories like bank accounts – account for everything that goes in or out, and monitor the contents of the repository continuously to look for any undocumented or unauthorized “transactions” (adds, removes, or changes).  We must also add validation checks to the deployment process to make sure everything is accounted for before the deploy occurs.  Yes, this will likely mean you need to do some extra work on your deployment workflow, add technology for instrumentation, etc. but it will more than pay for itself through better security, accountability, and availability.

    By the way – this same principle applies to the scripts and automation tools you rely on, as well.  If someone were to alter these tools without your knowledge, you may get results you don’t like.  You should also consider that your automated deployment repositories aren’t the only vector for untrusted code to enter production – continuous monitoring of the production environment is also a crucial activity.

    Protect and defend your repositories – it’s important.

  • DevOps and Security Are Compatible

    DevOps and Security Are Compatible

    When I speak with information security organizations faced with the prospect of moving to DevOps, one of the most common fears I hear is that this transition will degrade security of infrastructure and applications.  If you’re one of these folks, I understand this fear but you can rest assured:  when you do things correctly security will actually improve.

    One big reason security benefits in this model is due to improved alignment and tighter feedback loops.  You see, DevOps is about creating a unified, engaged team and doesn’t make it easy to fall into the “silo thinking” that traditionally leads to security as an afterthought.

    DevOps embraces automation and consistency, which benefits security by allowing you to add automated checks during coding to look for obvious security issues and flag things for human review (such as the linking of new libraries or the introduction of new third-party components that could add risk).  This means you will be able to identify “areas of concern” earlier in the development process where they will be much cheaper and more timely to address.  Twitter is a well-known advocate of this kind of automation, in which security checks are executed each time code is checked in and obvious security issues are presented to the developer immediately, giving them a chance to correct the problem right away and reducing the likelihood they will repeat the errors.  That is a much more effective process than the delayed feedback typically associated with manual code reviews.

    Security isn’t just better in the “Dev” portion of DevOps – you can also add automated configuration validation and scope checking into the environment build process to minimize the chance that a fat-fingered error or an out-of-policy change will introduce problems into production.

    This sort of automated approach also improves as you learn from retrospectives and post-incident reviews.  As you encounter preventable issues, you can often improve your automated checks to increase the odds of catching these issues before they impact production.

    Remember that security is an important stakeholder in the DevOps process, and the onus is on you to become a part of the conversation and find ways to add value to the goal of the organization.  Also, don’t forget to tap into the community of DevOps practitioners – others have walked this path before you and can help you in your journey.

    Finally, if there are specific security challenges related to DevOps that are vexing you (myths you would like to see busted) I’d love to hear from you.  Leave a comment below and become part of the conversation.