Tag: continuous

  • Continuous is the new Black

    Continuous is the new Black

    We have gone from the city that never sleeps to a world that is continuous. That’s right, it seems along with the world shrinking, it is always on, someone is always watching or doing. Continuous has become the new black.

    Nowhere is this more true than in IT.  We have Continuous Delivery, Continuous Integration, Continuous Testing, Continuous Monitoring, continuous this, continuous that, continuous everywhere. Infosec is not immune to this either. We are trying to protect against continuous attacks by deploying continuous scanning, continuous threat protection, continuous monitoring, all of it part of our continuous security. All of this sounds like an old George Carlin routine. But it’s not it, it is the way things are today.

    Of course all of this continuous around security had to lead to continuous compliance. After all who would want to be compliant at only a point in time (even if that point was at the time of audit)?

    The newest version of the PCI DSS tries to move compliance from point in time compliance to continuous compliance. Some question whether this puts too big a burden on merchants. How many resources can they dedicate to compliance to make sure it is continuous?  Others say that continuous is the only way that makes sense.

    For me there is a lesson here that security can learn from DevOps.  The key is automation.  Automation is the secret weapon in doing continuous.  Without automation, continuous would require too many resources to truly succeed. But with automation we can duplicate, perform act over and over again without a human initiating every step, every move.

    That is not to say that humans are no longer needed. On the contrary, automation frees up humans to work on tasks of a higher nature. Automation frees up humans from the repetitive, mundane tasks that need to be done over and over again. Working instead on things that are of a higher value (at least they should be).

    But wait there’s more! Not only does automation empower continuous, automation brings speed. Once we automate, we can speed things up as well. So not only are we performing continuously, we are also performing faster.

    Now we are automating tasks without human intervention and at the same time doing them faster. I know for many a security admin the hairs on the back of your neck are standing up. This can’t end well.  Things are going out of control. Automating and speeding things up almost goes against being cautious and careful.

    But as counter-intuitive as it may be, automating and performing faster can actually result in being more secure and more compliant. Wait lets repeat that again: by automating and speeding up performance we can actually become more secure and more compliant.

    How? You don’t believe me? Impossible you think? How can continuous make us more secure and more compliant? Is it even possible to do all of this continuously?

    This is the topic that I along with Jody Brazil, CEO of Firemon are going to explore as hosts of a P2P session at RSA Conference. The session is on Thursday at 10:20am, it is titled Continuous Network Compliance: Finding Flaws and Betting Futures, you can click on the link for more info. Put it on your RSA calendar if you are attending. The P2P sessions only hold 25 or 30 people so come early.

    I will post a follow up to this post after the session with some of the discussion points around this. If you would like to hear the answers before then though, you will have to come to the session.  See you in San Francisco at RSA Conference.

  • DevOps & Continuous Change

    DevOps & Continuous Change

    A remark by a colleague while waiting for the coffee machine to complete its cycle started my train of thought. “Should we have multiple minor releases or just do a few major ones in a year?”

    In large organizations, due to many factors, the turnaround time for a single successful release is quite extensive; but if the conceptual change which we are talking about bringing whilst using the #DEVOPS methodology would not only reduce these long cycles but deliver quicker and quality software.

    I have already discussed in my earlier article “DEVOPS: Getting our organizations aligned” on the traditional approach and how the combination of this two (DEV & OPS) has been proven beneficial in various organizations and how many are still hesitant in embarking on that path.

    Continuous Change which encompasses both Continuous Integration and Delivery is paramount to achieving this objective. To attain the objective of delivering rapid and consistent value to the stakeholders regularly, we have to have Continuous Change happening.

    I am sure we have already moved out from the Mainframe era to a more Distributed systems state. But unfortunately we have yet to let go of that culture where we pile up components on a single quasi Distributed Application, instead of creating smaller independent, yet interdependent systems.

    So, back to the same question; shall we go with a few Dinosaur sized, labored releases or make multiple short and swifter runs? In my opinion, we should strive towards the leaner model. Easier said than done you say! Agree. If I have to, then I would go about it this way:

    In an environment having a suite of Distributed Applications, start with Analyzing the components which make up the Major release, List out the changes (fixes and new features) which are planned, asses functional and technological impact, Identify dependencies, upstream/downstream connectivity’s, architectural deficiencies, list out the identified subset of components or group of components by functionality, which can go as an independent release.

    I am resisting the urge to say ‘standalone’ as with the current set of complex intervened collection of applications, it would be an incorrect statement. It would rather be a subset of components which can be independently upgraded. Now the big task is to review and decide if this subset or group would add functional value to the overall suite of applications without the rest of components going in.

    Once identified, the success of this release working as expected would be in the extensive testing (Unit/ Regression/Performance) of these components, and as we gain confidence in this split and deploy procedure, we can start with, prioritizing and scheduling critical must have features or bug fixes which also should be backward compatible with other components lagging behind. We can even come up with scenarios where these components are not just getting updated frequently, but would also have multiple independent tracks. Of course it goes without saying the testing and signoff process is strictly followed albeit in a shorter window.

    Deployments to production have always been a source of heartburn to the OPS teams, we have often seen finger pointing and firefights to contain change fails. In order to maintain stability and maximize availability, stricter controls and reduced deployment windows are put in place. In promoting frequent and smaller component level changes, OPS will also be in control of what’s going in, and smaller component level releases can be rolled back quickly within the Green-Zone window if the change is not working as expected. Though I would categorize this rollback, as a process fail than a deployment fail, and should be reviewed very seriously with right mitigation plan and lessons learnt retrospection, Continuous learning after every successful or failed sprint ensures competent and confident releases.

    Another major factor impacting multiple runs of Production releases is the long Green-Zone windows, by moving to the quicker and leaner deployments cycles, these requests can be substantially reduced, and can eventually aim to reach a stage where changes are deployed online without extensive downtime.

    Similarly, frequent maintenance activities when there are no planned deployments, effectively has the same risk on environment unavailability. Analyzing and moving to a state where these activities are conducted online without a Green-Zone would alleviate the maintenance downtimes and ensure seamless availability, adding stakeholder value.

    I have not touched on the Continuous Integration or the Automation of the deployment process for obvious reasons, we can’t achieve the objective of Continuous Change without having the Integration process in place where incremental changes to code are build into a package swiftly and seamlessly, even a sanity or basic testing might be integrated to ensure the quality of the builds. Same goes for automating the deployments, develop & integrate tools required for rapid deployments. We have seen instances where automating a single activity or revisiting a process has created substantial time savings.

    While it’s easy to propose changes to the Release Process or advocate rapid deployment cycles, we have to ensure adequate checks and balances are in place, compliance adherence and audit trails are created.

    Now to the most important measure of this article, Cost Savings!

    Proponents of the Dinosaur Releases would argue that the common activities which entail a single release like the builds, Testing, Release Management, OPS availability or reduced GZ would cut costs and multiple runs to PROD would logically increase costs.

    But it’s definitely the other way around, we wouldn’t need to feed the Dino anymore, it will be a cost worthy rapid delivery with optimized resources, a leaner process where DEV and OPS are working together to ensure quality value added deliverables are made available to business faster.. This cultural change and collaborative effort also creates new vistas of growth and provides a perfect platform to excel. The cost benefit analysis would eventually be in favor of Continuous Delivery.

    Opportunity cost is another variable which has not been considered; an innovation or a new feature available to customers quickly with reduced time to market cycle will provide the much needed edge in these competitive times. I recall some conversations where a few must have’s were pushed out due to various reasons, frustrating the business.

    DEVOPS was never meant to replace the traditional approach to software development; rather it is an efficient use of the existing resources within a collaborative model to rapidly deliver quality software products. A major cultural shift in our approach is required and to reach that goal we need to embrace the shades of grays, many a times over 50 (sorry, couldn’t resist) when it’s not truly black or white.

    Thank you for the time.

    Visit my blog http://askhurram.com/

    Read On, Live On.

     

  • Zen and The Art Of Infrastructure Maintenance

    Zen and The Art Of Infrastructure Maintenance

    There’s a famous passage I’ve quoted from time to time when I see someone “stuck” with a problem, flailing potential fixes and solutions wildly in hope that one of them does the trick.

    “Assembly of Japanese bicycle require great peace of mind.”

    Just like that. Improper grammar and all, quoted verbatim from Robert Pirsig’s philosophical classic Zen and the Art of Motorcycle Maintenance. The quote is in reference to the first printed words on a set of technical instructions, which the narrator discusses with friends and colleagues on a long-distance motorcycle trip.

    And the response I get is the same every time, likely the same response you’re having right now…

    What in the world are you talking about?

    Well good, now I have your attention. And I’d be willing to bet that attention is on 1) the ludicrous loss of translation from Japanese to English and 2) debating whether or not something as seemingly superficial as “peace of mind” is really required to assemble a bicycle.

    Language barrier aside, let’s abstract away from modes of bipedal transport into a more generic “machine”. Do we really require “peace of mind” to assemble or create a machine?

    cory2

    Reliable? Maintainable? Usable?

    We certainly all want to receive peace of mind from our machines. We want them to do the job they were created to do without fail so we can move our attention onto other things.

    What words do we use to describe such machines that give us this peace of mind? Reliable? Maintainable? Usable? When we think of the reliability, maintainability or usability of a machine, we’re really just describing a machine that can produce peace of mind. I’d argue if we want that machine to produce peace of mind, then it must be created under peace of mind.

    Why is that? If we, the creators of machines, are in a state of peace of mind when we start our endeavor, we’re highly motivated to do that which will continue our state of peace of mind. We don’t want to lose this state. Since if the machine does break, we the creators will be forced to lose our peace of mind and fix it, we’re motivated to create a machine that’s resilient against failure and resilient against our loss of peace of mind.

    Without this key precondition to creation, we’re likely to build our existing personal problems right into the machine itself! Under time pressure? We’ll cut corners to make deadlines. Dealing with a different machine breaking? Likely to gloss over a key failure condition in the machine we’re actively building so we can fight a fire elsewhere. Worried about the problems of today? Then I can guarantee there’s no thought going into a machine that will be resilient to the problems of tomorrow.

    Quality creations produce serene creators as much as serene creators produce quality creations.

    Those in a state of serenity will continue to create that which maintains the state of serenity or peace of mind; those that are not in a state of serenity are unable to escape a vicious cycle of build-fix-build-fix, never achieving peace of mind. Ultimately, the serenity of the creator is the key test for the quality of creation.

    And ultimately, this passion for peace of mind from our machines is what you’ll find when you meet the folks in the DevOps community. We’re guided by a desire to maintain our peace of mind — for ourselves, for our families and for the folks that count on us the most: our customers.

    If the machine produces tranquility, it’s right. If it disturbs you, it’s wrong until either the machine or your mind is changed.

    The material object of observation, the bicycle or [in this case, the global infrastructure], can’t be right or wrong. Molecules are molecules. They don’t have any ethical codes to follow except those people give them. The test of the machine is the satisfaction it gives you. There isn’t any other test. If the machine produces tranquility it’s right. If it disturbs you it’s wrong until either the machine or your mind is changed. The test of the machine is always your own mind. There isn’t any other test.
    – Robert Pirsig

    So the next time you find yourself struggling with a problem, take a lesson from Robert Pirsig: step back, achieve peace of mind, combat that which threatens peace of mind and strive to maintain peace of mind.

    You’ll be a better (and happier) creator for it.

  • Always-on: The New Normal

    Always-on: The New Normal

    One of the many challenges of DevOps is finding great team members that can handle the always-on nature of the job. Nearly everyone at VictorOps has been “that person” and has done the weeklong stint of carrying the virtual pager.

    What we noticed in our years of doing that job is that team member behavior tends to be polarized. If it’s your week to be on-call, then you are “all in”. You don’t venture far from home, you can’t attend special events and you have to apologize to your significant other a lot. When you are not on-call, you tend to unplug from the company. These are normal behaviors and completely understandable given the difficulty of the job.

    The problem is that this on-call way of life is not conducive to actually fixing problems faster. In a perfect, company-centric world, everyone would get paged for every problem with the logic being that one of the team members actually has the information needed to solve the problem. But that’s not going to happen. And unfortunately, the truth of the situation is that you’re not sure who that one person with the correct knowledge to solve the problem will be.

    The question then arises…how can team members have more knowledge about what is going on with the complex systems in their architectures while also maintaining their quality of life?

    Consider Twitter and Facebook user behavior. I have a cousin that lives in Chicago. I have seen her in-person once in the last 30 years, yet I know she remodeled her living room, I know about her work and I know where she took her last vacation. I know these things because the Twitter and Facebook experiences leverage something called Continuous Partial Attention, a term coined by Linda Stone in 1998. Basically, I pull my phone out of my pocket, swipe once, and gain a disproportionate amount of information for the effort (or conscience thought) I put into it.

    If you’re in DevOps, then the idea of Continuous Partial Attention is something you’re already living daily. Now imagine putting CPA to work for your team. Everyone can have an eye on the infrastructure, but not “feel like it”. The on-call team member is still responsible but if everyone has an idea of what is happening, the likelihood of someone knowing the answer is revealed sooner. Now everyone wins because the problem is solved faster, getting the team back to their lives and the company back to revenue generation.