Category: Features

  • Automating code helps businesses adapt to customer demand quickly

    Automating code helps businesses adapt to customer demand quickly

    This is the first article in a series of pieces on digital innovation, accelerating change and the rise of the coded business from innovators in the Chef (click here) community.

    How does the need for speed and scale affect businesses today?

    Let’s start with scale. As a technology company grows, its infrastructure grows – more machines (or machine images), more networking, more data storage needs, more code, more developers, and more users. The only way to scale with growth is with automation. And, as more and more of the world’s interactions happen online (be it commercial transactions, email, social networking, video streaming, etc.), that scale of infrastructure is breaking records all the time. With these new levels of scale come ever-increasing automation requirements.

    On the speed side, first-to-market has always been an important part of business – but what that means has evolved. It’s no longer simply about producing that new cool item before the other guys. Now it also means you must be able to iterate quickly, adjust quickly for changing user tastes, and adapt quickly to the ever-increasing pace of new technologies. To be able to do this effectively, instead of just developing fast, you need to be able to deploy quickly, react quickly, troubleshoot quickly, and update quickly. For many companies, this is no longer icing on the cake, but instead, the structure underpinning their business – in the same way that having an e-commerce site was 10 years ago.

    Speed is becoming ever more critical for business in the digital age. With the rise of social media, a lot of customer feedback is public or semi-public, allowing people to unite behind specific complaints or desires and provide more clear and unified messages to companies. This helps companies get more feedback of higher quality and – hopefully – to react to it more quickly. This is also dramatically accelerating the cycle of getting features/products/releases to users, and, perhaps more significantly, prompting users to provide more useful feedback faster. As I mentioned, the scale and speed of business operations in every industry are growing daily, and, because of that, automation has gone from being beneficial to critical. In the technology world, this automation comes in the form of code.

    Pushing your code faster and faster means you have to be able to update and deploy infrastructure faster, which in turn means infrastructure/systems teams must also be able to deploy, react, troubleshoot and fix faster.

    But code, in and of itself, isn’t necessarily sufficient to meet the challenge of escalating speed and scale. How quickly you deliver that code is what’s important. Continuous Delivery is one technique used to address this need.

    Continuous Delivery (CD)

    People talk a lot about CD and it’s important to remember that this phrase refers to a methodology that has pros and cons.

    For most technology companies, especially those whose product is primarily Web-based, CD is often the best choice to enable speed of delivery. The main benefit is that the closer to “continuous” your release-cycle gets, the easier it is to have an iterative process. Faster feedback from users means you can more rapidly deliver updates based on that feedback. As you approach CD, you approach near-real-time feedback, which allows you to hone your product or feature far faster than if it were sitting in long dev/qa cycles. Another benefit is best described with a comparison. In troubleshooting, a key rule is make one change at a time, so you know which change fixed the problem – or made it worse. CD is an extension of this idea, although it comes earlier in the lifecycle. If you are pushing/delivering a codebase more often, there are less changes per push, and so breakages are easier to track down – and it’s easier to push fixes.

    A common misconception is that an organization either achieves CD or it does not. CD is a spectrum with frequent releases on one end and less frequent on the other. A CD solution for a bank may mean something very different than a CD solution for a gaming Web site. The idea of the CD methodology is continuing to push toward faster deployments, in order to reap the benefits that come from doing so.

    In the end, no matter what part of the CD spectrum a company aims to achieve, they must be willing to adapt and change.

    From an internal IT perspective, this means realizing that this isn’t the old world, where businesses lock users into small boxes to make things simpler. We’re in a world where everyone has the power of the Internet in his or her pocket and there’s enough information out there for anyone to break out of his or her box. Businesses need to provide engineers, designers, assistants, VPs, etc. the flexibility and freedom to work and think in ways they do best, so they can take the business to the next level.

    From an external (production) IT perspective, companies must be able to quickly adapt to new growth and new technology. It took the Internet a (relatively) long time to start adapting to the small displays on mobile devices, and it adapted only slightly faster to tablets. But new form-factors are just the beginning. Broadband is becoming ubiquitous in first-world countries, and companies are just starting to rely on that fact. Yet there are still many users the Web isn’t taking into account very well. Mobile is becoming the default interface for a variety of locales and ages, and many countries have limited bandwidth and/or high latency. Further, it seems likely that we will see drastically different user input mechanisms, as well as display mechanisms, in the near future.

    Technologists need to be agile and readily adapt to technology and infrastructure changes, as well as speed and approach changes, such as CD, which will be required for the next generation of user and technology demands.

    About the author

    philPhil Dibowitz has been working in systems engineering for 12 years and is currently a production engineer at Facebook. Initially, he worked on the traffic infrastructure team, automating load balancer configuration management, as well as designing and building the production IPv6 infrastructure. He now leads the team responsible for rebuilding the configuration management system from the ground up. Prior to Facebook, he worked at Google, where he managed the large Gmail environment, and at Ticketmaster, where he co-authored and open sourced a configuration management tool called Spine. He also contributes to, and maintains, various open source projects and has spoken at conferences and LUG’s on a variety of topics from Path MTU Discovery to X509.

  • Automation, Operations, DevOps – What We Thought Was Missing

    Automation, Operations, DevOps – What We Thought Was Missing

    stackstorm-brandStackstorm is a new company focused on automation, operations and DevOps. It’s co-founder and CEO Evan Powell has been keeping a running journal as the company made ready to emerge from stealth. Over the next few days we will share Powell’s thoughts and experiences as a new DevOps company comes to market.

    Over the next few posts I’ll be sharing my thoughts on why StackStorm – a new company focused on DevOps based automation – was still needed. My first comment is that the users will decide how we overlap or complement existing automation solutions.  So far they are confirming our belief – that we are strongly complementary to trail blazers like Chef, Puppet and of course CF engine and more recently Salt.  Our approach is to focus on operations automation that wires together the data center and that – in the case of systems like Facebook’s FBAR and our own beta level software – learns over time. With that as some context, over the next five days I’ll unpack our key drivers for us starting StackStorm – starting with something I think all readers of devopsy.kinsta.cloud should agree with:

    DevOps and the shift to the cloud is much bigger than generally assumed. Over the last few years, my co-founder Dmitri and I have met with thousands of enterprises, service providers, government agencies, integrators and sundry analysts and experts. And I’ve been speaking about the inevitability of the shift to Software Defined Data Centers for many years while helping to create the software defined storage market as founding CEO of Nexenta Systems. More recently, as StackStormers, Dmitri and I have met with approximately 100 DevOps practitioners with OpenStack projects either ongoing or kicking off. The shift to DevOps – sometimes lumped in with the shift to Software Defined Data Centers and also the shift to the cloud – is the biggest shift in IT we’ve seen in our careers. The top DevOps practitioners are 10-100x more productive than your legacy siloed IT operations department. Software is “eating everything” AND the approach to building and operating software has improved radically. Nearly every company and government in the world must either leverage this step function improvement in productivity of software development and operations or face disruption. It might be worth discussing the 10-100x productivity improvement:
    a. Is it real? Organizations like PayPal and eBay have published their own data showing at least a 7x improvement in development velocity. And of course the DevOps surveys by folks like Puppet Labs have shown even higher improvements in the speed and quality of development. And, again, our own experience indicates it is very real.  One operator tells us that they went from building and deploying 1 new service a year before DevOps to 54 in the 12 months after they got up to speed with a DevOps approach. Revolutionary.
    b. Is 10x a lot? Software getting built and operated 10x better is changing the world. A comparison that Dmitri made in his talk at SCALE12x this past February was to the tractor. As I think any casual student of the 20th century knows, the tractor changed the world by enabling urbanization and more. According to most economic historians, the tractor was about 4x more productive as measured by crops produced vs. input in human and machine energy. IT fairly suddenly figuring out how to develop and operate software 10-100x more productively compares favorably with the most important improvements in human history.
    horse-tractor-1
    So – GREAT. The world is changing. What do we and other DevOps solution providers have to do with this? Ask yourself – why don’t we ALL just jump on the DevOps bandwagon and start building and operating software in this new way? I’ll take on that question tomorrow as the second driver that led us to start StackStorm.
    Feedback welcome.  Comment below please.
  • DevOps and faster feedback: fewer problems, better features (part 1)

    DevOps and faster feedback: fewer problems, better features (part 1)

    As organizations begin to institute DevOps patterns and continuous delivery into their IT shops, one of the most dramatic long-term returns on their investment is that it bolsters the class and consistency of work produced by existing development teams.

    Organizations that evolve to DevOps don’t necessarily need to hire rock star programmers to boost code and feature quality. That’s because, in the long-run, continuous delivery actually fosters an environment that makes it possible for the developers already in place to learn more efficiently and better improve the day-to-day quality of their work. The secret ingredient to this? Faster feedback loops.

    “Shorter time to feedback is the key difference between DevOps and the traditional waterfall approach,” says Aater Suleman, CEO of Flux7 Labs and a professor in computer systems design and architecture at the University of Texas. “The main challenge with waterfall was the fact that there was a very long loop for feedback, which removed the agility from the picture. It could be a month or longer before developers would actually get feedback about what they’d done.”

    The longer the cycle, the more code to sift through to find problems and the longer it takes to fix issues, says Curtis Poe, a consultant and Agile development coach for All Around The World. Shorter feedback cycles make it easier to find issues quickly and attend to them as soon as possible.

    “Having continuous delivery means that typically very little code needs to be reviewed for failure and as a developer, if I get a bug report  for code I wrote in the last hour, I’m going to be far more efficient in fixing in than for a bug report for code I wrote last month,” Poe says.

    This is true not just for out-and-out failure, but also tweaking for improved performance.

    “When frequent deployments are coupled with  well-instrumented monitoring metrics on all aspects of your application,  small nuances in performance can be detected and teased out for relatively  small amounts of code change,” says Jeff Behl, chief network architect at Logic Monitor. “Developers want quick feedback to know changing this particular small part of code can affect this aspect of the application. The sooner you learn how to optimize the application code, the better.”

    Even more important, fast feedback loops are not only critical for fixing past problems, but making sure they don’t happen again in the future. The shorter the time between when a developer writes code and receives information about how well it did out in the real world, the easier it is for them to learn from those lessons and apply them to future work.

    With longer release cycles, be they quarterly or monthly, there’s often a lot of energy expended trying to find the person who created an offending line of code that broke something, says BLANK. The top triage person will go through the ringer to figure out who broke the code, until they might pinpoint something like, this person made this code commit six months ago that broke the application.

    “Does anyone remember what they were doing six weeks ago? Of course not. But if you were writing code at ten o’ clock and I came by at one o’ clock and told you exactly what was broken and I sent you a zip file that had everything that you needed to do to reproduce that on your desktop do you think you’re going t o be a better programmer?,” Gary Gruver, vice president of quality engineering release and IT operations at Macy’s  said last fall at FlowCon, explaining that every developer wants to do a good job and is typically trying to do the best job that he or she can. “If you can’t provide them that rapid feedback , real-time in an efficiency manner, how do you expect people to get better?”

    In the scenario above, the process of developing the offending code is long gone in a developer’s memory, which means it takes more work for the coder to review their work, remember the project and then mentally link that process to the infrastructure dependencies that were affected by it.  With shorter cycles, devs spend less time remembering what they did and more cognitive work on understanding dependencies and the context in which the code operated. Long-term, that’s going to make for better code from the get-go later on.

    “Developers who can understand how their applications work under additional loads, abnormal conditions, and degraded circumstances, will end up designing better applications and systems, and in the end play a more strategic role at the company,” says Arnold Goldberg, vice president of customer engineering for PayPal.

    According to Goldberg, context in development is critical and the faster, richer feedback provided by DevOps allows developers to better understand how application improvements work in production context. And that is huge for informing best practices.

    “Much of it is specific to the fabric of the existing organic infrastructure that a company has, so it needs to be a pragmatic application of it in the context of the company’s current situation,” he says.

    Faster feedback loops aren’t just about making technical coding improvements, either. These more rapid assessment cycles can also greatly improve features and business value offered by developers. We’ll explain more in part two of this series.

  • DevOps and PaaS: ‘Give me a platform. Let’s rock, let’s rock, today’

    DevOps and PaaS: ‘Give me a platform. Let’s rock, let’s rock, today’

    Give me a platform. Let’s rock, let’s rock, today.

    – Dewey Finn, School of Rock

    The term Platform as a Service (PaaS) has become overloaded to define any ‘platform’ that provides ‘services’ to its users. The term PaaS is formally defined by NIST as:

    “The capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages, libraries, services, and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, or storage, but has control over the deployed applications and possibly configuration settings for the application-hosting environment.”

    – The NIST Definition of Cloud Computing, NIST, US Department of Commerce, Sept. 2011

    Like most terms in the IT industry (or to really generalize, the human communication medium known as language), PaaS is overloaded, overused and misunderstood. A quick web search or even a visit to the Wikipedia page on PaaS proves my point.

    I prefer to look at such overloaded terms in context. In this post I am going to look at PaaS thru the lens of DevOps (no surprise to a reader of this website). I will explore what it means to an organization looking to adopt DevOps and leverage the services offered by utilizing a PaaS platform. I will explore and contrast a ‘build your own’ in-house platform as a Service or a subscription based commercial PaaS offered by a vendor. I will use PaaS offerings from IBM to illustrate my points. (Disclosure – IBM pays my mortgage).

    Lets start.

    Defining PaaS:

    The most critical section of NISTs definition of PaaS, that differentiates PaaS from an ‘Infrastructure as a Service’ (IaaS) consumption model for cloud is:

    “… The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, or storage, but has control over the deployed applications and possibly configuration settings for the application-hosting environment.”

    This picture better illustrates the differences.PaaS-IaaS

    As you can see in this picture, what distinguishes the two cloud adoption models – IaaS and PaaS – is how much of the stack is managed by the user (or client) vs how much is managed by the Cloud Platform provider. What is important to highlight is two things:

    1. Every capability in the stack is available as a managed service – this would be a shared, multi-tenant, service available to users, with the underlying implementation abstracted from the user
    2. The Client/user only has to be concerned about managing their own applications, data and user access, leveraging the services available on the platform, which manages the rest

    Platforms as a Service may be either public (IBM BlueMix, CloudFoundry from Pivotal, Google App Engine, etc) or private (hosted CloudFoundry; self built and hosted platform). Organizations may also build their own hosted and managed leveraging multiple technologies, self hosted on a private cloud. For a private instance of course, the organization hosting it in their own data center would be responsible for managing all the services hosted on the platform. Building a PaaS on an IBM PureApplication System, would be a good example.

    DevOps and PaaS:

    When one looks at DevOps on PaaS one has to look at the services that need to be hosted on the platform in order to implement a DevOps services platform. If one looks at a DevOps pipeline and the core components that make up a delivery pipeline up, each of the components need to be available as a service, in order to provide a complete ‘DevOps on PaaS’ solution. Namely:

    DevOps Services

    • Development tools as a service (and/or an ability to connect the services on the platform to a desktop IDE)
    • Build as a service
    • Deploy as a service
    • Test as a Service
    • Monitoring as a Service

    These would be what one refers to ‘DevOps Services’.

    In IBM’s new PaaS offering – Codename: BlueMix (in open beta), these DevOps are services are an inherent part of the platform. There services on Bluemix are:

    • Git hosting as a Service
    • Web based IDE
    • Agile planning & tracking, team collaboration as a Service (Jazz.net)
    • Mobile Quality Assurance (MQA) as a Service
    • Continuous Integration as a Service (Jenkins as a Service)
    • Deployment Automation as a Service (powered by UrbanCode Deploy)
    • Performance Monitoring as a Service

    These capabilities working in concert, provide a Continuous Delivery Pipeline on IBM Codename: BlueMix.

    DevOps as a Service (DaaS?)

    So, to conclude. The value proposition for adopting a PaaS platform is self-evident. If you are an organization that delivers software, and is looking to adopt DevOps, a PaaS offering that includes DevOps services allows you to adopt DevOps at a very low cost of entry. You do not need to craft a Delivery Pipeline and implement the whole Continuous Delivery toolchain. Integrations, hosting, servicing – not your problem. Pay-as-you-go, and allow for scale. It is however, not a viable option for all organizations, as it is a drastic shift in corporate culture and potentially policies. Especially if you look at a public PaaS. If you are a born-on-the-web organization, or an enterprise with born-on-the-web applications, you should take a closer look. Check out Codename: BlueMix at www.bluemix.net.

     

  • Q&A:  Bill Burns on Security Within DevOps

    Q&A: Bill Burns on Security Within DevOps

    Last week’s story, Caution: DevOps Ahead, caused a small stir on Twitter and sparked a response, DevOps: Security’s Last Best Hope, from devopsy.kinsta.cloud’s Alan Shimel.

    Coincidently, I recently had a conversation with Bill Burns, former director of IT security and networking at Netflix. Burns is currently an executive-in-residence at ScaleVP, where he provides strategic advice in the security market. He has experience managing security in hybrid cloud/IT environments and traditional IT environments. We thought now would be a perfect time to share his thoughts on how security can potentially be improved in organizations that embrace DevOps.

    Here’s our talk:

    devopsy.kinsta.cloud: What impact do you think DevOps has on its security teams? How should traditional security practitioners look at that movement?

    Bill Burns: DevOps could improve security if it’s done well, and more importantly I think if the business really buys into DevOps. What I noticed, especially at my prior company, was that as product teams were moving to DevOps, everybody was coming to the table with bite-sized building requirements. They weren’t ginormous, “Okay, we’ll do this in the next six months,” but instead: “What do we accomplish in the next two weeks? The next sprint cycle?”

    It forced you to think of the problems early on and know that you would attack them in bite-sized chunks. The security team had to not just throw requirements over the wall, but had to participate in the design phase and in the negotiations.

    With DevOps, you don’t have this ginormous build cycle where many things can go wrong and tests are often done at the very end. In that sense, if it’s done well, incorporating security into DevOps is sort of holy grail for a really robust software development lifecycle with security.

    Everyone says they want to put security upfront within the design requirements and in the architecture. Because you don’t have the upfront cost of a large architecture phase, and you are iterating, it’s an opportunity for the security team to do that and to participate in tighter feedback loops both with developers and with operations.

    devopsy.kinsta.cloud: Well, it sounds like security teams could very well fall on their faces if they tried to be too rigid with DevOps.

    Bill Burns: Yes. It’s the same as if an operations team came in and said, “Well that DevOps stuff is great, but this is the way we do operations.” They would become irrelevant pretty quickly.

    It’s also an opportunity for the security team to become more consultative and advisory in nature. With DevOps, security can say, “These are the most important things to focus on now.” Maybe those things are the session handling or the encryption. For those critical things, it’s best to start on them right away. Whereas the things that are important, but not critical, can be handled later on and over time.

    And security should be agile enough to pivot with the developer team to handle what it needs to do and make the incremental security improvements as the product changes.

    devopsy.kinsta.cloud: What are some of the barriers security teams face when trying to be a part of the conversation in a DevOps environment?

    Bill Burns: What I’ve seen to be effective is, first of all, to approach the product team or the development team as an adviser. Also, speak honestly and say that you may not have all the answers right away, but that you’ll be a part of the process and learn as the project matures. Start with some of the best practices that you know should be a part of development early, and list things that you will want to be added later.

    If you can prioritize the most important security issues along with the rest of the product team, they’re going to go through the process and will have a better understanding – and security is improved.

    I also think it really helps if the security team members are not solely infrastructure-focused. That they’re not just trying to figure out how to get their intrusion detection system to participate, or how to get your security product to snap into the system. I think DevOps is forcing the security teams to become more like developers. I’ve noticed that a security team becomes more of an application security team and less about buying products and bolting them on.

    Ultimately, the security team is not the risk owner. It’s the adviser. And I believe this is true for many organizations even though many don’t realize it. The security team, in my view, should say: Here’s our analysis. Here’s our impact. Here are our threat models.

    And in that process, security can say that it thinks something is really important and a big risk. But, ultimately, the decision whether to accept the risk is on the product owner or business owner – not on the security team.

  • Some changes to devopsy.kinsta.cloud

    Some changes to devopsy.kinsta.cloud

    Well if you are reading this you are one of almost 40,000 unique visitors to devopsy.kinsta.cloud in the last 6 or so weeks. In this relatively short time we have learned a few things. Based upon these we are making a few changes to devopsy.kinsta.cloud. Some you will see right away, some you will see phased in over the coming days and weeks.

    Here are some of the changes we are making and why we are doing it:

    A. Blog Neighborhoods – From the beginning my vision of our many bloggers and blogs was to organize them into “neighborhoods” that would allow people to hone in on what is important to them.  Unfortunately the neighborhood aspect was not well implemented and all of the blogs were lumped together. Going forward you will see our bloggers and blogs grouped into the following areas:

    1. Enterprise DevOps – This is for DevOps in the large enterprise. Organizations doing DevOps by choice.  This neighborhood will feature stories, tips and tricks on adopting DevOps in a large organization and how to make it work. We have some excellent bloggers in this section and will be adding more.

    2. Doin’ DevOps – Formally the DevOps Life section, this is a neighborhood of DevOps practitioners. Not vendors, people who practice DevOps and work and are sharing their experiences. If you are looking for more technical content, this is a good place to find it.

    3. SecOps – This is for DevOps and Security related content.  Again we have some excellent writers in this section and the security tips range from very technical to how to integrate security into the Dev and Ops process.

    4. DevOps in the Cloud – This is primarily startups and companies using the cloud almost exclusively for their infrastructure. Sort of the opposite of the Enterprise DevOps neighborhood.

    5. DevOps Toolbox – This is going to be primarily vendors writing about their DevOps tools here. Yes, you can take what they say with a grain of salt. But to discount them because they are vendors would be a mistake. No one knows their products and use cases better than the vendors themselves. Not to mention that they have experts in the field who will be contributing. There will be much to be learned here!

    6. The Business of DevOps – like it or not, DevOps is on its way to being a genuine commercial business process. There are companies, executives, investors and managers who are exploring the business of DevOps. Sort of the Yin to Doin’ DevOps Yang.

    We hope by calling the neighborhoods out like this it will make it easier for you to zero in on exactly the type of content you enjoy. We recognize that many of our readers are very technical engineers. There are also many others coming to DevOps who are not. We will continue to strive to make devopsy.kinsta.cloud the best it can be, pitching a tent big enough for all.

    B. Newsletter

    Thanks to so many of you who have signed up for our newsletter. Try as we may, we have just not been able to get our RSS generated newsletter to look the way we want it to. Instead if you receive our newsletter you are getting a fire hose of posts. There are too many of them and they are too long. So instead we are going to move to a hand curated newsletter that will only be sent 2 to 3 times a week depending on how much content is on the site.

    C. Building out community features – The business directory is up with some listings.  I encourage vendors to take ownership of their listings and make them better. The pages are actually getting lots of views every day. It wouldn’t hurt. Our jobs board beta is up, but we need more listings. If anyone knows of job listings in the field, this is a good time to put a free listing up there. We will add more community features shortly, so stay tuned.

    We are also adding more content and writers. If you would like to contribute to devopsy.kinsta.cloud, write to me ashimmy@devops.com and we can see if you have what it takes.  Thanks for taking the time to read to the bottom and supporting devopsy.kinsta.cloud

  • Embracing a culture of continuous incident response

    Embracing a culture of continuous incident response

    In a world of continuous development and rapid iteration combined with being connected online 24/7/365, you are bound to encounter threats. In fact, attackers use automated tools so you are under constant siege from continuous threats. The natural response is to implement some form of continuous incident response.

    The standard development model involves different stages of requirements gathering, writing code, and beta testing, resulting eventually in a defined, periodic software release. That model is being replaced by the DevOps culture of continuous development. Security faces a similar transformation from the pervasive legacy model of the “us vs. them” protect-the-perimeter model to a new system of continuous incident response that addresses the current threat landscape in the real world.

    There is no “us”, and there is no “them”. There is no perimeter. Your authorized users are actually more likely to expose sensitive data, or compromise your systems—whether intentionally or inadvertently—than external bad guys. Your authorized users are also just as likely to be “outside” of the mythic perimeter—connecting from personal smartphones or tablets, or logging in from a Starbucks across town. The traditional security model is simply not equipped to deal with today’s technologies or threats.

    So, what is continuous incident response? I spoke with TK Keanini, CTO of Lancope, who explained, “Some people when thinking about process think of it as a linear model where there is a beginning and end; a better way to model adaptive processes is with a loop and continuous incident response is a loop.”

    He elaborated, “Borrowing from John Boyd’s OODA loop, continuous incident response is where there is an observation, orientation, decision and action cycle that outpaces or runs at a faster tempo than the adversaries.”

    Zane Lackey, director of security engineering for Etsy.com, gave a presentation at the Hacktivity conference on the topic of “Attack-driven Defense”. The concepts he described are essentially the sort of logic and pattern that need to be applied for continuous incident response.

    There are a few key elements organizations need to embrace in order to effectively implement continuous incident response. The first is the loop Keanini described. In order to be “continuous”, you have to continue to cycle through the various phases. There is no “end”.

    One of the crucial components of continuous incident response is continuous monitoring. It isn’t good enough to scan the network once a month, or once a week, or even daily. The threat landscape is constantly changing and evolving, and you have to be continuously monitoring for new vulnerabilities, new attack techniques, and anomalous or suspicious activity that might indicate a security incident is occurring.

    Another important factor is to view continuous incident response as a cross-functional business process. Keanini stated, “I think continuous incident response—when implemented well—touches all critical parts of the business, not just IT. Making it an IT thing is just too narrow and people who experiment in this manner are destined to fail.”

    Companies that have a DevOps mindset, or have adopted DevOps principles will find it easier to relate to the concept of continuous incident response. DevOps organizations are not constricted by the departmental barriers of traditional companies. The hierarchy and red tape of an org chart can get in the way of effective continuous incident response, but DevOps businesses have a more open, fluid organizational structure that makes it much easier to adapt to the dynamics of both operating on, and responding to threats from the Internet.

    Adopting continuous incident response may be easier said than done. It is a different approach to security, which may make it more difficult to get buy-in and support from executive management. You may also encounter resistance from the IT admin and existing security personnel because they’re used to doing things the old-fashioned way, and change is uncomfortable.

    Keanini sums it up well, though. “Just step back and ask yourself how important being connected to the Internet is to your business or customers—and then just consider that continuous incident response is the cost of doing business on the Internet.”

  • Security and DevOps showdown

    Security and DevOps showdown

    caution

    George Hulme’s article on proceeding with caution on DevOps implementation offers some advice from a few well known security experts. Article here

     

     

    vs.

    helpmeobiwankenobi Alan Shimel says today’s speed of business isn’t compatible with :go slow”. Security people have to understand that DevOps is our last best hope. But in order to have DevOps help us to security nirvana we need to change how we interact with the rest of our IT teams. Article here

  • DevOps: Caution Ahead

    DevOps: Caution Ahead

    Despite the continued adoption of enterprise DevOps practices, some organizations, especially those in highly-regulated industries remain cautious about moving forward too quickly.

    “There’s no doubt that DevOps brings benefits for some organizations,” says Martin Fisher, director of information security at Atlanta-based WellStar Health System. “However, many pushing for DevOps underestimate the amount of technological and cultural change that is necessary to overcome before moving ahead to boldly, especially among those in security and regulatory compliance roles that are concerned with securing and auditing processes they see as they’re potentially losing control over,” Fisher says. “You can’t audit and secure what you can’t control,” he says.

    Eric Cowperthwaite, former chief information security officer at Providence Health and Services agrees. “There are some who want to move too fast. In many organizations the culture just isn’t yet there, especially where they are used to very rigid quality assurance and audit controls,” says Cowperthwaite, currently a VP at Core Security.

    It’s certainly not uncommon to find chief security officers in heavily regulated industries who are concerned that the move to DevOps is an excuse to cut corners and move more quickly by cutting out necessary oversight. “You’ll find if you have a developer that’s producing a dozen mistakes a day, if you poke at them about it then you’ll find out that they’re tired of the structure, of the bureaucracy,” he says. “DevOps need to be done in a mindful and thoughtful way,” Fisher says.

    Cowperthwaite agrees, and contends that in certain environments DevOps will be – should be – taken slowly. “You have legacy systems and very highly audited and controlled systems where DevOps is just not going to flow for some time,” he says.

    It’s not just the cultural shift, but also the shift in necessary for good governance capabilities to keep up, they contend. “Not all systems are geared to handle 200 commits a day, nor do they have the ability for reasonable tracking and accountability for that kind of speed. In fact, there are many legacy systems were it’s not even possible at all,” Fisher says.

    However, not that all companies that move toward DevOps are attempting to automate everything – at least not at first. But it is higher levels of automation most certainly seek. A survey recently conducted by automated server management software provider JumpCloud found that many companies are turning to DevOps to automate time consuming activity, such as deployment, patching, user management, log file analysis, and forensics activities.

    Some of the trepidation is warranted, to be sure, and some of it may be due to the media and industry focus on the outlier DevOp implementers. “I think, justifiably, a lot of the fear comes from the fact that a lot of the companies that have gotten the most press regarding DevOps are doing things that are radically different than what is going on within the bulk of IT work at traditional enterprises such as banks, healthcare, and organizations like that,” says David Mortman, chief security architect and distinguished engineer at Dell Software Enstratius.

    When it comes to enterprises with trepidations when moving toward DevOps, all of the interviewees we spoke with agreed the key is to move slow. But still move and develop the culture and the toolsets slowly. “Pick one system that is separate and use it as a concept,” Fisher advises. “The DevOps ‘true believers’ are given a hall pass to fully run their systems and group in a DevOps model. As they learn and improve you slowly assimilate into the broader environment,” he says.

    Mortman sees it in a similar way, but notes that it’s not just about technology and absolute speed, but also collaboration, culture, and relative speed. “How fast is fast for your organization? If you’re only making changes to your big medical records database every 18 months, and you switch to every three months – that’s faster. That’s more continuous deployment than you were doing before. No one has to jump right in,” Mortman says.

    “I started working with those sort of DevOps-y concepts back before I was aware of the term “DevOps,” Mortman adds. “I was calling it “Agile InfoSec” and “Agile Ops.” And with that mindset, even the most regulated, paranoid enterprise can embrace DevOps. Start with small, modestly achievable goals and build from there. Collaborate and more tightly integrate the teams.

  • What Cisco’s OpFlex means for DevOps

    What Cisco’s OpFlex means for DevOps

    Some big news in networking land at Interop this year revolved around Cisco’s latest contribution to open source and the Internet, its proposed “OpFlex” control plane protocol.

    The protocol itself is focused on communicating with network elements and specifies encoding in a variety of formats, including JSON, that are developer-friendly. That’s nice, of course, but the big question as someone interested in operations and devOps is why should you care about what appears to be a networking protocol? After all, beauty is only interface deep. What’s in it for you?

    To see where DevOps fits into the picture, consider the description of OpFlex as specified in the IETF draft of the OpFlex protocol:

    The OpFlex architecture provides a distributed control system based on a declarative policy information model. The policies are defined at a logically centralized policy repository (PR) and enforced within a set of distributed policy elements (PE). The PR communicates with the subordinate PEs using the OpFlex Control protocol. This protocol allows for bidirectional communication of policy, events, statistics, and faults. This document defines the OpFlex Control Protocol.  [emphasis mine]

    What’s being offered is not just a protocol for bidirectional communication, but an entire architecture for sharing policies and actionable data. What’s more important to those who for whom “networking” is not their native technical tongue is that policies are designed to be declarative and abstract. That means instead of issuing a set of commands, often specific to the device or service, a policy declares (describes) what’s needed and each component participating in the architecture interprets those needs according to their specific function and capabilities.

    You might be thinking at this point (I know I certainly was) “What is this policy and who is defining it?” because that seems pretty central to the value proposition of this whole endeavor.

    For that, we need to look to the OpenDaylight (ODL) project which has created a new project called the ODL Group Policy plug-in. As explained by Cisco in an OpFlex white paper, “the goal of this project is to provide a policy-based API that can serve, in practice, as a standard information model in OpFlex implementations.”

    Jumping over to the ODL Group Policy Plug In wiki, then, the intention of the Group Policy Plugin is to introduce “the notion of groups of endpoints and policy abstractions that govern communication between these groups. The goal is to support a northbound API that accepts abstract policy based on application requirements while offering numerous concrete southbound implementations for programming network element with configuration/programming stanzas rendered from application policies.”

    Okay, so that’s a whole lot of .. words. What it boils down to is that instead of a policy describing a set of rules governing routing with the appropriate ACLs, one might simply declare that “Web Server A can talk to Database Server A to perform database transactions” while “Administrative Console B can talk to Database Server A about administrative tasks.” If you were to translate that into specifics you would likely have multiple imperative command lines, each describing VLAN participation and routing  as well as entries allowing traffic between specific VLANs or IP addresses on the ports appropriate to the application layer protocols. It would be long and prone to misconfiguration simply due to the number of commands required. Using a declarative (abstract) policy language simplifies not only the policy but also addresses the disparity inherent in competing products across a given market by eliminating it from being problematic.

    The problem with traditional proposals to solve the “different definitions and object models” across a given market space (like ADCs or IPSs or switches) is that they’ve all relied on finding the lowest common model, which inevitably ended up with all offerings needing to dumb down (commoditize) their value in order to ensure consistent support for all “XYZs”. By abstracting to the policy – and not the object model – level, the OpFlex approach effectively eliminates this issue and emphasizes defining what is desired by the application, but not how.

    Now, the actual policy declaration is unlikely to take the form of natural language (that’d be nice, but we’re not quite ready for that… yet, though Watson might disagree with me on that) but the general premise is to provide a far more, shall we say, friendly policy language.

    What this means is the operations folks most likely to be familiar with the application architecture and who understand what it needs and how it needs to work, could be the ones to actually codify that into policies. You’ll note that the intention for OpenDaylight, at least, is to “support a northbound API that accepts abstract policy based on application requirements.” That means someone – hint, hint, that’s you – might be able to integrate into the application deployment lifecycle provisioning of the network services and topological constraints required based on an understanding of the complex web of connections between each piece of an application. Someone who knows from experience that a “three tier architecture” means more than just three components, and can more effectively describe how they should be interacting.

    But that’s just the beginning. You’ll note that in the RFC summary above it specifies “bidirectional communication of policy, events, statistics, and faults.” That means actionable data is being exchanged between participating components. Actionable data enables, well, action. Actions that might address a fault that occurs between end-points. Actions that might address performance issues. Actions that might mitigate attacks in real-time.

    Ultimately this level of capability will drive new solutions – new workflows and processes – that take advantage of event-based systems and policy-driven networking. OpFlex should enable statements like “When X happens, do Y” to be executed upon, even if it requires coordination of three or four or more network components.

    What OpFlex means for the devops practitioner is a way to influence (or, one hopes, specify directly) what services the network provide for the applications you’re tasked with deploying and delivering, and how the network responds to the things that happen on any network that impact the performance, security or stability of those applications.

    As networking continues to recognize and address the need to provide application-driven services, including provisioning and real-time adaptation, it’s going to continue to become more programmable and integrated with the systems driving a more automated, orchestrated data center. And that means devops practitioners are going to have more and more options as to how they manage the entire application (and network) service lifecycle.