devopsy.kinsta.cloud

Category: Features

  • Inject failure to make your systems more reliable

    Inject failure to make your systems more reliable

    Injecting failure into your infrastructure to test your services resilience has been gaining popularity. Most people think of Netflix and their use of “Chaos Monkey”, in fact, they have an entire “Simian Army” of tests and drills they run to make their systems more reliable. At PagerDuty, we couldn’t just copy what has been published about their practices. Instead we needed an approach that would work for us. We have a relatively small infrastructure compared to Netflix and wanted to develop a practice that enables a team of our size. In response to this need, we created Failure Friday.

    The idea behind it is simple: We block off an hour of time and deliberately shut down certain pieces of our infrastructure to see how our service holds up. For example, our first attack will test process failure. We’ll typically run something like “sudo service cassandra stop” command, keeping the service down for about 5 minutes. In this time, our service should be resilient enough to continue processing traffic. If not, immediate action will be required. The main goal of injecting failure into our systems, of course, is to establish what will happen in the event of a node, datacenter or network outage. We can then use this knowledge to make our infrastructure more fault-tolerant.

    Benefits of Breaking Your Systems… On Purpose

    On the surface, injecting failure into your systems seems counter intuitive. Why break your systems, especially on a Friday just before everyone is about to go enjoy their weekend? Since beginning our weekly failure testing in June 2013, our results have uncovered a number of benefits, which we believe give reason for all teams to develop failure testing scenarios of their own.

    • Uncover implementation issues that may reduce resiliency. We have the opportunity to fix this issue before they cause an outage resulting in an alert outside of business hours.
    • Proactively discover deficiencies in our infrastructure. These deficiencies may eventually become the root cause of an outage, allowing us to fix them before they break in the wild.
    • Builds a strong team culture. By coming together as a team once a week, we share knowledge. Our Ops team learns how the development teams debug production issues, while Devs gain a better understanding of how their software is deployed.
    • Helps us onboard new employees. Failure Friday has allows us to walk through actual failure and alerting scenarios with new hires so they can shadow and learn to handle outages at 11 AM instead of 3:00 AM.
    • Adjust alerting and on-call schedules. Team members participating in failure testing should receive all the alerts that they would during an unexpected outage. This is a great opportunity to make adjustments to limit noise and also ensure you are receiving all of your critical alerts. When we get an alert, but no action was required we know we need to make adjustments to your monitoring thresholds. Or if we don’t receive an alert, but we need to take action, we need to adjust our thresholds so we don’t miss that or a similar alert in the next time.
    • Reminds us that failures are inevitable. Failure is not a just a random freak occurrence, it can and will happen. All code that engineering teams write is now tested against how it will survive during a Failure Friday scenario.

    Develop Your Own Failure Friday

    When you consider putting together your own version of Failure Friday you must remember that communication between teams is key. Failure Friday will not likely be successful if performed by individual teams.

    Step 1: Bring your teams together.

    Get key stakeholders for each of your teams together to discuss each other’s testing needs and decide what to test. It will not be possible to test every possible scenario, so putting together a list of priorities that is meaningful for everyone will help you accomplish quite a bit in a short session.

    Step 2: Find a time that works best for your team.

    At PagerDuty, we choose Friday for a one-hour session because there tends to be fewer deploys on Friday. This way we aren’t creating any roadblocks for our business with our testing practices. Since it’s a recurring event everyone in the company can expect Failure Friday to occur and be prepared. Though Friday can seem like a risky time, having run failure testing you may catch potential threats, leading to a quieter weekend.

    Step 3: Set Up a War Room

    Having a common place where all your teams can sit together will allow your teams to seamless work together. They will also benefit by observing and learning from members of other teams in how they resolve incidents. For PagerDuty, we set up camp in the corner of our dining area. You don’t need to do anything fancy.

    Step 4: Prepare for your first attack

    Before injecting failure, we’ll disable any non-critical cron jobs that are scheduled to run during the hour. The team whose service we’re going to attack will be ready with their monitoring dashboards to track changes and be take action, if necessary. (note: cron jobs are disabled because we don’t want them to interfere with the attack. For instance, we wouldn’t want a Cassandra repair run in the middle of taking a service down.)

    Step 5: Set up your communication Channels

    Because communication is imperative during Failure Friday you will want to use Failure Friday as a way to also test your communication channels. Using a chat room is an essential useful way to exchange information non-verbally. Chat rooms also provide timestamps so you are able to log and review actions that are taken in order to correlate any actions to metrics you may capture during your testing scenario.

    It’s All About Reliability

    Failure Friday doesn’t end when we resume normal operations. From our learning we will assign action items to team members in our ticketing system (just as we’d do if a piece of our system went down for real), allowing us to take precautions in order to prevent these scenarios from occurring in the wild. By putting people in one room and throwing problems at them, we make both our team and our product stronger.

  • Security in a DevOps world: Q&A with Gartner’s Ben Tomhave

    Security in a DevOps world: Q&A with Gartner’s Ben Tomhave

    As the infosec community catches up with the spirit of the DevOps movement, many CISOs and IT security leaders still need guidance on how to make the paradigm shift. Not only are these security folk scratching their heads to figure out the additional risks posed by continuous delivery, but they’re also learning how DevOps can actually help them improve their effectiveness and within their IT ecosystem.

    ben tomhaveGartner recently took a deep dive into this topic with a report entitled “Security In a DevOps World”. devopsy.kinsta.cloud caught up with one of the lead authors of the report, Ben Tomhave, research director for Gartner, to discuss some of his findings and to learn more about how he thinks security staff need to adjust philosophies and practices to peacefully co-exist with DevOps.

     

    devopsy.kinsta.cloud: How do you think security people need to change their philosophy in order to get the most value out of DevOps practices?

    Tomhave: Getting away from that adversarial ‘party-of-no’ attitude is very key. We have to get away from “Come to us, we’ll give you things” and start thinking of partnering, collaborating and mentoring. On the development side, security needs to be pulled in earlier in the process, providing faster feedback after code commits. And then it needs to be working with operations to provide secure foundations, really inherently secure environments that are secure by default so that nothing is changing after the code is written.

    Because that’s been a huge problem, too, right? Where developers go write code in who knows what environment and then it propagates to QA, which looks a little closer  and sends it to production. Then it launches and suddenly there are all these surprises, things are breaking left and right and there was no communication along the way. Everybody was working on different platforms, and ops maybe wasn’t even managing or maintaining the development platforms. We’ve got to get away from that and security is no different—it should be a part of the process. All the monitoring tools, all the hardening tools, all that stuff needs to be applied from dev through QA to production.

     

    devopsy.kinsta.cloud: What would you say are the biggest security concerns or risks that crop up in DevOps environments? What kinds of things could go wrong if security teams aren’t embedded in the DevOps patterns?

    Tomhave: I think that it ‘s the same old stuff, pushing out code that may be functionally  correct but woefully  insecure  and all of the other gotchas, whether it be bad crypto, bad handling of identity management and insecure management of data. DevOps shops are not immune to these problems, either. One need only look at the Spotify disclosure that they had password resets because of a breach. That’s a big DevOps shop and clearly some error was made.

    The good news is that a DevOps shop that’s doing lots of code pushes should be able to resolve issues  relatively quickly. But that means it is  incredibly imperative that they work closely together so that when there’s a break it is quickly triaged, remediated rapidly and there’s good clear communication between the incident responders–whether they be security incident  responders or operations incident responders—and developers, getting the feedback back to developers in a  timely fashion.

     

    devopsy.kinsta.cloud: It’s clear from your report that you think monitoring is more important than ever to support the rapidity of the DevOps cycle. Why is that?

    Tomhave: You know, we expanded that DevOps mantra of fail fast, learn fast to add recover fast in there, too. If you want to fail fast, that’s cool. And if you want to learn fast, that’s also cool. But you also need to have rapid recovery as well, because the sooner you  can detect a breach or an incident  of some sort or a security event, the sooner you can start interdicting and  remediating and taking positive actions to clean it up.

    If you do DevOps with lots of operational transparency into process, but you don’t have adequate  transparency into what the project is doing with the data involved and overall security monitoring capabilities, you could have a lot of problems. Suddenly now we’re going to be rapidly  iterating through  features and code pushes that are  expanding capabilities and then we never know whether or not that environment is being attacked or whether or not it is being adequately secured or if it is breaking down. So that transparency meme needs  to expand out to security monitoring as well so that you have security operational transparency as well as process transparency.

     

    devopsy.kinsta.cloud: I was really interested in your advocacy for a security architect to help facilitate security meshing into DevOps. What does an organization need from this role and what are the challenges of finding the right person to do the job?

    Tomhave:  The challenge here is what you really need is someone who is really competent with development and preferably has some actual experience that allows them to establish credibility with the development team.

    But they also must be really strong security people as well. So they understand static analysis, dynamic analysis, they understand what he tools are doing, what things are looking for. They must also understand design principles from a security perspective, things like here’s why you don’t roll your own crypto, why you integrate with existing federated identity systems, why you do transport encryption and what an appropriate level of protection is for a given system.

    Finding those two sets of qualities in a single person can be really challenging. It is really difficult to find them and, more importantly, once you do find that person, they tend to be very expensive, especially for smaller shops. You think of all these mobile start-ups, for example. Their feet are being held to the fire; they’re being subjected to regulator actions. We’ve seen the FTC go after some of these mobile app companies for violating privacy policies and things like that. So how do you get those security bodies in there when you may not be able to afford to hire someone?

    Alternative staffing models really speak to that. They may need to find consultants through specialty shops or something along those lines to get someone to come in and be engaged either through a staff augmentation type of role or just on a per-project basis.

    I think the big key is being able to do that around the design review up front and catch some of these problems early. Because no matter how much you invest in tools and how many times you scan the code base–either statically, dynamically or whatever–that’s only going to check maybe 40 percent of your code. And a lot of these functional weaknesses, you aren’t going to be able to detect them with tools. You have to have somebody look at the design and say, ‘Oh, you’ve forgotten completely to do encryption,’ or, ‘Oh, I see you’re hashing, but you haven’t salted your hash, so how are your passwords secure?’ or ‘Why are you dealing with credit card data here, this shouldn’t be done this way, you should just be interconnecting into our payment system over there. Quit reinventing the wheel.’

    Those types of things need to be caught by architects early on, rather than trying to catch it using tools in the code.

     

    devopsy.kinsta.cloud: Speaking of tools, you dedicated time in your report discussing the importance of tool chains and automation. Automation has always been important to security, but what’s changed now with DevOps?

    Tomhave: Well, I think that the big thing is almost all of the testing has to be automated up front. Static code analysis should just be automatically triggered on a nightly repository of check-ins,  as much as possible. Same with dynamic analysis. Dynamic app sec testing should be done automated and feedback should just be fed back directly into the issue management systems. There shouldn’t be much human touch there at all, aside from monitoring for, addressing and resolving false positives in those tools. And possibly adding some customized checks looking for specific libraries or frameworks that should be in place or should be used within the tools.

    The only time where I think we want to have any sort of manual interruptions are the first up-front quick risk analysis. Deciding is this a high risk project that’s dealing with sensitive data or has uniquely high uptime requirements, or is this a lower risk project where we’re not really concerned about losses here? I think that, as well as some of the design review considerations would need some manual hands and eyes involved.

    And then everything should automate until you get to the end where you say, ‘OK, this is a  high risk project, we need to put extra testing into it, so  now we’re going to kick off pen testing, where we need to  have some manual validation in place.’ That validation not only deals with false positives but really does a deeper dive into the application. So you fuzz the interface, it looks like that broke some stuff and you say ‘Yeah, we validated and proved it and here’s the exploit. This is how it broke.’ Then feed that back in a credible fashion to developers to show there’s the spot in the code with the errors or this is the logic error that led to whatever the condition may be.

     

  • Your End Users Want Everything: Here’s How to Give it to Them

    Your End Users Want Everything: Here’s How to Give it to Them

    In today’s application-centric business environment, the end user is king. Whether we’re talking about being first to market with the next big thing or adding new capabilities and features to an existing application, so much of our current development time, resources and energy is spent on trying to meet the needs of end users.

    It’s not difficult to see how we got here. The proliferation of social media and mobile and other connected devices has empowered end users to make their voices heard – and whether their opinions are positive or negative, we have no choice but to thoughtfully consider them if we want to produce market-leading applications.

    At this point, you might be thinking, “How is this different? End users have always been king.” You’re right, but the difference between then and now is that we used to build roadmaps and select features based on what we thought they wanted. Now, thanks to social forums, user reviews and other avenues for collecting feedback, we know exactly what they want. The trick, of course, is finding ways to deliver on that knowledge quickly and at a high level of quality without breaking the budget.

    Taking Agile to the Next Level

    Agile was one of the first methodologies to incorporate user feedback directly into development. You know the drill: choose a small set of capabilities to develop, release the build to a select group of users, collect and incorporate feedback into another build and continue iterating until the application is ready for production.

    One of the goals of DevOps – which CA defines as best-practice application delivery methods that improve communication, integration and collaboration between development and operations – is to enable Agile-like development practices that incorporate end-user feedback from after an application is on the market.

    We talked a bit earlier about collecting such feedback. Whether you read forums, poll customers or even collect usage data directly from the application, you know that getting the feedback isn’t the hard part – it’s incorporating that data into the development lifecycle in a way that meets your speed, quality and cost requirements.

    Starting with Pace-Layered Approach

    When it comes to being nimble in the software development lifecycle (SDLC) and reacting to the marketplace, one of the challenges many enterprises have is that their IT infrastructure is complex, comprised with legacy and modern technologies that have been assembled over time. According to a major analyst firm, the key to speeding development in such environments is to segment the infrastructure into layers: namely, systems of record, systems of differentiation and systems of innovation.

    For systems of record, think foundational technologies like mainframes that have typically lengthy development periods, versus systems of innovation, such as mobile apps or cloud services, that have a lighter footprint and are more flexible. Once you’ve categorized your components into these layers, you’ll be able to react to customer feedback most quickly with your systems of innovation – and there are two types of tools to help you do just that.

    Application Delivery Tools That Speed Development and Delivery

    With a service virtualization tool, you can simulate constrained or unavailable systems across the SDLC, allowing developers, testers and performance teams to work in parallel for faster delivery and higher application quality and reliability. In other words, service virtualization enables you to decouple your systems of innovation from the slower moving systems of differentiation and record, so you can quickly deliver on customers’ requests, even amidst all the change and complexity in your IT environment.

    To further accelerate the SDLC, a release automation tool can automate complex, multi-tier release deployments through orchestration and promotion of applications from development through production. This tool is your assembly line, driving the application from code development to production, so you can get new innovations into your customers’ hands faster than ever before.

    Addressing end-user feedback is an ever-evolving, constantly moving target. But if you follow Gartner’s pace-layered approach and plug your systems of innovation into the right application delivery tools, it’s a target you’ll always be able to hit.

  • Enterprise DevOps is Real and Here to Stay

    Enterprise DevOps is Real and Here to Stay

    A recent column in the Wall Street Journal’s CIO Journal argued that while DevOps is gaining inroads in small and mid-sized businesses, it’s facing a much less rosy future in large enterprises. The story cited organizational challenges, many of which we covered, such as large siloes, regulatory compliance, and resistance to change as great impediments enterprises embracing DevOps.

    The column also spoke of the buzzword “DevOps,” how it’s a software development methodology, and spoke about “holistic” DevOps tools. Interestingly, that’s not how we view DevOps. Not at all. DevOps is more a movement about organizational collaboration and driving improvement.

    Still, the author wasn’t completely wrong. There are significant hurdles to overcome when moving to a DevOps culture, and those that the author cited are the primary obstacles for both enterprise and SMBs – mainly cultural changes the processes to support it – and they are being overcome. Just, typically, at a slower pace among larger organizations.

    “I think the SMB vs. enterprise is the wrong axis to use,” says Donnie Berkholz, an analyst at Redmonk. “Like any change, it’s going to be a lot easier for greenfield teams and projects, so new startups will have an easier time starting from scratch with a DevOps mindset. And there’s also no guarantee that SMBs are new — in fact many of them have been around for a long time and have similar inertia, relative to their sizes, as large enterprises do,” says Berkholz.

    And, as Berkholz explains, one of the big challenges for companies considering DevOps is whether the benefits of the move exceed the switching costs. “Fortunately we’ve got growing data, such as the Puppet Labs survey, showing that DevOps greatly accelerates time to delivery, although quantifying “first to market” in terms of dollars can be challenging,” he says.

    Bigger ships turn more slowly, but they still turn
    Jeff Sussna, founder of the IT service innovation consultancy Ingineering.IT, concurs and says change always happens more slowly in enterprises than SMB’s, simply because they’re larger and more complex. And just because enterprises are bigger ships that turn more slowly, they’re still turning, though it may be more difficult to perceive immediately. “I think people are confused and [they] think DevOps requires either smashing org charts and mushing everyone into shared teams, or else rolling out some big expensive overarching tool across the entire organization,” he says.

    The reality is DevOps is manifesting when teams collaborate and partner much more closely – no matter their size within the enterprise or if such change is occurring at a sweeping pace, at first. It’s when operations teams seek developers’ input, and when developers reach out proactively to operations. When teams are working to improve continuous QA, they are performing DevOps. “I think the fundamental problem with DevOps at the moment is we think it’s a destination, when really it’s a path,” says Sussna.

    And Berkholz adds that just because it’s a slower and more cautious in the enterprise doesn’t mean it’s not transformational. “Like any other organizational transformation, the path of least resistance tends to be one of incremental change rather than a wholesale shift. Switching one tool at a time to a tool chain that encourages DevOps-style behavior and discourages silos can help catalyze that transformation. Continuous integration and continuous delivery/deployment pipelines are great examples of this,” he says.

    And those are exactly the kinds of transformations underway at the enterprises we speak with every day.

    Big rivers often begin as small streams
    As for the business benefits, many organizations view it as a matter of survival. “I think over the years folks have been amazed how long (not to mention how expensive) it has taken just to bring basic levels of good, let alone best practice in to IT. There have been more than enough failed Bataan Death March style best “implementations” to make any executive wary of the next new thing. In my opinion this is the real opportunity cost of old scientifically disproven mechanical metaphors for human organizations,” says Kevin Behr, chief science officer, at enterprise opsflow consultancy Praxis Flow, and co-author of The Phoenix Project and Visable Ops.

    “I believe if your org can not improve at the individual level it is incapable of getting much from DevOps, or anything else for that matter,” Behr says. That is why I have focused on first evolving some sort of continuous improvement form or kata using the scientific method as its base. This capability starts day one helping to create resilience, which is the ability to handle change. With that you can “improve into DevOps” iteratively and within the guardrails of your organizations overall direction and strategy. We call this approach Opsflow, as you’re flowing Ops in to Dev, then to test, then to infosec, and then on to the rest of the business,” says Behr.

    And the data increasingly show that the move to DevOps is in fact well underway. And happening it already is, at least according to numerous and recent studies. One is the Vanson Bourne study (funded by CA Technologies) of 1,300 senior IT decision-makers, at enterprises with $100 million in revenue, and within financial services, healthcare, manufacturing, public sector and telecommunications industries from 21 countries around the world. In that study, a decisive 99% see a greater need to move foreword with DevOps.
    Additionally, the 2013 State of DevOps Report by Puppet Labs found that the majority of those that have implemented DevOps have found significant benefits including improved quality of software deployments, more frequent software releases, improved visibility into IT processes, cultural change, and IT being more responsive to business needs.

  • DevOps and Enterprises, it’s a culture thing

    DevOps and Enterprises, it’s a culture thing

    Today author, DevOps evangelist and devopsy.kinsta.cloud Advisory Board member Gene Kim published an article in the CIO Journal of the Wall Street Journal.  The article was a response to an earlier article by Rachel Shannon-Solomon in which the author claimed that DevOps was not likely to be successful in the enterprise at least not at the present time.  Rajat Bhargava, CEO of JumpCloud and another Advisory Board member of devopsy.kinsta.cloud wrote the below in support of and amplifying Gene Kim’s thoughts in the WSJ article.

    Recently an article was published in the WSJ CIO Journal that DevOps likely won’t work for enterprises – at least not now. The author goes on to state that there are structural impediments to enterprises adopting DevOps including siloed organizations, lack of platform technology, cultural issues, and lack of ROI. I completely agree with the author that enterprises face an uphill battle on the implementation of DevOps. She is right that they have different structural challenges than startups or SMBs.

    DevOps bring companies close to customers

    However, I have to flat out disagree with her conclusions and many of her data points that are extrapolated into the conclusion that DevOps won’t work in the enterprise. Let’s start at the beginning where the author states that DevOps is “…a software development method that stresses communication and collaboration between developers and IT professionals (QA and ops)”. I’m going to respectfully disagree with that definition. DevOps as a name is a misnomer. DevOps is about collaboration and communication, but not just between developers and ops in the enterprise. This is about communicating across the whole company: sales, marketing, finance, security, and any other relevant area. DevOps is about increasing company performance through better IT execution. The hypothesis is that by more closely aligning what cus with what gets built in a more timely fashion, organizations will sell more products and services. That’s not a small company or SMB issue, that’s an every company issue.

    The author continues by saying “…the value of implementing devops beginning in the early days is clear—achieving radically higher productivity at lower costs resulting in more reliable systems.” I have to disagree here as well. DevOps is not about higher productivity at lower costs resulting in reliable systems. That may be a byproduct of DevOps, but the goal of DevOps is to more closely build what customers want in a shorter timeframe. DevOps is about cycling a product or service quicker to have it match more closely what the need is in the market. From there, if that holds, the organization will sell more products and services. DevOps organizations hope that it will be cheaper and more reliable but it very well may not be right away. In fact, DevOps may be more expensive if you factor in the time and energy required by the organization to run much faster product release cycles.

    DevOps: a cultural shift

    Unfortunately with those fundamental tenets at issue, the rest of the article becomes harder to agree with. The author states that there aren’t any real legacy vendors for DevOps which is a matter of perspective. DevOps isn’t about the vendors or tools ultimately. It is about culture. Every legacy vendor out there is going to adapt their tools to work with DevOps methodologies. Some will succeed. Some won’t. “Shiny” new startups will also compete for the business, but in the end, it’s not about the vendors or tools for enterprises. It’s going to be about cultural change.

    The author is right that silos will make a difference in the pace of adoption. Enterprises are big, complex animals. Cultural change at these organizations is next to impossible. Add in departmental and divisional factions and it’s even harder. But cultural change has happened before and it will again. Will it be easy? No. Will it be fast? No. Will it be free? No. But it will happen. Or the company will fall behind, because their competitor will be able to move faster, innovate quicker, and introduce new products and services.

    The next challenges described focus in on selling to enterprises and the lack of hard dollar return on investment. The first focuses in on vendors trying to sell to enterprises. The author states that enterprises are predisposed to buy. She’s correct that it’s hard to sell to enterprises, but many startups and larger companies have solved that problem successfully before. Historically, that hasn’t been an impediment to adoption of technologies. Does it take longer? Yes, it does, but if the technology is powerful and enterprises believe that it will help them, they will look for solutions to those problems. The lack of ROI is an interesting observation. As with many things in IT, ROI is elusive and it likely will be with DevOps early on as well. Over time, though, DevOps should be a clearly measurable activity. Product cycle time, defects, and ultimately revenue and profit will be measurable. Whether DevOps is contributing to the bottom line or not will be evident and clear – just maybe not in the short-term.

    Fundamentally, DevOps is a different animal at a startup than at an enterprise. This is a topic that we have studied before. The WSJ perspective is an interesting one and can be taken in the right light as a cautionary tale. Enterprises do face different challenges when it comes to DevOps – or any cultural change for that matter. DevOps at an enterprise is a cultural challenge first and foremost. Those enterprises that embrace it as such and focus their efforts on that area first will likely have more success than those who treat it more as a tool issue. There are countless successful enterprises leveraging DevOps. Gene Kim’s response makes that clear. Gene, has been studying these “unicorns” and “horses” and so his response is based on the enterprise DevOps adoptions, successes and failures he has observed. I look forward to more data coming out soon that will shed more light on DevOps in the enterprise.

  • Simplify and expedite server management

    Simplify and expedite server management

    Deploying, managing, and maintaining servers can be tedious. Deploying, managing, and maintaining thousands, or tens of thousands of server instances is overwhelming—bordering on impossible—without the right tools. Thankfully, the right tools exist.

    The Pain of Doing Things the Hard Way

    I know I am dating myself, but when I was an IT admin for a dot.com startup, I had to deploy servers the old-fashioned way. UPS would show up with a box from Dell. I’d unpack it, and mount it on the rack in the datacenter. Then, I’d begin the process of configuring the operating system, and installing the application software. Each server was essentially a full-day project, and we seemed to constantly be adding new servers, so my full-time role was basically “racking and stacking” for a while.

    Adam Jacob was experiencing similar obstacles when he developed Chef. Working as an IT infrastructure consultant, Jacob was frustrated by the configuration tools that were available, and the lack of a simple, repeatable process. Rather than starting from scratch and reinventing the wheel for each client, he developed Chef to enable IT admins to create modular, reusable “recipes” that automate the process.

    Server management doesn’t end with deployment and initial configuration, though. You also have to maintain the servers, and resolve issues that might occur. When an issue arises, it could mean hours of troubleshooting to identify the root cause, and another few hours to work out and implement a resolution.

    I used to get into philosophical debates with management and other IT personnel because it was my opinion that we should just skip the troubleshooting and rebuild the server from scratch. My theory was that it might be nice to know what caused the problem, but from an operational perspective I knew I could rebuild the server and restore normal operations in a couple hours. Troubleshooting, on the other hand, was an unknown—both in terms of whether or not it would even be successful, as well as how long it would take.

    The Beauty of Doing Things the Easy Way

    Tools that automate configuration and deployment of server instances automatically solve both of these problems. They expedite deployment and configuration, and simplify management and maintenance—freeing IT admins and IT personnel to focus on more important, less mundane tasks.

    Tools like Chef help IT manage complexity. It is great from a DevOps, automation perspective to have a tool that enables you to quickly deploy and configure servers, but the real value comes from the ability to maintain a stable, dependable network with minimal effort.

    Chef nodes “phone home” periodically to ensure the configuration matches the recipe it has been assigned from the Chef server. Any deviations are automatically overwritten—so any changes that may have been made locally on the node are erased, and the server remains configured as it should be.

    In order to implement lasting configuration updates, the changes have to be made to the recipe at the Chef server level. The bonus is that once the recipe is updated, all Chef nodes working from that recipe will be automatically updated.

    When an issue occurs on a server instance, it should ostensibly be corrected automatically when the node communicates with the Chef server and overwrites the configuration. However, even if that doesn’t happen for some reason, the use of DevOps tools like Chef still makes the task of troubleshooting and/or resolving server issues much more efficient.

    With the push of a button, you can just deploy a new server—properly configured and ready to go. You could choose to kill the server experiencing issues and just replace it with the new one, or you could launch a new server to take over the roles and responsibilities of the affected server while you take the time to troubleshoot if you really need to understand what the root cause of the problem was.

    Embrace simplicity

    Apparently, I was an IT admin at the wrong time, and I got out of that career path right when things started to get interesting. Necessity being the mother of invention and all that, Adam Jacob created a platform with Chef that reduces the complexity and tedium of deploying and managing servers.

    Chef isn’t alone. Tools like this form the foundation of DevOps because a culture that combines development and operations, and iterates new releases on a rapid scale needs to have this sort of automated IT infrastructure to keep up.

  • DevOps Best Practices: Break Down Silos, Avoid Drift and Optimize for Flow

    DevOps Best Practices: Break Down Silos, Avoid Drift and Optimize for Flow

    In organizations trying to achieve agility with stability and quality, silos (of teams, tools, and processes) create major stumbling blocks to managing configuration drift between pre-production and production systems. Breaking down those silos, improving visibility into configuration drift and facilitating collaboration for improved flow are the keys to overcoming this challenge.

    The challenge of staying in sync when both sides are moving

    Production environments aren’t static. Whether you are applying patches for improved security, optimizing settings for availability and performance, or making changes for a technology refresh, production systems are continuously evolving.  As new releases are being developed, tested, and staged, pre-production environments are continuously evolving too. Ideally you want to develop with production in mind, but that gets harder and harder when both sides are changing and different teams are responsible for different environments and live in their “own worlds.” (e.g. the common split between QA teams, release teams and production operations).

    Even within pre-production, staying in sync is a significant challenge. For example, because applications ride on middleware and middleware settings are unique to each application, keeping systems in sync from QA to staging can be a real challenge.

    QA and staging environments are regularly reconfigured to support multiple releases that are in process at the same time.  The costly way, often employed to minimize changes, is to “protect” QA environments for a certain application or release leading to low utilization and bottlenecks.  The rapid proliferation of mobile applications to front end legacy systems magnifies the challenge by adding another dependency layer.

    Disconnected teams, information, and perspectives (those pesky silos)

    Why are these challenges so hard to overcome if we know the issues?  Many IT teams operate in their “own worlds” with tailor-made but disconnected tools and information resources.  Silos have evolved because they permit the specialization required to deal with the immense, growing complexity IT deals with everyday, but they have now become the biggest impediment to agility and quality.

    For example, the operations team has access to systems management alerts that most others don’t, engineers have access to build systems and scripts that others don’t, developers have access to functional requirements and stories that others don’t, and change approvals are stored in a system of record that only certain people (often managers) have access to – just to name a few silos.

    Coordinating work and sharing information across these silos of teams often equates to lots of meetings, conference calls, and email threads.  This is not the most efficient way to get work done and it’s not particularly effective in sharing information to keep environments in sync.

    “Didn’t you see my email from yesterday with the attachment listing all the middleware settings required for today’s test?”  “Weren’t you in the meeting three weeks ago when I said a Unix server was probably going to be needed in addition to a Windows machine?”  “I told so and so in an instant message on Friday to run this script before he went out on PTO, but he must have forgotten. Who else has root permission to run it while he’s out?”

    Sadly, these kinds of statements are the norm, not the exception, in most IT shops.

    The reality is there is inefficient coordination to manage all the steps and handoffs between application owners, development, QA, release and production functions.  Instead, I recommend thinking about the software development and deployment lifecycle as an “uber change” and “uber collaboration” process that must be viewed and managed from end to end; from business requirements, to development, to QA to staging to production.  All of the different silos of teams must have a way to work together in a unified way with access to critical information from upstream and downstream teams to ensure a smooth lifecycle end to end.

    Automation ≠ DevOps

    The heart of DevOps is collaboration and managing change, yet the conversation often seems to start with automation.  The fallacy here is the belief that we can eliminate the need to collaborate and manage change by removing people from the process and relying on the machine.  This out of balance perspective between the machine and people creates more problems than it solves.

    Automation tools, while important and necessary, are not all that’s needed to truly solve these synchronization and deployment issues. In addition, to have environments only changed by automation tools and never by people during the firefighting of incidents is unrealistic.  You need your people armed with intelligence and tribal knowledge to drive your automations, but all too often we rely on individuals with disconnected perspectives and limited awareness so things get missed or overlooked.

    Automation does what it’s supposed to do, but it makes a critical assumption; what’s being changed is configured how it’s supposed to be configured.  If drift has occurred because critical settings were modified to remediate an incident or compliance issue, without the automation engineer aware of the change, the automated deployment will result in you hearing a loud “boom!”

    As one popular DevOps Twitter handle has said, “To make a configuration error is human – to do it across a thousand servers is DevOps.”  While that may seem like an extreme statement designed for comedic effect, it isn’t extreme to say that thousands of deployments go south every day because systems aren’t configured the way we expect them to be given that the complex software stacks that multi-tiered applications now depend on are touched in so many ways.  Automation alone doesn’t solve the DevOps challenge.

    Negative consequences to business

    Of course, the issues I’m raising here aren’t just IT issues; they are real business issues.  Time to market is typically the number one impacted area. For example, if a two-week development sprint is used to get a business critical need addressed, but then it takes weeks and weeks to get the sprint through the quality cycle and into production due to resource constraints and configuration issues, what have we achieved?  

     Stability and quality are the next big area of impact.  When change is rapidly introduced through new releases and unexpected drift occurs in environments, performance and availability impacts occur.  It is one thing for IT to be perceived as slow and unresponsive to the business, but it is quite another when they are perceived as slow and quality is poor resulting in business impact.

    Just when IT thinks it can’t get any worse, the business asks a presumably simple question, “What happened?”  IT’s typical response, “Let me get back to you on that.”  And then begins a scramble of hours, days or weeks as IT’s best people desperately try to figure out what went wrong.  This is obviously not the best confidence builder for the business.

    Compliance is typically the other casualty as there is often very little awareness of operational policies on the development side of the house so deployment packages and instructions aren’t sensitive to compliance issues.  Building and testing an application without full awareness of policy governance across the whole, multi-tier stack is bound to introduce delays and errors when attempting to deploy in production.

    Visibility, configuration transparency and collaboration; a better way!

    1. Understand drift for the entire application service. Rather than everyone having their own disjointed perspectives on databases, app middleware, network or firewall, and server configurations, provide team members with an understanding from the business service (or application) perspective with direct visibility into the differences between how things should be configured and how they are actually configured for the entire multi-tier application stack.  This enables you to create smarter automations and initiate change processes to bring configurations back in line when necessary rather than waiting for a failed deployment to occur, identifying the root cause, and correcting it while the business waits.
    2. Optimize for flow not individual processes.  As stated in the previous example, making one process faster (e.g. development) without considering its impact on QA, staging and production will generate new constraints downstream.  Instead, view DevOps as one continuous process from goals and requirements all the way through to deployment into production and define success for the team in those terms. When you optimize for flow instead of silos, you will be amazed at how teams share more and collaborate better.

    The benefits of breaking down silos through better visibility, understanding drift and collaboration are significant:

    • faster time to market;
    • quicker ability to react to competitive pressures;
    • deployment with fewer errors;
    • better compliance with policies; and
    • greater productivity for everyone.

     To sum it up, if you break down silos, identify and resolve drift, and manage for flow (while avoiding islands of automation, information, and people) you can unlock agility, maintain stability and ensure the quality your business demands.

  • How to turn executives into DevOps champions

    How to turn executives into DevOps champions

    IT practitioners must bridge the gap of understanding about DevOps with their managers and executives if they really want continuous delivery to take root in their organizations. Next week at Camp DevOps @ Gluecon, Andi Mann, vice president in the office of the CTO for CA Technologies, will propose ways to do that in his talk, Building Top-Down Support for DevOps.

    “You can only push so far from below until you need to start getting support from above,” Mann says, explaining that the practitioner audience that tends to dominate events like Gluecon are running into a ceiling with DevOps adoption because the organization-wide changes necessary require top-level support from managers and executives to carry them out. He believes passionate advocates of DevOps need to do a better job garnering that top-level support.

    From practitioners and even middle managers, that will require some mind shifts and improved ways of approaching people further up the food chain. He gave devopsy.kinsta.cloud a quick preview of some of the suggestions for practitioners he’ll lay out in the talk he’s developed to help them start improving their DevOps executive pitches.

    Understand management’s goals

    Mann: “To start with, they need to understand their manager’s goals and even their manager’s manager’s goals. Employees often don’t think about that, they have tunnel vision and only consider their own goals or sometimes their team’s goals. Practitioners often fall into the trap of looking at something and saying ‘That’s obviously good for the business’ without really knowing if that’s the case or not.”

    Think about politics

    “If you’re looking to go into this major cultural change which DevOps must affect, then it needs to be in line with the organization’s goals. When you think about accelerating delivery of new capabilities, the question might be whether the rest of the organization is ready for that. So if you’re creating a new feature in a front office application, but you’ve got to train 15,000 front office staff in that application, is that something you want to be doing. You might think it’s a great thing, but your manager may say, ‘Hang on, this is just causing work for other people.’ You’ve got to understand where the boundaries are politically within an organization and figure out how to break those down.”

    Bring fact-based results projections to the table

    “How will you make your boss a hero? That’s glib, but you need to think of it in those terms, particularly when it comes to ROI and the business case for DevOps. Start by proving other people’s successes first to your managers and then show how your success will be achieved second. There’s survey data out there, there’s gartner data, and there’s data from vendors on what the outcome of DevOps are. It’s not enough to say to a manager, ‘I’ve got this idea, it’s a philosophy and amovement and a cultural change and it’s going to make our company better.’ Not many managers are going to spend a lot of time listening to that proposal. You need to come in with the ROI and how DevOps addresses their business challenges.”

    Understand and communicate costs

    Mann: “Not enough practitioners actually put together business cases for their managers to take to their executives to support the change in culture—and a big part of that is budget. Think about what training you need, whether there’s going to be additional hires to fill out skill sets. There’s cost of software—the DevOps tool chain is rapidly maturing and not all of it is free. Even the open source stuff costs money. If you want to get the most out of it you’re going to be looking at support contracts, services and training. So there’s cost involved and that is an important part of making that business case.”

    As Mann will explain in his talk, these considerations need to be the building blocks of  a complete and polished proposal. Bringing executives that “finished work” will more likely sway them than a few stray conversations made up of hints and gut feelings about DevOps benefits.

    “Go to your boss with a finished idea. Explain ‘here’s what we need to do, this is the spend, this is the result we’re going to get in terms of dollar returns and savings, speed to market and so on,’” he says. “And then explain, ‘Here’s a project plan for how we’re going to do it,’ preferably with a paced, layered approach proposed rather than a big bang.”

  • DevOps: Getting Past Audit

    DevOps: Getting Past Audit

    New business processes (and technologies) too often throw auditors into a tailspin. It’s no one’s fault, really: the decision making on what IT and process controls that need to be in place moves slow – and it is the tail to the rapidly moving technology dog. In fact, while there is no way a standard or regulatory body can predict technological change from year to year, there are ways to help ease the relationship between auditors and organizations embracing DevOps and continuous deployment practices.

    “DevOps audit is an interesting problem,” says David Mortman, distinguished engineer at Dell (formerly Entratius). “Audit is something you just can’t avoid, especially in any industry that’s regulatory controlled. And auditors are a lot like security people, in my experience, which is to say that they have a lot of responsibility and they are justifiably concerned about things they don’t understand,” Mortman says.

    And concern their certainly is when it comes to any organization that must deal with acronyms like HIPAA, PCI DSS, SSAE 16, SOX, FedRAMP, among others means few things will raise auditors’ eyebrows more than hearing that developers are performing deploys on their own code.

    That’s why it’s important to know how to manage these audits, and continuously improve the audit process.

    Rapid technological change, glacial control placement
    One of the first things to note is that these standards and management programs move slow. “If you are familiar with ISO 27001, which is a security management program, they just released a new version at the end of September [2013]. From 2005 to now, the standard hadn’t changed,” said James J. DeLuccia, a senior manager in the advisory services practice of Ernst & Young LLP in his Flowcon presentation, Successfully Establishing and Representing DevOps in an Audit. But technology certainly has changed in that time: increased use of virtualization, the rise of cloud computing, ubiquitous mobile, as well as changes in development and operational processes such as continuous deployment, continuous integration, and DevOps.

    One of the biggest issues that come up with disciplines such as DevOps and continuous deployment is separation, or segregation, of duties – that development, testing and operations should be separated duties. Not only do many government and industry mandates require it, so does many enterprise security policies. That’s why DevOps, when not explained or approached properly, can create bug-eyed expressions in unfamiliar auditors.

    The goal of separation of duties comes down, essentially, to proper change management. If the developer who wrote the code can deploy the code into production, and they are given too much access (such as root) or shared access, there’s a lost control check, and potentially no log of the event. But it’s a concern that is overstated, experts say.

    “In a proper DevOps environment, the person deploying the code doesn’t need to have root access. There’s no reason why, when properly setup, developers can’t logon, push the deploy button, and create an auditable log that says at 2:32 on Friday the 18th, David deployed a version of the code. And you have it down to the hundredth of second who did what,” says Mortman.

    As DeLuccia explained in his talk, when auditors are asking to see change control logs they are really trying to see that these controls are in place. “They’re really asking for help to understand this linkage,” he said. Depending on the policy and requirements, the auditor needs to see reports that demonstrate, in a statistically relevant way, that everything in IT is working the way it needs to and is said to be working. “Because remember, when we’re putting together these reports, we’re trying to say, everything is working the way you say it’s working,” he said.

    Auditing DevOps: An education process
    When done right, Mortman says DevOps and continuous deployments create more accurate logging and change management. “You have a log of every change that’s ever happened and it’s all generated automatically. You don’t have to worry about someone changing something. When setup properly, they don’t have the access to change anything other than the very limited roles of what that application they can deploy to,” Mortman says. “So in a lot of ways, in fact you’re actually maintaining better control over your environment.

    The challenge there, as DeLuccia explained, is that audit programs are built a year in advance. “When they come to see you, they’re already have it planned out based off last year’s risk. That’s already too late [to keep up with rapidly changing technology and processes]. You really want to plan this a year in advance,” he said. The goal, he explained is to become part of the process and show how the procedures they are asking the IT team to execute against are in line with reality of the processes and the technology on the ground.

    “Much like any other industry, you have an education problem,” says Mortman. “Security people run into this all the time. The business will say, “Hey, we want to do this new thing, and the security people reply, “Whoa, hold on a second. You can’t do that. That’s not secure. Or, we can’t audit that.” So a lot of the challenge with DevOps is an educational challenge. And you need to educate auditors that the control processes they are looking for are in place and that there are adequate logs that can be audited and evaluated,” says Mortman.

  • DevOps and faster feedback: fewer problems, better features (part 2)

    DevOps and faster feedback: fewer problems, better features (part 2)

    As the experts established in part one of this series , the tightening of feedback loops through DevOps gives developers a better chance to learn how well their coding choices perform in the context of production environments. This makes for higher quality code in the long-run and constantly improved skillsets among development teams.

    But fast feedback loops don’t just improve the line-by-line coding work the development team puts out. It can also push through a huge boost in the value of the features they’re producing for end users.

    “Feedback loops are important not just for technical feedback or feedback about things not working,” says Aater Suleman, CEO of Flux7 Labs and a professor in computer systems design and architecture at the University of Texas. “There’s also the feedback around what customers are looking form in terms of particular features and whether or not those features are doing well in the market or not.”

    The tighter the feedback loop is about feature sets—and the more quickly the team can release responses to that business-related feedback–the faster the pivot time to account for market conditions. The quicker assessment of how well features are doing gives development teams and the business units they answer to a more accurate barometric reading to decide where to direct programming time. As a result, it isn’t just the quality of work that goes up, but also the business value.

    “With a DevOps flow that allows developers to start seeing feedback from end customers right away, it helps them be more agile on a business level so they can be more strategic about where to spend their actual time,” Suleman says, explaining that this exposes developers to a big-picture view of their work. “You may have the most productive development environment in the world, but if a feature developers have worked on for two months is not going to be liked by the end users, it doesn’t matter how productive the developers were during that time period.”

    Not only that, but it eliminates the hamster-wheel of playing catch-up with competitors’ features.
    “I know of many  companies which still hold to quarterly release cycles. Did you competitor  just release an awesome new feature? Have fun waiting three months to catch  up,” says Curtis Poe, a consultant and Agile development coach for All Around The World. ” In today’s competitive world and fast-paced business environments, do you  really want to playing catch up all the time? Switch to continuous delivery and never worry about a lengthy  release process blocking you from offering what your competitors offer.”
    Poe says that the best DevOps teams can get the most differentiation from the crowd by understanding that they’re not just monitoring technology and code, they’re monitoring people. Namely, end users.

    “Do you sell shoes online and you have a  typical conversion rate of 3.2 percent? What happens if you make a release and the conversion rate drops to 2.5 percent?” he says. “Assuming that nothing technological is going wrong , monitoring business value, when appropriate, can instantly tell your designers,  ‘Hey, moving that button from the left to the right side impacted our sales.’”

    Additionally, that kind of business monitoring makes it more easy to understand the dependencies between technical and business functional aspects of software updates. Then the team can more easily explain to business leaders what technical obstacles stand in the way of functional improvements, and how to overcome them.

    “You always want to deliver the best experience for your customers, and you want to iterate on both the product and the non-functional aspects as often and quickly as possible,” says Arnold Goldberg, vice president of customer engineering for PayPal. “Integrating new features might be the wrong thing to do if, for example, there are challenges with performance, stability or availability.”