devopsy.kinsta.cloud

Tag: automation

  • DevOps by the Book in 2017: Where we Really Stand

    DevOps by the Book in 2017: Where we Really Stand

    Now that the year-end prediction season is over, it’s time to focus squarely on the actual here and now—where the DevOps movement actually stands at the start of 2017.

    As we’ll need to wait another six months or so for the annual State of DevOps Report to emerge from the good folks over at DORA and IT Revolution, which forces us to take stock of the situation using other resources. In this case, I’ll point to the recently released book, “DevOps for Digital Leaders,” authored by my colleague Aruna Ravichandran along with co-authors Kieran Taylor and Peter Waterhouse.

    Register today for the upcoming webcast, “Chart a Winning Course: DevOps for Digital Leaders,” and enter the chance to win a free signed copy of the book [U.S. only].

    Why should you value this book over any of the other DevOps-related titles inundating the technology literature scene, in particular those issued by (gasp!) technology vendors? I’d cite two very specific reasons: real-world metrics and use cases.

    I often preview any comments I make in cross-functional meetings with the phrase “shameless marketing.” That is because I work in marketing. For 12-plus years prior and sometimes in between marketing gigs I’ve served as a journalist, wherein my job was to attempt to be as objective as possible. Today, I work for a vendor that would really like to sell you stuff.

    That said, this current career path does not preclude me from being able to see beyond my own not-so-secret agenda. While many onlookers understandably groan and roll their eyes at the manner in which DevOps has been co-opted by marketers as a means to advance products, this doesn’t mean we’re all full of it.

    Like myself, working at CA—in particular, leading marketing efforts that span a huge array of strategic technologies actively being leveraged by global enterprises to attack various elements of DevOps—has given Aruna and her partners tremendous insight into where many people stand in terms of maturity.

    To that end, this book is written around some fantastic stories about how some of these organizations are refining best practices and measuring real success in domains including continuous testing, release automation and shift-left monitoring. IMHO, these are some of the hottest areas of DevOps being advanced as we speak.

    In addition to these use cases and related practices, the book takes an up-close look at critical aspects of the DevOps build, test, deploy and manage stages, along with key processes related to tuning and continuous improvement. And it even goes so far as to talk about essential DevOps tooling. I hope people are ready to acknowledge that beyond culture and leadership, automation absolutely plays a significant role in all of this.

    Specific to metrics, one of the areas where the State of DevOps peeps attempted to break new ground in the 2016 report, there’s a lot to appreciate. The book takes a close look at how these same organizations are attempting to tie DevOps-driven change directly to ROI, including the all-important customer experience.

    Worth noting, in the coming days we’ll also see the release of a new report from Coleman Parkes that charts numerous KPIs being used to this same end, so stay tuned.

    So there you have it, the marketer has officially hijacked this blog space to do little more than market a book about DevOps written by marketers. But, the truth is that’s not all.

    If you really want to hear the best parts of this story, sign up for the upcoming devopsy.kinsta.cloud webcast, “Chart a Winning Course: DevOps for Digital Leaders,” in which Aruna and Alan Shimel will discuss the high points, with an emphasis on fielding questions from the audience live Q&A style.

    If you don’t have time to read the book—or, better yet, you want a preview to ensure that what I’ve written here is true—post up and ask your own questions. We’ll also be conducting live polls about DevOps maturity, audience use cases and the leading types of DevOps metrics.

    It’s worth noting that all proceeds from the print edition will go to Foundation for Excellence, which provides STEM scholarships to children in India. Order your own copy today (or download it for free).

    — Matt Hines

  • Automation: Achieving Faster Rollouts with Better Accuracy

    Automation: Achieving Faster Rollouts with Better Accuracy

    A changing landscape requires new processes

    At the recent Symposium/ITxpo, Gartner shared a list of technologies and trends that will be important for organizations to embrace and prepare for in 2017. These trends, which are listed in the report, “Top Strategic Predictions for 2017 and Beyond: Surviving the Storm Winds of Digital Disruption,” reinforce the breadth of technology that IT must support, roll out and engage with. This includes everything from the 100 million consumers who will shop using augmented reality to dealing with the roughly 21 billion endpoints expected to be in use in 2020.

    The report also demonstrates how fast technology changes directions. For example, Gartner predicts that in 2019, 20 percent of brands are expected to abandon their mobile apps. “Many brands are finding that the level of adoption, customer engagement and return on investment (ROI) delivered by their mobile applications are significantly less than the expectations that underpinned their app investment,” the report notes. However, this doesn’t mean less work. IT departments continuously have shift gears to support new processes, news apps and new technologies, such as the Internet of Things (IoT) and the massive amount of data that it produces.

    The dynamic needs of today’s businesses are driving organizations to increasingly adopt new technologies and applications that oftentimes don’t play nice with the existing set of infrastructure and legacy systems in place. So, while it’s clear that the technology landscape is changing, what isn’t so obvious is how IT and business decision makers should be adapting to an industry that is in constant motion.

    In-Steps DevOps

    To keep up, IT must implement agile methodologies that can adapt quickly to market changes. The DevOps mentality is making its way into many organizations and involves the development team creating, building and evolving products to ensure the organization is up to date, while the operations team confirms everything is running smoothly prior to a launch going live. This level of checks and balances isn’t just a nice to have, it’s a necessity—one mistake and the entire system can be brought down. Fostering a greater level of collaboration between development and operations makes it possible to efficiently and effectively build, test and correct new features and new products.

    While the shift to a more collaborative environment between the Dev and Ops teams can take time, using modern IT automation solutions can make this transition easier. Such solutions automate many of the routine—and even not so routine—elements of the application release process within DevOps. Many of the actions necessary to move from code to production—not to mention maintenance, testing and cleanup tasks—can be taken off the backs of developers and operations, saving them time so they can focus on higher-level projects and reduce manual errors.

    Finding the Balance Between Speed and Accuracy

    In 2012, Facebook announced it was increasing the speed at which it released code to twice a day, every day. As Chuck Rossi, director of Release Engineering at Facebook said, his team follows the advice “ship early and ship often.” Facebook continues to follow this model, and while the company certainly didn’t invent it, the model has caught on and is spreading among companies developing virtually every type of technology. The catch? It’s an incredibly time-intensive process and not every company has the resources necessary for a dedicated release team. Plus, a true shift to DevOps doesn’t happen overnight, especially since there is a ramp-up time for folks to get on board. For example, Phil Dibowitz, systems engineer at Facebook, told TechTarget that it “led its Chef team through a three-year process to rethink configuration management to scale on par with the web-scale company’s data centers.”  This took three years, he said, “not because of the technological change, but because of the structural change.”

    A slow transition to a new technology or new way of operating is a common occurrence because historically the production systems are cordoned off from development groups. While the Ops team has to ensure the rollout will meet certain standards, the Dev team wants to speed the process to begin the final rollout process as soon as possible. A happy medium exists, but if it isn’t found, things can go very wrong. Unfortunately, 2015 gave us two major examples of what can go wrong if products can’t speak to each other and rollouts are completed too early: Remember the New York Stock Exchange and United debacles?

    Breaking Constraints Through Automation

    This is where modern workload automation comes into play. Automation and solid change management allows IT to flexibly add on new technologies with automated processes for building, testing, upgrading and configuring IT—all while reducing the slack time between various steps. For instance, a workload automation solution should be able to trigger a build, deploy the build to the test environment, run the test suite and, assuming testing meets acceptable criteria, promote the build to production. If something does go wrong, necessary remediation can occur, and through monitoring and alerting capabilities, the required parties can be notified proactively.

    Additionally, automation solutions with pre-built integrations make it easier for disparate applications to communicate, creating an end-to-end approach to workflow creation and maintenance. As a result, IT organizations can minimize the risk of business disruptions and achieve a more fluid technology delivery to the business.

    Although it’s not a silver bullet that will solve all DevOps transition issues, automation does reduce manual intervention, which leads to fewer delays and errors and improves speed at which tasks are completed. However, beyond the technological aspects, there must be a clear understanding by both sides (the Dev and Ops group) on what their counterparts need—and this requires a high level of trust and coordination.

    While the specifics around the DevOps methodology certainly will change, automation will remain a critical piece of the process. With IT automaton, DevOps teams are able to refocus on critical IT business and operational processes, thereby reducing delays, decreasing errors and improving the speed at which tasks are completed. Ultimately, automation makes it possible to increase the speed of product releases and enable companies to stay ahead in the age of unpredictable customer preferences and ever-changing technological developments.

    About the Author / Mehul Amin

    Mehul Amin, a 21-year veteran of Advanced Systems Concepts Inc. (ASCI), previously held roles as product engineer and technical support manager and played an integral part in building the company’s workload automation solution, ActiveBatch. In his current role as director of Engineering, Mehul drives high-profile initiatives and leads both the Product Engineering and Release Engineering teams. Mehul received a master’s in computer science from the Stevens Institute of Technology.

  • Rethinking Unemployment in the Age of Automation

    Rethinking Unemployment in the Age of Automation

    In my last article concerning the impact of automation and robotics on human employment, I asserted that commercial job loss is more a result of automation than outsourcing. Also, I asserted that this trend was going to continue at an accelerated rate. I said we need to address three areas of concern to mitigate the impact:

    • Income
    • Birthrate
    • The meaning of work

    In this installment, I am going to look at income—specifically, the need to rethink how we approach unemployment insurance.

    Let me share a piece of personal information with you. A while back I lost a gig and was eligible for unemployment benefits. The amount I was eligible to receive back in 2012 was $300 a week. (Today, the maximum benefit is $450 a week for up to 26 weeks.) In the scheme of things that’s not much money. In a major metropolitan area such as Los Angeles, $1,200 a month does not even cover my housing costs. Luckily, I was able to get another gig in no time at all. But, I am a tech worker. Others are not so lucky.

    Unemployment benefit in California is calculated by using the following formula:

    highest quarterly earnings/25

    Thus, the average programmer makes around $75,000 a year, which comes out to $18,750 a quarter. So doing the math:

    $18,750/25 = $750 a week

    But that $750 number exceeds the maximum allowance. So, if you are a programmer who has been “let go” in California, the most you’ll get is $450. Also, any benefit you receive is subject to income tax. If you find a part-time job to make up the difference, your benefit will be reduced.

    The calculation is not special to California. Arkansas is pretty much the same. In Texas you get a little more.

    If you are accustomed to making $75,000, which after taxes is about $1,100 a week, making ends meet on $450 pre-tax dollars is going to be a stretch.

    This is not a pretty picture in a few ways. First, there is the built-in incentive to not work legally. You are not encouraged to use your creativity to launch your own business. Forfeiting earned money does not encourage you to take on small jobs that allow a company to get to know you. (Try and buy has led to many a full-time position.) So what do you do? You go into the shadow economy. You work under the table: Have the check made out to your wife on her W9 or have it show up on Paypal as a gift. Is the practice shady? Yes. Is it necessary? Yep, particularly as the end of the month rolls around.

    Now, if you are in a major city such as New York, San Francisco or Los Angeles and you are in prime employment age, between 25 and 45 years old, getting another tech job quickly is usually not a problem, provided you know what you are doing. But, what do you do if you are a 50-year-old Visual Basic programmer in Glen Falls, New York, with two kids still in high school and a housing market that is anemic? You just can’t get up and move. And, learning programing technology is going to take time.  Now, imagine the same scenario, only this time you work in a small furniture factory in Alabama and your job lathing table legs has just been replaced by a robotic CNC machine. The picture is now a nightmare.

    So, what do we do?

    I am a supporter of the notion of Universal Basic Income (UBI). UBI is a system in which citizens of a given nation are provided with enough basic income to meet the essentials of life. Typical in just about all the plans for UBI is the stipulation that there are no strings attached. Under UBI, if you want make more money you can. Your UBI allowance is not decreased. UBI can be more about developing your value in society rather than living off the dole.

    Still, I am a realist. I am not of the belief that one day a magic wand will be waved over state and the national capitals, that politicians will come to their senses and amazingly we will have UBI. Big change does not work that way unless something gets blown up (Think Pearl Harbor and 9/11). I am of the thinking that when it comes to large-scale change, the best thing to do is focus on one thing. For me, that one thing is to eliminate the “payback” clause in unemployment benefit policy.

    Given that many, if not most, jobs will be automated before the end of the century, the notion that unemployment is a benefit that tides us over until the next job comes along is outdated. There might not be any next job. Rather, unemployment might be a time to reinvent ourselves. Reinvention means using the unemployment benefit as a baseline of income upon which we can build more income. We should not be penalized for making money on top of the unemployment benefit. Rather, the benefit needs to be a subsidy from which greater income grows. Once a livable level of income is achieved and a trajectory of reliable, personal revenue is established, we can revisit the size and limit of the benefit allowance. Until that time, the allowance remains in place.

    Think of eliminating the “payback” clause as training wheels for full UBI. Removing the ““payback” clause in unemployment benefit policy is a concrete first step to having realistic policies about labor and income in the Age of Full Automation. The legislative debate and public education that will happen in the course of eliminating the “payback” clause of unemployment insurance will put Universal Basic Income on the social and political agenda.

    UBI is not the stuff of fantasy. It is already has an experimental implementation is Finland. The National Digital Council of France recommends testing UBI. UBI exists in a modest way in the United States in the form of the Alaska Permanent Fund.

    As I mentioned in the previous article, using automation to replace human labor is nothing new. But, what is new is the rate at which the human labor is being replaced. If we do not start implementing concrete ways to address the long-term ramifications that result from the ongoing elimination of human labor from the commercial landscape, the storming of the Bastille in 1789, the bread riots of 1863 in the southern United States and the removal of the Bonus Army from Washington, D.C., in 1932 will look like a warmup act for what might very well transpire. Removing the “payback” clause from current unemployment benefit policy is but a small step toward addressing the issue at hand. However, put a lot of small, achievable steps together and after a while you’ve made significant progress solving a big problem.

    So here is something we can do.

    Take 15 minutes out of your day to copy the following and send it to your local and state representatives as well as your representative in Congress:

    Dear Representative _____________
    Please initiate legislation immediately to remove the practice of deducting money earned while receiving unemployment benefits from a recipient’s payment. Also, make it so that unemployment benefits are no longer taxable income.

    As automated labor continues to do more jobs formerly done by humans, unemployment insurance benefits need to be adjusted in favor of empowering unemployed workers to participate in the modern economy, not penalizing them for using any and all available resources to improve their condition.

    Thank you,

    YOUR NAME HERE

    The problem of automation replacing human employment is real and it’s not going away. The 15 minutes you spend will be well worth the time invested.

    — Bob Reselman

  • DevOps Chat: Huw Price, Smart Automation Is Key

    DevOps Chat: Huw Price, Smart Automation Is Key

    Before the new year, I had a chance to sit down and chat with Huw Price, Vice President, Application Delivery, CA Technologies. Given Huw’s title what do you think we spoke about? You guessed it, automation. Actually there is more to it than that. Automation for automation’s sake alone is not necessarily a good thing. As Huw points out, smart automation is key.

    Huw and I will be participating in a full webinar on the topic of Smart Automation and how to build it on January 24th. You can register for this free webinar here.

    As usual with our DevOps Chats, below is a streaming audio player with our interview and below that is a transcript of our conversation. Enjoy!

    Alan Shimel:                Hi, everyone, it’s Alan Shimel, Editor in Chief at devopsy.kinsta.cloud here for another DevOps chat. Today’s guest on DevOps Chat is Huw Price of CA Technologies. Huw, welcome to DevOps Chat.

    Huw Price:                  Hi, Alan, nice to be here.

    Alan Shimel:                Nice to have you here, Huw. Huw, if you wouldn’t mind, if you could give our audience a little bit – I said you worked at CA, but what’s your title and what are you responsible for over at CA?

    Huw Price:                  Yeah, I’m a VP at CA. I was the managing director of Grid Tools, which venture you may have heard of, where we specialized in all things test data, and laterally we moved into all things test case optimization and into test automation. My role at CA – I’m kind of an evangelist QA specialist practitioner, and it’s actually fitting quite well with the DevOps land of trying to improve quality whilst improving speed at the same time, which is a bit of a magic trick. But it can be done and it needs some new thinking.

    I would say that over the years, it used to be cost was the top driver for businesses, but now due to the pressure of business, it’s all about competition and it’s all about getting stuff to market quickly, and you can’t have poor quality code going out to customers because they’ll just move to someone else. And I think that’s really what’s kind of driving a lot of our thinking at CA at the moment.

    Alan Shimel:                Absolutely, and it is somewhat counterintuitive when they tell you you could go faster but have higher quality. And I’ll even throw one more improbably into it – and be more secure by doing that.

    Huw Price:                  Yeah, that’s an interesting area. I think actually what I’m finding now is that I’m a big fan of doing everything in parallel, the days of sequentially waiting for one team to hand stuff off to the other, and what we’re finding is things like performance testing and security testing are kind of job one, and you need to build those in almost as functional requirements as you’re designing your software really. That sort of hopping from one piece of functionality or requirement to another seems to have vanished. It’s all hands to the pump I think at the moment is what I’m finding out there.

    Alan Shimel:                It is a bit of a Chinese fire drill, as they say. But out of chaos comes order, and that’s kind of the good news. Working with you and your team we’re actually doing a webinar after the New Year, in January, and it’s titled “How To Build the Right Automation.” And you and I were talking a little bit offline, of course, and the idea that you just automate for automation’s sake, maybe that was a couple years ago, but now we’re trying to be smarter about designing what’s to be automated and how we do it. Can you share with our audience a little bit of maybe your insight into that?

    Huw Price:                  Yeah, I think so. Automation’s got a bit of a bad press, and to be honest, I think rightly so, it’s considered very fragile. You only need a small change, a pop-up appearing on a screen, some kind of slight delay in a network and some of the automation breaks. And then everyone looks for the results and you spend days and days actually finding out that the program is fine or the functionality is fine, but it was actually the automation script that was wrong. That’s the first problem.

    And the other one is the time it takes to build the automation in that it’s not uncommon for automation to be built three months after a sprint is finished, which of course means you’re automating something that doesn’t exist any more, which is not very continuous. And the value of what you’re actually doing is pretty poor. I think that is kind of the backdrop of where people have been.

    They’ve moved towards test-driven development, behavior-driven development etcetera, but they still have the problem that actually maintaining the automation scripts, even if they’re key word or Gherkin, it’s still an issue. Because, you know, requirements don’t exist any more, just change requests exist, and a change request means you’ve got to go back through all of your existing automation and say, “Do I have to change all these automation scripts, which ones do I change, which ones are duplicates, which ones make no sense?” And that is a massive amount of work.

    I’ll just give one example. There was a bank in Canada, and basically for every single requirement change, they would go through every automation script by hand to see if it needed to be changed. So one change request took one tester all day to see if it impacted any existing automation scripts. So that’s just one change request. So how many testers do you have, how many automation guys do you need, how many change requests can you fit through that very narrow pipe. I think that’s where people, even with some of the new techniques – and I’ve got to say that the automation frameworks are becoming better. They’re becoming more structured; they’re becoming more malleable key word based. But they still don’t really address the problem of if I make a change to a requirement how quickly can I update my automation framework.

    Alan Shimel:                Excellent.  Huw, I want to talk to you a little bit about the go to market with your group. You’re obviously probably dealing with mostly large enterprises, how does a large enterprise realize they have this problem? Is the problem one where, “Geez, we need help automating,” and they haven’t even cut across the threshold of “we need to be smarter about automation” because they’re still at just the “we need automation.” Or is it organizations who have already, let’s say, dabbled with automation and are now realizing, “Wait a second, this is just a giant cluster waste of time.”

    Huw Price:                  Yeah.

    Alan Shimel:                “And we need help being smarter about automation.”

    Huw Price:                  I think the answer is yes to all of them. Even within the same organization – and I can think of one we’re dealing with at the moment – you get such a sort of disparate set of skill sets, capabilities, maturity models, and quite often we’ll go into a large company and introduce teams to each other and say, “Do you realize you have a really nice sort of key word driven framework sitting over here with a very clear object library, which you’ve got all this work and you’re going off and building yet another selenium framework, maybe you guys should talk to each other.” I just find that we do a lot of sort of introducing companies to other people in the same company.

    Alan Shimel:                Sort of like matchmaking, huh? Matchmaking all that stuff. Interesting.

    Huw Price:                  I’d like to add on that, certainly what I do and members of my team and other people in the business are doing, we are genuinely hunting for best practice here. This is not about us selling software any more, the market is moving so quickly and this ability to be a sort of lightening rod of best practice amongst other companies, genuinely it’s quite fun and we’re enjoying it, and we do share around what we’re finding out in the market in terms of what’s good, what people are struggling with, and hopefully we’re bringing not just software to the market, it’s more some best practice thinking as well, and I think that’s quite interesting.

    I’ve got to say that chatting with you when we met in Vegas, I learned a couple of things from you. And I find that when I talk to pretty much everyone. And I’m trying to feed that back, whether that’s by webinars or with the direction of how we build the software. We’re trying to just keep pushing the boundaries, keep communicating, keep thinking. It’s kind of fun. We’re really enjoying it at the moment.

    Alan Shimel:                Absolutely. I always tell people these are interesting times to be in the IT world, there’s just so much change happening – and change I think in a good way. New tools and trends and ways of doing things, it is fun, it really is fun.

    Huw Price:                  I’m going to take a slight tangent here. I was listening to an agile coach talking and I agreed with 80 percent of what he said, and disagreed with 20 percent. But then I challenged him and said, “Can you tell quickly whether an agile team is good or bad?” And he said, “If they’re having fun, that’s generally good.” And I know everyone is into hero culture at the moment, charging around trying to get the release out the door at the last minute. That’s what we’re trying to focus on is move away from the hero culture and actually getting that fun back into – you know – solving challenging problems and automating as much as possible and joining stuff up and getting feedback loops going and all that kind of fun things. So it is, as you say, very interesting times.

    Alan Shimel:                Sure. Steve, let me ask you a – and I’m not being facetious but a real question – can we have too much automation?

    Huw Price:                  Actually, that’s interesting. I did a presentation at Star West and I talked about risk based testing. It’s actually kind of touched on an interesting subject here. The reality is, if you have a very stable system and you’ve got a bunch of very experienced programmers, you know, you run it through a code quality engine and it’s very solid and stable, and they also understand all of the data that flows in and out of it – the reality is that actually you might need a very minimal amount of automation to test that.

    If you’ve got something which is highly, highly active – I’ll mention an Indian company, they’ve got one API which brings in 150,000 new customers a day. That API, I mean, that is staggering. 150,000. This is to do with the 4G rollout across India. That API, you know, I would be automating the hell out of that. I would have so many automation scripts around that. I would have docker images spun up in every different direction, I would have visualization wrapped around it, I would have negative testing up the wazoo, so that if you make a change – performance testing, blaze meet up, whatever it takes, to make sure that that baby, when you make the slightest change, you don’t mess with it — because if it’s down for ten minutes, that’s 5,000 customers. So I think people maybe should start thinking a little more sort of holistically and higher up in terms of the different levels of risk associated with it. I think that’s one thing.

    So I think that’s one thing, and then the other one is the speed at which you can automate. The reality is that what we do, we will generate – we’re very much into building a model of what the system should do. You could potentially build all your automation scripts before you’ve actually built the application. I was doing some work with Eggplant the other day and we actually generated a whole bunch of Eggplant scripts based off a Microsoft PowerPoint of what the system should do. So in other words, before the developers started, we’ve actually created the automation framework.

    Now, if that takes 15 minutes, great. If it took 3 days, then it’s a little bit of a tradeoff. So I think this balancing things up based on risk, and also the technology that you have to hand, is probably the two criteria you should start thinking about.

    Alan Shimel:                Agreed, agreed. Steve, as I had told you offline, the time here goes incredibly quick and we’re coming up on our 15-minute outside limit. I just wanted to bring you back home to the webinar in wrapping up. I believe our webinar is scheduled for in the latter half of January, and it is called “How To Build the Right Automation.” For people who are listening right now, give me two or three points why they really should make it their business to attend the webinar.

    Huw Price:                  I think firstly they’ve got to understand what they’re automating. If you look at most automation scripts, you’re probably only covering about five percent of the functionality. We would look to automatically generate 85 to 90 percent of the functionality of the system in either the same time or less time, so I think that’s firstly. To really understand coverage, I think that’s important.

    The other one then is to think about that actually automation is not just about the scripts themselves, it’s about getting the right test data and the right – I’m going to use the phrase visualization, whether that’s service visualization or message visualization or sort of network visualization or visualization environments. So trying to create an environment such that your automation is actually testing the bit that you really want to test. If you think of an end to end test, to get to something like a share trade close where you might have 200 functional logic gates I need to test, is actually really difficult, because you’ve got to have all of the right bits around your automation to make sure that you’re testing the right stuff.

    So better automation in terms of the design and also the assets around the automation to make sure that you’re actually testing the correct stuff. I think that’s probably what I’m going to focus on during the webinar.

    Alan Shimel:                Got it, got it. Well, obviously on devopsy.kinsta.cloud we’ll be publishing links to the registration page for people who want to join us on that webinar. It sounds like it could be – it sounds like it could be a great one if folks are interested.

    Huw Price:                  And it’s lots of demos. I’ll do a couple of slides, but I prefer to do demos.

    Alan Shimel:                Oh, that’s greatest.

    Alan Shimel:                Yeah, most techies love to see stuff.

    Huw Price:                  Actually, we’ll drive right in here.

    Alan Shimel:                They’ll give you a slide, but anything beyond that they’re just like, “Show me stuff, show me stuff.”

    Huw Price:                  All right, so let’s stay tuned to devopsy.kinsta.cloud. I would imagine we’ll probably have the landing page up within the next week or two as we head into the holidays.

    Alan Shimel:                Well, Huw price, I’m going to ask you one last question – not a curveball or anything, but I usually ask a lot of our guests this – for our audience listening, one book that you would highly, highly recommend they read.

    Huw Price:                  Actually there’s a book called Model Based Testing. What we’re discovering is that people are now moving pretty much – they’re realizing that without a model it’s very difficult to proceed. And there’s a book called, it’s actually an adjunct book to the Model Based Testing Course. We’re massing this out to our customers now to kind of reset their thinking. Everything we do in life is a model, everyone’s making decision gates, logic gates, risk based profiles, and this is going down really well. People are really taking the time to read it. What I’ll do is during the webinar I’ll put up the pages so people can go and get it. I can’t remember the exact title. It’s called Model Based Testing – The Course Guide, or something like that. But I highly recommend that.

    Alan Shimel:                Excellent. Huw Price, thank you so much for being our guest on this episode of DevOps Chats.

    Huw Price:                  Oh, it’s a pleasure.

    Alan Shimel:                Thank you, it was our pleasure and we look forward to the upcoming webinar, “How to Build the Right Automation,” including demos, and we’ll have more information on that. And we hope to have you back on DevOps Chat soon, Huw.

    Huw Price:                  Great, all the best. Thanks so much.

    Alan Shimel:                All right. This is Alan Shimel for devopsy.kinsta.cloud and DevOps Chat. Thanks very much, everyone, and we’ll see you on the next DevOps Chats.

    — Alan Shimel

  • A Holiday Tradition: 2017 Continuous Delivery Predictions

    A Holiday Tradition: 2017 Continuous Delivery Predictions

    It’s that time of year. As the calendar year comes to a close, many of us are following time-honored traditions: making preparations for the holidays, planning some much-deserved time off and enjoying time with friends and family.

    In the business world, the year-end also brings its own traditions, and first among those is making predictions for the coming year. I’m not one to forgo traditions, so in this article I’ll share my thoughts on the upcoming year within the context of the continuous delivery movement.

    To level set, continuous delivery is the most evolved and modern state of software development and testing—and is core to a successful DevOps strategy. Organizations that are able to achieve continuous delivery can rapidly develop, test and release applications—with higher quality and at lower costs.

    Simply put, the ability to more continuously deliver software is one of the biggest advantages to attain in the application economy, where the battleground for consumer loyalty is most often found in the ability to deliver superior applications.

    Now that we know what continuous delivery means, let’s get to my predictions for 2017:

    1. Continuous Delivery Means Continuous Everything

    Continuous delivery is still a relatively new concept, and one that many organizations strive to achieve. As a new concept, it suffers from some of the misconceptions that often plague emerging technologies or trends. Many technology providers latch on to the term and claim to provide “continuous delivery solutions” that in reality only address a small part of the continuous delivery challenge.

    Truly delivering software continuously means that all the tasks within the software development life cycle (SDLC) must be optimized to occur in parallel and continuously. This means development must be continuous, testing must be continuous, and likewise for release and operations.

    2017 will be the year that industry embraces the importance of an end-to-end continuous delivery value stream. In 2016, many organizations began their continuous delivery journey by addressing certain aspects of the SDLC; in 2017 many organizations will begin to realize a truly open and integrated continuous delivery workflow.

    1. Democratization of Continuous Delivery Tools (via the Cloud)

    Optimizing all aspects of the SDLC for continuous delivery, as mentioned above, means that related tools need to be easier to use and easier to access. Developers who have access to testing tools (left-shift) can improve the quality of code they produce. Similarly, QA engineers who have access to design requirements sooner can create more accurate test data and scenarios, earlier.

    However, for this to occur, the tools have to be both easy to access and simple to use. Developers have avoided complex testing frameworks, but with simple, easy-to-use tools available—often in SaaS forms—developers can certainly incorporate testing into their methodologies.

    In 2016, CA acquired BlazeMeter, which pioneered the notion of cloud-based performance testing employing a simple and easy-to-use (yet very powerful) product. We expect to see this trend expand into other tools within the continuous delivery value chain in 2017.

    1. The Industry Embraces Open Source (Really)

    In the past, we’ve noted that commercially motivated technology providers have tried to keep open-source technology “at arm’s length.” There may have been some lip service paid to openness, but in reality many commercial providers felt threatened by open-source solutions that were seen as competitive to their products.

    This perception has already started to change, but I believe 2017 will mark the tipping point where companies will truly embrace open-source solutions as part of their strategy; those companies that continuously resist (pun intended) will be at a significant disadvantage.

    Open source is important for many of the same reasons as the cloud: It is broadly adopted because it creates advantage in time to market, quality and cost effectiveness. Embracing open source—and by that I mean both integrating with open-source solutions and using them as part of a product strategy—only makes good business sense. It helps software development teams meet their objectives in delivering quality software faster, and at lower cost.

    Another year is rapidly coming to a close and we have a lot to look forward to in 2017.

    On the topic of continuous delivery, looking forward means spreading continuous delivery throughout the SDLC, using the cloud more and embracing open source.

    All of those things combined will help us advance the continuous delivery movement together as an industry. Here’s to another year of continuous delivery advancements!

    Don’t forget, you can also take a closer look at the ongoing evolution of continuous delivery and other DevOps-related trends by picking up a copy of my colleagues’ new book, “DevOps for Digital Leaders: Reignite Business with a Modern DevOps-Enabled Software Factory.” If you have some extra time over the holidays, you’ll find it to be an interesting deep-dive.

    — Jeff Scheaffer

  • What Do We Do When Everything is Automated?

    What Do We Do When Everything is Automated?

    Recently, I came across a study the caused me to think beyond the day-to-day world of code-slinging in which I live. According to the study conducted by Center for Business and Economic Research at Ball State University in Indiana:

    “Almost 88 percent of job losses in manufacturing in recent years can be attributable to productivity growth, and the long-term changes to manufacturing employment are mostly linked to the productivity of American factories.”

    Just translate the term “productivity of American factories” into the word “automation” and you get the picture. Other workers are not taking jobs away from the gainfully employed, machines are.

    This is not a new trend. It’s been going on since before Eli Whitney invented the cotton gin. Industry creates machines that do the work of humans faster, cheaper, with more accuracy and with less failure. That’s the nature of industry; nothing new here. However, what is new is rate by which the displacement of human beings from the workforce in happening.

    The Bureau of Labor statistics projects that by 2024 the U.S. economy will need 50 percent less car mechanics, 42 percent less telephone operators, 26 percent less postal clerks and 20 percent less parking enforcement officers. Or, to put it another way, cars will be better at self-repair, call routing technology will get better at conducting verbal conversations with humans, self-service postal machines will get better at taking care of human transactions in the post office and parking meters will know how to write tickets.

    Here’s another another point to ponder. As the quality and reliability of self-driving vehicles improve, it’s not far-fetched to imagine self-driving trucks becoming the norm on the nation’s highways. How many truck drivers will be displaced once self-driving trucks are everywhere?

    Also, think about this: As Uber and Lyft mature in the business landscape and self-driving cars become commonplace, the ride-share companies will replace human driven vehicles with self-driving vehicles. Then, Uber or Lyft will start to offer a pricing model that is flat fee for 24/7 service with unlimited mileage. So, to keep it personal, imagine Uber/Lyft offering me a deal in which I pay $150 a month to have all my driving needs taken care of. Or, I can continue to pay the $400 it costs me for my car lease and insurance coverage (yes, I lease a modest vehicle). Which way do you think I will go? Spend $400 for the pain of driving in a car in LA traffic or spend $150 to get where I need to go while I read a book in the back seat? Me? I’d fork the car over in a New York minute.

    As if this isn’t startling enough, try this one out: Walk down your street. See all those cars parked on the side of the road. In the near future, using the Uber/Lyft automation scenario, they won’t be there any longer. Any car made will be used 24/7. All this translates into less cars doing more work. And less demand for cars means less cars being made. Less cars being made means less people working making cars. You get the picture.

    What about software? Let’s take a scenario that is part of my day-to-day coding world, logging. In the old days, you’d log to a file on disk. Over time, that log file got so large that a human would need to move the file manually onto an archive disk so as to not eat production space. Somebody comes along and says, “Hey, I can write a piece of code that watches the size of the log file and moves it automatically when the file gets too big.” So much for the person doing the manual file move. But, still, you had to keep adding capacity to the archive machine, which typically meant a human walking down to the server room and twiddling with some hardware. Then a someone has a great idea, “Let’s use this thing called cloud storage as a back-end for a log collection service. We’ll leverage the elastic capacity features of cloud storage and write out logs to that service. Hell, why make a service? We’ll subscribe to an existing one and save the development costs.” Thus, no more humans needed to monitor capacity, move files, write incidental scripts—any of it. In terms of logging, a bunch of automation is doing the work a lot of humans used to do. Is this progress? Might be. But the trend is unmistakable, so much so that I have no problem imagining a future in which 90 percent of work that used to be done by humans is done by low-cost, automated, robotic, labor.

    We’ve always been on our way to automating everything. The only difference between then and now is that now we are really, really close to full automation. Asking the question, “What will we do when everything is automated?” is no longer fantastic, it’s essential.

    So what will we do?

    “Simple,” you say. “We’ll retrain the displaced workers.” Really? Many of us in the software game know that notion of retraining is a joke. If you are not constantly learning the new stuff, you are out of work. For a knowledge worker, continuous learning is an ongoing requirement for commercial viability. And, the baseline from which you are learning is very high. In other words, you need to know a lot to get the basics of how to write a deployment script in Ansible that runs against AWS. And, even as you are learning that basic technique, somebody is already creating a technology that will make the technique you are learning obsolete.

    So back to the question, what do we do?

    First, let’s do the obvious. When near total automation has the result that there is no longer any work for most humans to do, we need to come up with other means of survival. Survival in the modern world means having money to live on. Charles Murray suggests in the Wall Street Journal that we plan to implement Universal Basic Income. Finland is starting a pilot project based on the idea. Other countries are experimenting too. Universal Basic Income is a program in which citizens of a state are provided with a base income on which to live. Some programs vary basic income according to income produced otherwise. The common thread is an understanding that jobs are not plentiful and that depending on one for survival is not realistic.

    The notion of Universal Basic Income is not that new. It’s been around since the 1960’s, proposed by folks such as Milton Friedman as the negative income tax.

    Second, we need to address the elephant in the living room: the birth rate. In the old days when agriculture and small, private farming was a major industry in the United States, having a large family made sense. You needed your family members to work the land. As the Industrial Revolution created demand for factory workers regardless of age, large families still made economic sense. But now the trend is that we need less people to do more. Thus, having large families becomes counterproductive to the economy overall. Yes, there are certain sensibilities out there that encourage large families and adherents to such thinking have large families as a matter of choice. Still, many families end up with large families for reasons other than intention. It just sorta happens.

    Thus, to ensure that humans have a chance at a viable existence in a fully automated world, implementing effective family planning policy and practice becomes as useful to the society of the near future as the anti-smoking efforts that started in the 1960s does today. I am not advocating draconian, government mandated family planning. We cannot legislate family size anymore than we could smoking behavior. Rather, we need to create a social condition in which the wisdom of the idea prevails.

    Finally, we need to address the need for human dignity in a fully automated world. For better or worse, we need to accept that in the near future the traditional relationships between demand, productivity and income no longer may be applicable. The old days of human workers making stuff and getting paid for their product or labor are coming to an end. Yet, one of the intangible benefits of the old days was the dignity and satisfaction that went with pulling your weight and earning your keep. Being able to provide for yourself and your family defined your worth in your own eyes as well as those around you. This need for dignity will remain no matter how the technical and economic landscapes evolve.

    When I started to doing the research for this piece, I asked others in the know to respond to the prospect of a world that is fully automated. One the people I talked to was Phillipe Kahn, founder of Borland, Starfish Software, LightSurf Technologies and most recently, Fullpower Technologies. Here is his response:

    “I remember being in India and watching a construction site: Lots of people, elephants, activity. I asked, ‘Why don’t you all use bulldozers and tractors and prefabricated concrete parts?’ The answer was enlightening: ‘How would this community make a living with dignity?’ That’s the whole paradox that we are dealing with: Productivity versus dignity for communities that could be unemployed for generations.”

    There is a difference between work and labor. Work is an activity a human does to give one’s life meaning. Labor is what a human does to survive. In the old days work was your avocation, labor was your vocation. Some humans were fortunate enough to have their activities be both avocation and vocation. Others went to work in factories, farms and offices to earn a buck. They pursued other interests at other times. But whether you sang in the New York City Opera full time or in the choir on Sunday morning, the dignity of your work, bringing music to the ears of other humans, was in full force.

    So, the final part of answering the question, “What do we do when we’ve automated everything?” is this: Preserve the dignity of the person and the community. A fully automated world will be a wealthy world unless we blow it up. Productivity produces wealth. That’s the way it is. Regardless of who is or what is the owner of the wealth—government, corporation or private citizen—sharing will become more important than ever. Concretely, this means allocating a significant amount of money to people and organizations that do work that has little or no commercial value, but extraordinary amounts of human value. When it comes to commercial value, the machines will have that end of the economy covered and then some. However, stuff that has little commercial value—curing low occurrence diseases, space exploration, academic research, expanding the arts—this is where human activity can excel.

    In the old days we had a notion called the commonwealth, that shared space where our individual assets grew. We shared some to get some. In the automated future it might have another name. But no matter the name, it will be a place where humans work with dignity in communities and most labor is the left to automated robotics. If not, the common thread of our individual survival might indeed be nothing more than alms for the poor while automation creates wealth for a few.

    — Bob Reselman

  • Assembling the Key Components of Continuous Delivery

    Assembling the Key Components of Continuous Delivery

    As agile methodology was popularized at the start of last decade, engineering teams gained the opportunity to begin producing software in shorter cycles, while releasing it reliably at more frequent intervals.

    This concept garnered greater attention in 2009, when Tim Fitz published his thoughts on continuous deployment. By the time Jez Humble authored his seminal book, “Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation,” in 2010, the approach started to get significant mindshare. In the book, Humble advanced the three building blocks of continuous delivery: configuration management, continuous integration and continuous testing.

    Thereafter, Agile Operations—the practice of using end-to-end monitoring, fueled by analytics, to increase operational agility—appeared. This complemented continuous delivery, with the goal of increasing the quality and velocity of software delivery while minimizing overhead.

    Since then, and particularly over the last six years as DevOps adoption has exploded, there’s been significant discussion (and confusion) over the opportunity to embrace “continuous everything” across the software development life cycle (SDLC) to deliver functionality continuously and optimize end-user experience.

    Today, having spoken to hundreds of organizations employing agile and DevOps, it’s possible to expand on these concepts to better define the key components of continuous delivery.

    In this blog, I’ll detail the precepts of “CD” that CA’s customers recognize as critical in reducing the time, risk and expense of software delivery by frequently releasing smaller batches of code with extensive visibility and traceability.

    These building blocks of continuous delivery are:

    • Continuous development & integration,
    • Continuous testing. and
    • Continuous release.

    Continuous Development and Integration

    When it comes to delivering quality apps faster, every second counts.

    That’s why it’s important to automate the planning, development, unit testing and integration of code with different inter-dependent components, all while packaging it for quality assurance. We categorize these activities under continuous development and integration, as shown in Fig 1.

    This all starts with sprint planning and modeling of requirements in agile management software. Unit tests can be created automatically, based on requirements and use case stories. These unit tests then can be executed automatically based on pre-defined criterion, using automatically generated synthetic data.

    Fig 1: Continuous Development and Integration

    Next, virtual service interfaces can be generated automatically by simulating unavailable systems across the SDLC, allowing developers, testers, integration and performance monitoring teams to work in parallel—driving faster delivery, along with higher quality and reliability.

    The integrated code can be built and tested automatically against validation criterion, and upon passing, made available for continuous testing. This later phase, after code development and before QA, is also commonly referred to as continuous integration.

    Continuous Testing

    Continuous testing across app delivery not only enables enterprises to shift testing left, engaging it earlier in the SDLC, but it also helps them shift right, leveraging the same processes throughout staging and production.

    After the unit tests are passed and integrated builds are made available for QA, functional and performance tests can be completed—before the build is made available for staging and production. Various activities during continuous testing are shown in Fig 2.

    Before the functional and performance tests are performed, the build promotion criterion and configuration also need to be validated. Testers require provisioning of QA environments to deploy the build, along with virtual services and test data. Using test data management to create a subset of production data and mask sensitive information is another continuous testing requirement.

    Fig 2: Continuous Testing

    It’s also important to engage application performance monitoring in the QA environment before running automated tests, providing feedback about any problems back to developers.

    After the automated tests are executed and promotion criterion are met, the build is ready for deployment in staging or production environments.

    Continuous Release

    Continuous release matches release velocity to development velocity through controlled and scalable deployments.

    The goal of continuous release (sometimes referred to as continuous deployment) is to continually and automatically deploy the build into production; similar activities also can be performed in a staging environment.

    As shown in Fig 3, build configuration needs to be validated, with the production environment configuration updated before being sent to production. As with application code, configuration is also maintained in the version control system and can be automated for deployment.

    Fig 3: Continuous Release

    After the build is deployed into production, application monitoring should be initiated to track performance and customer experience. Feedback from the production environment can be passed back to developers to guide enhancements or sprint backlogs.

    This blog aims to outline core CD activities, and in subsequent articles I’ll provide additional details on entry and exit criterion, as well as metrics useful in tracking continuous development & integration, continuous testing and continuous release.

    In the meantime, check out, “DevOps for Digital Leaders: Reignite Business with a Modern DevOps-Enabled Software Factory,” a new book authored by my colleagues Aruna Ravichandran, Kieran Taylor and Pete Waterhouse.

    Based on learnings gained from numerous customers who have already embraced CD and DevOps, it highlights key practices for starting this journey, related ROI metrics and other useful tips.

    — Anand Akela

  • You’re Not Doing DevOps if You Can’t Pull the Cord

    You’re Not Doing DevOps if You Can’t Pull the Cord

    There are many concepts in Lean manufacturing that have applicability with software development “factories.” One example is the Andon Cord.

    The Andon Cord concept involves using signals and alerts to indicate that a station on the assembly line has a problem. Alarms are activated manually by workers pulling a physical cord or automatically by the equipment itself.

    Developed as one element of the Jidoka quality method and pioneered by Toyota and the Toyota Production System, the Andon Cord empowers workers to notify co-workers and management of quality and process problems across the assembly line—and even stop production.

    While pulling the cord to stop production sounds crazy, if a defect is found and can’t immediately be fixed, then pulling the cord means stopping the problem from flowing downstream, where it will be costlier to fix and have a severe impact on the customer experience.

    But the point of Andon Cord pulling isn’t only to fix problems. As part of a continuous improvement program, much of the “pulling power” comes from how it helps teams build quality into products and processes. When production stops, improvement begins.

    Many would argue that the application of Lean management concepts such as the Andon Cord is more challenging in software engineering. With software, there are many more opportunities for defects, and testing for problems only occurs late in the development cycle. So late, in fact, that in some cases customers end up doing most of the testing.

    But as the digital and physical world becomes inseparable, excuses and outdated practices are no longer acceptable. Just as we wouldn’t ask a customer to test air bag deployment in a car, we shouldn’t expect them to test whether a new software-enabled telemetry system is subject to security hijacks.

    So having the power stop software development production line to prevent defects, security problems and safety issues during the actual development process makes perfect sense. But how do you start?

    Automate Andon with Continuous Testing

    In many organizations people are often afraid to speak up about problems. Even if bugs are suspected, teams will cover up problems by coding around them, even fudging tests. Owning up to problems doesn’t come naturally, so defects and crippling technical debt accumulates. DevOps practitioners need their own version of the Andon Cord: fully automated testing.

    With DevOps promising software throughput at dizzying rates, quality must accompany the velocity. Fail here and the continuous delivery glass can only be half-empty. This is why the process must incorporate its own automated testing Andon Cord to prevent problems moving downstream. Automation must be comprehensive, providing maximum coverage at every stage in the software pipeline—from requirements definition to full production and including code quality, performance, security, compliance and test data. Then, when a test fails, the system must be capable of stopping everything until the problem isn’t just fixed but can’t happen again.

    Eliminate Waste at the Source

    If you’re pulling the cord but not making improvements you don’t have an Andon Cord. Consider a factory scenario where machine operators always have to sweep up dust that’s clogging production equipment—obviously, they should be eliminating the source of the dust.

    Unfortunately, many software engineering practices involve lots of dust-sweeping.

    When developers need access to dependent systems for testing, they might mock up test code. If QA needs test data in a hurry, it could copy production data but fail to mask sensitive customer details. In IT operations, staff console watch in the hope of gleaning signals from all the noise—the list goes on. Lots of sweeping without ever eliminating the source of the problem.

    Fortunately, tools with Andon-like capabilities can help. Service virtualization techniques can emulate dependent systems to remove testing constraints, while automatically generating synthetic test data helps eliminate the possibility of costly compliance breaches. Modern performance monitoring systems also play an essential role. By providing performance insights before, during and after every software build, they help cross-functional teams swarm problems when the cord has been pulled by an automated test.

    Build the Ultimate Cord – Customer Experience

    If an organization considers itself in any way customer-centric, it will be able to “pull the cord” when it detects anything that detracts from the customer experience as delivered across multiple channels—web, mobile, call-center. The winners go further, using these events as fuel for continuous improvement programs.

    This all requires some serious smarts, not at the least application analytics to monitor the customer experience in real time. Such advanced systems become the ultimate Andon Cord, since they treat every sub-optimal customer experience (e.g. functional issue, usage behavior, performance problem) as both a defect and opportunity for improvement. Through advanced analytics and integration this improvement could take many forms—such as immediately prescribing code improvements to developers, or support systems notifying customers when a poor experience is about to occur—maybe even (shock-horror) compensating them financially for any inconvenience.

    A strong DevOps culture thrives when individuals call out mistakes and inefficiencies. Fail in this respect and organizations risk paying a heavy price, such as broken customer trust and lost business. Seek out automated methods that help you metaphorically pull the cord and keep on pulling it. Only then can teams truly excel in driving real improvements that will help business gain and retain customers.

    — Peter Waterhouse

  • An IT Practitioner’s Guide To DevOps

    An IT Practitioner’s Guide To DevOps

    Two technology innovations of the past decade have fundamentally changed how IT works and given rise to the “DevOps” phenomenon. Before we dive into what DevOps is and how IT organizations should adopt this culture, let’s review what gave rise to it in the first place.

    Pull from Agile Development

    The last decade has seen many organizations abandon the traditional “waterfall” software development process in favor of “agile” development. Agile is all about small, specific user-centric stories, described by a product owner and delivered in short sprints of two to four weeks by small, focused development teams.

    Agile development led to the emergence of more frequent code drops from software engineers, which needed to be deployed more quickly. No longer was it okay to take days or weeks to deploy a new software version; developers working in two-week sprint cycles want deployment to work within minutes with a high degree of predictability.

    Enabled by the Cloud

    Before the cloud came along, access to infrastructure required human interaction and manual operations. Processes typically could not be automated effectively. This meant that operations were slow and error-prone and created a huge point of friction in the software development life cycle.

    Around the same time that developers were adopting agile development, the cloud became real, and infrastructure could be orchestrated effectively using APIs to provision everything from machines and networks to storage, routing policies and firewalls.

    DevOps: Closed Loop Automation Between Development and IT

    Agile development and cloud computing created the perfect storm in which DevOps was born: this new way of doing IT is all about ensuring closed loops, with 100 percent automation.

    picture2

    Figure 1: DevOps is about ensuring a closed-loop, fully automated workflow between software development and deployment.

    In the simplest terms, this meant that each of the steps outlined in Figure 1 need to be fully automated, and integrated with preceding and following steps. Doing this ensures software can be developed faster, deployed quickly, and iterated upon more rapidly.

    Benefits

    Done right, DevOps leads to a virtuous cycle of greater productivity across development and IT, greater agility in bringing new experiences to market and greater impact on the business.

    Adopting DevOps: You Can Phase It In

    Organizations looking to adopt DevOps should do so in phases and define metrics for each phase. Here’s a typical phased approach that you can use as a baseline and customize to your organization’s unique needs.

    Phase 1: Adopt Agile Development and Cloud Computing

    Agile development and cloud computing are profound technology shifts and create the underlying culture in which DevOps can thrive. So, the first step is to ensure organizational buy-in for both of these trends:

    1. Agile development can be adopted in stages. The most important is to agree that software will be built and shipped in short, two-to-four-week sprint cycles, and with a focus on user stories.
    2. Cloud computing can be adopted no matter what your IT infrastructure looks like today. Use hybrid cloud platforms such as OpenStack to provide programmatic access to a VMware or Linux environment, or use a public cloud platform such as Amazon Web Services.

    Phase 2: Commit to a CI/CD Workflow

    Continuous integration (CI) systems such as Jenkins or TeamCity provide a way to pipeline and integrate the various steps in a DevOps workflow. The choice of system doesn’t matter as much as ensuring that all steps of the workflow will be integrated (eventually) into this workflow.

    Phase 3: Repeatable, Automated Software Deployment

    Many organizations tend to have some manual steps in the packaging and deployment of their software, especially for getting it into production. This often reflects underlying assumptions about manual oversight and intervention, and it breaks the closed-loop philosophy of DevOps.

    To succeed with DevOps, you need to fully automate the software packaging and deployment process. In addition, ensure that production builds are deployed in the exact same manner as developer builds, using identical environments.

    Cloud technologies such as orchestration (OpenStack HEAT or AWS CloudFormations) and configuration management tools such as Ansible, Puppet or Chef can be used to implement this phase.

    Phase 4: Automate Software Testing (Continuous Delivery)

    Once software deployment has been automated, the next phase is to automate the validation process for new software releases. If you’re like most organizations, testing involves several rounds of back and forth with QA teams and product managers, and it often takes weeks to resolve the last few bugs in each release. To reach the next level of DevOps automation, this process has to be completely lights-out and automated.

    This requires a fully automated software test suite, across the back-end and front-end, so that product managers have confidence in the quality of the software being released. Once implemented, you finally have a closed loop between the critical elements of the DevOps life cycle. While release to production may still require someone to approve it or “push a button,” this is a very respectable state in the DevOps journey.

    Phase 5: Continuous Deployment

    To get to the most advanced DevOps phase, you need to automate the “go live” process for software releases. This in turn will allow you to take even smaller, more frequent code drops and push them into production.

    The most advanced software teams in the world now operate this way, and the organizational implications can be transformative: Developers come into work feeling that they can implement a feature and have it go live immediately, and product managers can make commitments to the business with the confidence they can deliver, because they have this incredibly efficient software delivery machine backing them. Most importantly for you, IT is no longer seen as a cost center; it’s seen as the fundamental enabler for business productivity and time-to-value.

    Closing Thoughts

    Getting to Phase 5 represents the state-of-the-art in the industry today. However, another significant breakthrough promises to raise the bar even higher.

    In each of the prior phases, the DevOps workflow is integrating software development with IT operations; but there is nothing influencing software architecture to take operations into account.

    In other words, we’re still bolting IT operations automation onto software development.

    This is about to change. We’re on the cusp of a new era in software: instead of bolting operations onto the software development process as an afterthought, a new framework promises to bring development and operations together by design. With declarative, infrastructure-agnostic constructs to describe how applications are composed, interact and managed, Kubernetes promises to increase the operability of modern software systems by an order of magnitude.

    Everyone interested in the DevOps journey should be learning about Kubernetes and evaluating how it will improve your software development and operations workflows.

    About the Author / Sirish Raghuram

    sirishr_platform9Sirish Raghuram is co-founder & CEO of Platform9. As CEO, Sirish quarterbacks the team at Platform9 in realizing the company’s vision, which, on a given day, can range from strategy and road map to shipping code and running tests. Before co-founding Platform9, Sirish was an early engineer at VMware, where he held technical and management leadership roles and helped ship multiple vSphere products. Sirish’s work was reflected in several patents, VMworld talks and internal product innovations within VMware Engineering. Sirish has a B.E. in Computer Science from the University of Pune, India. Connect with him on LinkedIn.

  • So Long to Configuration Management Systems?

    So Long to Configuration Management Systems?

    When I was an operations architect at Ning way back in 2009, we were struggling to manage our deployments and configurations of more than 50 different application servers, or what the cool kids today call microservices. Applications were deployed across three distinct environments: development, staging and production. None of these environments were exactly the same as another, and differences in these dependencies, together with human errors, frequently broke our deployments.

    Ning was nowhere near the first company to encounter such problems. In the mid-1990s, as the public internet emerged, much of what was understood at the time about IT operations was challenged by the need to scale internet-facing systems. For many years preceding the internet boom, the maximum number of users was strictly limited to the number of employees in a company. That all changed with the World Wide Web, where the need to serve thousands and millions of simultaneous users easily exceeded the computing capacity of even the largest and most expensive computers.

    As three-tier application architectures were broken down into services and distributed across dozens or hundreds of individual servers, this proliferation of servers proved to be far too much for operations teams to handle. The worst problems lived within application deployment processes, where environments and requirements were ever-changing in successive releases of software. The lifetime of a server was a long one. Once deployed, servers were subjected to repeated changes by operations and developers in what would come to be called “configuration drift.”  The combined effects of changes, human errors and unknown states would create disastrous outages that damaged brands and destroyed online revenue. It would take sites days and even weeks to recover from these outages. Some would never recover.

    It was in the midst of this chaos that modern configuration management systems were born. CFEngine was the first to emerge in academia, where the need to manage multiple UNIX systems was felt early on. Later, Puppet, Chef, SaltStack and Ansible emerged directly from web-scale systems. Configuration management systems fulfilled several important needs. First, they abstracted the differences between operating systems, providing a high-level language for the system administrator to describe the desired state. Second, they automatically synchronized the desired state across large numbers of servers, reducing costly human errors. Finally, they provided idempotency to ensure that destructive operations were not applied unnecessarily.

    There is never a free lunch. The benefits of better abstraction were offset to a degree by the need to build and maintain evermore complicated recipes. Recipes frequently broke from time to time as package management features were updated, renamed or otherwise changed from the time the recipe was first written. Even when recipes worked, they were frequently broken by differences in environments or runtime errors. The configuration management systems themselves began to suffer from scale challenges, and evolved message bus architectures to improve their speed and responsiveness. By overcoming many of these obstacles, configuration management systems have now become a first-line tool for managing web-scale systems.

    But back at Ning, modern tools such as Puppet and Chef were still in their infancy and simply could not cope with the scale of our 60,000+ node virtualized environment. Ning had a lot of brilliant software engineers floating around, and they set about creating their own deployment automation system called Galaxy, and later a meta-orchestrator called Cosmos. Galaxy and Cosmos were great tools, but they required constant care and feeding and suffered from severe scale problems. There was just a lot of complexity in dealing with these automation systems while also trying to scale a top-tier social network for a hot startup company.

    As we began to analyze and post-mortem our deployment failures, it became clear that a majority of the problems stemmed from host and environment configuration errors. We needed a simpler way to deploy our software. Virtual machine images were a promising way to ensure consistency between environments with the tried and true “gold master image” approach. The problem with this approach was that building full system virtual machines was difficult in automation terms. Once deployed, virtual machines are also subject to configuration drift. Furthermore, while we were eager to adopt virtualization to get better resource isolation, we were wary of the overhead that hypervisors would create. Our benchmarks showed that if we adopted virtual machines, we would need to increase our physical server capacity to accommodate the overhead.

    At the time, Linux containers were already available in the kernel, but the distributions had almost no support for managing them. At the insistence of our VP of Operations, I cooked up an alternative solution based on jails, writing a small hack of a program called Warden to build and manage them. (Note: Totally unrelated to the Warden in Cloud Foundry). Jails gave us bare metal performance, resource isolation and a way to consistently deploy the entire environment for each application server.  With Warden, we could build jail images on the fly automatically and deploy them consistently with our existing tools, galaxy and cosmos. Hey, it was a Band-Aid for a difficult situation, but my experience with building Warden flipped a switch in my brain. I had seen the way lightweight virtualization could eliminate complexity, and I knew right then this was the way of the future.

    Today, with the success of Docker we are finally experiencing sweeping changes in the processes and tools we use to build, deploy, and manage distributed applications. At the same time, we are about to enter a brief period of containers disillusionment as early stage solutions get washed out. When that is over, what comes afterward is something of a golden era for distributed applications. I call it the container application platform, and its somewhere between what we think of as platorm as a service (PaaS) and Serverless today. In that not-too-distant future, containers will be the default output of the software build process, and where that container gets deployed will hardly be an afterthought. This unicorn magic will come from the ability to introspect containers to determine their environment dependencies, and a fully integrated tool chain that deals with deployment, scaling and dependency resolution automatically. I will give you just one guess which company is in the best position to own that future. “One ring to rule them all.”

    What happens to our beautiful configuration management systems? There has been much debate about this among smart people over the last few years. Most of these arguments boiled down to some kind of coexistence model. Indeed, configuration management systems are useful for dealing with a wide range of operational problems outside of the container platform. While it’s no surprise that Chef, Puppet, Salt and Ansible all moved quickly to include Docker support, I think we will see these tools begin to focus more on what Rob Hirschfeld calls achieving “ready state”. That is to say, everything that still needs to be done before you can lay down your container platform also still needs to be automated. Right now, that is still a great deal of work, and it’s a pretty safe place to draw the line on using the right tool for the right job.

    About the Author / James Thomason

    james-thomason_dsc0492James Thomason is chief technology officer at HyperGrid from Dell, where as CTO he led the vision, technology strategy and product road map for Dell Cloud Marketplace. He joined Dell through the acquisition of Gale Technologies in 2012, where as CTO he headed the product road map, architecture and engineering of the company’s flagship cloud and converged infrastructure automation software, GaleForce. Prior to joining Gale Technologies, Thomason was the CTO and Founder of Virtiv, a San Francisco cloud automation startup focused on Linux virtualization, acquired by Gale Technologies in 2011. For over 17 years, as a specialist in distributed systems and large-scale infrastructures, Thomason has been an entrepreneur and innovator at a number of notable Silicon Valley startups, including Exodus Communications, Digital Island, Netli, NetVMG, Netscaler, 3Leaf Systems and Ning. Connect with him on LinkedIn / Twitter.