devopsy.kinsta.cloud

Tag: puppet labs

  • The Sea Change that is DevOps – How it all started: Part 1

    The Sea Change that is DevOps – How it all started: Part 1

    Over the last 20 years, software has slowly taken over the world. Businesses that historically operated on 8-hour business days are now 24×7, navigating a global economy that never sleeps. The Internet enablement of everything has thrust what was originally an important support role into the lifeline of a business.

    A decade ago, Agile software development methodology forced more pressure into the IT system by deploying software at an ever-increasing rate to the point it’s at today – where innovative companies can deploy new software multiple times a day. DevOps is at the forefront of addressing the question of how IT deals with these problems, but like Agile before it, there’s a learning curve.

    In this two-part series, I will discuss how the IT world worked before DevOps and how development and operations teams united to adopt more Agile practices.

    How things were

    Before this sea change, a company would hire IT professionals who would purchase and configure servers, place them in a data center, copy mission critical software onto them and with some help from software engineering, get things running. After some quality checks, the IT team would lock down the datacenter controlling both physical and virtual access and “let things run.” The common way to provide stability was simply to not let anyone touch anything, as most problems have been (and still are) self-inflicted wounds. Keeping the system “static” kept a lot of problems from surfacing. The software was probably not more stable; you just didn’t find the problems as fast.

    In this world, IT professionals had a feast and famine existence. If a new service deployment was happening (usually every 3 to 6 months) it was an all hands on deck activity reminiscent of a rocket launch. The same manual deployment would happen, along with software engineering, database engineering, etc. Data migration would occur, new software would be installed, and the system would manually be brought back up. In the event of a software problem, or data migration problem, the team could elect to rollback and bring the old software back up. If things went well, then again some basic end-to-end quality assurance would take place and the new software release would be unleashed on the world. More progressive companies would do this for part of the system, diverting traffic to avoid perceived downtime. It would often take an entire night to do a new service release, and companies generally did them at night because downtime was not unusual.

    Then software development changed

    The competitive landscape that the Internet created along with the now ubiquitous SaaS business model made companies recognize that their ability to innovate the product was a competitive advantage. A lot of companies had adopted Agile software development practices in response to the increasingly high failure rate of larger efforts. Part of Agile is creating smaller development efforts that get code functional faster, reducing the size of the endeavor. This is commonly referred to as a “software sprint.” This promise of innovating the product faster was very alluring, especially to Web 1.0 companies that were now in feature battles with more nimble Web 2.0 companies who had more quickly adopted Agile. It was becoming more and more evident that the problem was shifting from an issue of creating software fast enough to deploying the necessary code in an expedient manner.
    The big problem was the deployment stage of the process. IT teams were accustomed to changing production systems every three to six months. This combined with a trusted model that largely kept software developers out of production systems, effectively nullified any Agile gains in SaaS businesses. In other words, half the product development cycle had been fixed with Agile, but the other half deploying the software had not yet caught up.

    Operations as a changing landscape

    Major changes were also occurring for operations inside companies, many of which were driven by the development side of the house.

    • Virtual server technology started to get major traction within production environments. This allowed teams to treat servers as software. You no longer had to run the operating system and application on a physical server, you could run them on a virtual server (that ran on physical servers). The configuration of servers that, by hand, took an incredible amount of time, could essentially be defined once and then cloned, the same way you would clone a file. Deploying a new server was much easier, and a corrupt server could simply be deleted and reborn in minutes. All these virtual servers could be managed by a single piece of software called a hypervisor, with which creating a new “server” would take a minute, when it used to take weeks.
    • Puppet and Chef were born. At the same time development organizations, recognizing this major problem with deployment, started addressing the production configuration problem with tools such as Puppet and Chef that essentially allow software and IT teams to codify the configurations of virtual and physical servers and deployments of new code. This allowed for repeatable processes that could be rapidly done and redone. No longer did your IT team have to install software by hand, instead they could code it.
    • Continuous integration and delivery enabled software teams to use newly developed tools like Jenkins to do nightly software builds, perform unit tests of parts of the software and automatically deploy software into production. All of these tasks had been highly manual operations historically and fraught with mistakes and inconsistencies. Codifying these steps would give way to deploying software within minutes or seconds rather than days or hours.

    Most importantly, all of these efforts were largely driven by the development of more progressive operations in the company. The operations side of most companies was never conceived as a “maker” culture; the handwriting was on the wall – operations had to change or it will would have become a victim of this revolution. I’ll take a look at some of the resulting changes in the second part of this series.

  • Inside Look at State of DevOps Report: Q&A With Puppet Labs

    Inside Look at State of DevOps Report: Q&A With Puppet Labs

    Last week Puppet Labs released the analysis behind its annual study of DevOps and continuous delivery practices in the 2015 State of DevOps Report. Now in its fourth year, the report offers an in-depth look at the findings of the team on what it means to be a high-performance IT organization—the cultural characteristics these organizations share, the lean engineering practices they engage in and the technical architectures they depend upon.  Like last year,  however, the report tends to be a bit of a black box when it comes to actual statistical findings. While some figures do make it through, for example the oft-flogged statistic of high-performance organizations deploying 30 times as fast with 200 times the lead time of low performers stands pat, the report itself is somewhat oblique about the statistical basis upon which the designators of low-, medium-, and high-performance were determined.

    devopsy.kinsta.cloud spoke to Nigel Kersten, CIO for Puppet, and Alanna Brown, senior product marketing manager for Puppet,  about the report to get a little more detail, to have them explain the process behind the analysis, and to offer up some of the most important facets of their findings over the last few years.

    devopsy.kinsta.cloud: Can you tell our readers a little bit about the high points from the report?

    Kersten: Big finding from 2013 was the orgs that used Devops’ practices deployed code 30 times more often and had 50 percent fewer failures.  In 2014 what was quite fascinating was that we actually found a link between these practices and overall organizational performance, not just IT performance. Higher-performing IT organizations are twice as likely to exceed their own profitability, market share and productivity goals.

    If we fast-forward to today, the high-performing organizations using Devops practices are more stable and reliable than they’ve ever been.  They’re recovering 168 times faster from their failures and have 60 times fewer failures due to their changes.  So we’re not actually seeing throughput increase from the year before, but what we are seeing is the quality and speed has definitely increased.  So people are producing changes that are of higher quality and that require fewer rollbacks.

    And so what we’re concluding from the data that we’ve seen this year in the report is that there’s been a really huge focus to the left of the assembly line from production. So if we envisage the developer work flow application delivery, essentially we start with the developer laptop like on the far left, and on the far right is the application actually existing in production.  So from the point between the developer’s laptop and the actual release, we’re seeing organizations focus much more on that stage, putting more testing in the hands of the developers, and all of these practices not just being focused around the release but the pre-production stage of the application life cycle having a really huge impact upon quality.

    devopsy.kinsta.cloud: Why doesn’t Puppet publish the actual data from the survey? You have great analysis here, but there’s very little statistical proof behind it—why don’t you provide that?

    Brown: Well so we have a ton of data, and part of the reason we don’t issue the data is because you wouldn’t be able to actually get to the same results without doing a ton of data cleansing and statistical analysis.  So our analysis is actually fairly rigorous, and some of that analysis doesn’t really lend itself very well to graphs and charts and visuals and things like that because it’s all based on correlations and predictive regressions.  So that’s part of the reason.

    Kersten: I think in many ways it’s analogous to releasing software to the world. We have all of these tools inhouse.  We have all of these Excel spreadsheets we’ve put together.  We have all of these you know pivot tables and different bits of cluster analysis, but they only – they’re not particularly usable for anyone else right now.

    As much as we would love to work out how to polish all of that up and share it so it’s one big data set along with all of the tools, our biggest worry is honestly being that people would actually draw sort of misleading and incorrect conclusions from the data because of the level of analysis that was required.

    devopsy.kinsta.cloud: Can you tell us, then how you came to the determination of what’s a high-, medium- or low-performance organization? Because if you look at some of these numbers, you say they’ve got thirty times more deployment frequency. But of course they do because they’re high performers. So how does that work out statistically?

    Brown: No, no, you’re absolutely right, and I will agree that IT performance metric is a bit circular.  However, those are the actual metrics that are important to an IT organization, so they care about deploy frequency, deployment lead time, their mean time to recover and change failure rate.  So we feel like that’s a pretty good approximation or a proxy for IT performance, and it’s better you add at doing those things that is correlated to higher performance overall.

    Kersten: One thing, too, is that even though there’s a whole spectrum in terms of how fast people are able to actually deliver and ship code, what actually came out really strongly out of the survey this year and in previous years is that the high performance (orgs) definitely cluster in terms of data points. It’s not just a smooth spectrum, but there’s this really obvious clusters of high performance that leap out that all have a very high correlation of the kinds of practices they use.  So it is a little bit circular but it’s also it’s not that we’re seeing this smooth spectrum from zero to 100.  We’re seeing the masses and then the high performers, and they divide into quite clear clusters.

    Brown: Yeah, and they’re not arbitrary clusters; they’re actually statistical.  So the low IT performers share all the same characteristics and the high IT performers share all the same characteristics, so they’re very clear.

    devopsy.kinsta.cloud: OK, so let’s talk about some of those common characteristics between these groups if you’ve got statistical bases for these.  Tell me a little bit about the low performers and we’ll kind of go on up the spectrum and we can talk about the characteristics they share.

    Brown: So for IT performance, deployment pain was actually the number one most highest-correlated construct. And deployment pain consists of a couple of questions.

    Like how painful are your deployments?  Are they very disruptive events, etc?  So there’s three questions that we ask around deployment pain.

    The next thing that’s highly correlated with IT performance is version control. We use the version control for all production artifacts.  Following that is culture, and the model we use for culture comes from a guy named Ron Westrom who is a sociologist.  And he created a typology of cultures, and you can see the table in the report itself.  So he has a pathological culture, bureaucratic, and then generative.  So the more generative your culture is, the more likely you are to have higher IT performance.

    Kersten: And I think this is what was so fascinating for us when we first started adopting Westrum last year We actually adopted Westrum as part of the model in that this pathological versus generative cultures, this is the language our practitioners in this space use to talk about you know what’s so great about working there: “My ideas are valued. I get to create ideas.  I can create impact.” Similarly around what’s horrible about working at this other place.  It maps to the pathological and generative descriptions of organizations perfectly really.

    devopsy.kinsta.cloud: So what would you say is the key characteristic in that first leap from low performance to medium performance, based on your data?

    Brown: So the lower performers in terms of IT performance will be doing less automation, less testing.  They haven’t automated their deployments, for example.

    Kersten: I think one way I at least think about is that there are these characteristics that are necessary but none of them are sufficient on their own, and so you see people who say just adopted version control.  They’re definitely heading ahead of the pack of the people who haven’t adopted any of these tools.

    But there’s this transformative leap once you actually adopt version control and continuous delivery and the highly-automated conflict management system, that those things together are greater than the sum of their parts.  That is at least how the data seems to us, ’cause we don’t actually have this smooth little spectrum between low performers and high performers who just adopted one of those technologies.

    We definitely see improvement from those people, but it’s when they adopt all of them in total that they suddenly take this sort of order-of-magnitude leap.

  • Deliver faster, but don’t sacrifice quality

    Deliver faster, but don’t sacrifice quality

    Developing applications, and deploying servers faster are great goals. Being agile and streamlined can give an organization a clear competitive advantage. However, faster does not equal better, and being speedy for the sake of being speedy can lead to avoidable mistakes. Faster is only better if you can achieve greater speed without sacrificing quality.

    At the same time companies are racing to be faster than their rivals, they’re also dealing with an increasingly complex IT infrastructure. IT environments are becoming highly heterogeneous—merging physical, virtual, and cloud platforms, and combining legacy systems with cutting edge technologies. The challenges of managing such a diverse IT infrastructure simply add to the potential issues that can arise as organizations try to work faster.

    That is where DevOps plays an important role. DevOps tools and technologies establish a process chain between development and deployment. It enables organizations and IT admins to reliably track and understand the components in use, and any changes or updates that occur. In order to achieve speed without sacrificing quality, it’s important to maintain as consistent an environment as possible, and to know exactly what has been changed.

    “For hypervisor based virtual images, this depends on the strict process around handling the images, but for container based virtual images depending on a cascading snapshot mount namespace, where there are layers of images built one on top of the other, strict handling becomes unnecessary because everyone can see transparently (via the image cascade) what changes at each stage of the process,” explained James Bottomley, CTO of server virtualization for Parallels. “This gives everyone in the DevOps chain the ability to verify each step and see the changes for themselves, leading to much higher assurance that the resulting deployment is correctly tested and released.”

    Ken Cheney, VP of business development for Chef, describes another way that DevOps helps boost speed without affecting quality. “Automation turns infrastructure into code. For example, Chef recipes automate all facets of management and orchestration—from provisioning bare-metal and cloud servers to automated configuration and application deployment. You can replace manual, slow, error-prone procedures with code that is versionable, testable and repeatable.”

    “When infrastructure is code you can test it just as you would applications—using automated tests that find problems early, before you release to production,” added Cheney.

    In its 2014 State of DevOps Report, Puppet Labs drew an interesting correlation between speed and quality. Essentially, Puppet Labs found that speed and quality become a self-feeding circle.

    “There’s a virtuous circle at work here: As stability improves, IT performance improves. This improved performance helps to create a better-functioning business that can pay attention to the communications and processes that enhance and improve stability.”

    The converse is also true. Poor quality results in issues, and necessitates allocating resources to troubleshoot and resolve problems, which impedes the speed at which an organization can function.

    DevOps tools and technologies enable organizations to streamline development, and work more efficiently without sacrificing quality. Businesses that attempt to work faster, without also working smarter by using DevOps are putting themselves at a disadvantage.

  • Equalizing IT infrastructure with DevOps

    Equalizing IT infrastructure with DevOps

    IT is getting more complex. Managing on-premise servers, cloud servers, and hybrid environments that span various operating systems and standards creates challenges for IT admins. DevOps technologies and platforms, however, can be an equalizing force among the cloud chaos—making it easier for organizations to deploy servers and applications quickly and consistently regardless of platform or environment.

    Once upon a time most organizations were fairly standardized. Servers were located on-premise in some sort of closet or data center, and IT admins knew exactly what operating system they were working with. Now, those servers may be physical or virtual, on-premise or in the cloud, Windows or Linux, Azure or Amazon, and in many cases IT admins are dealing with an infrastructure that amounts to an all-of-the-above approach.

    As the name implies, the core of the DevOps movement is to remove the barriers between developers and operations. Many of the tools and technologies that comprise the DevOps movement, however, also have the added benefit of providing a consistent, modular approach to developing and deploying applications and infrastructure. Those technologies also reduce friction between on-premise and cloud environments, and allow organizations to streamline operations across different platforms and operating systems.

    “One of the biggest reasons for the increased interest in DevOps tools and practices is because businesses have to move and iterate faster today than at any other time in history,” declared Tim Zonca, director of product marketing for Puppet Labs. “So IT teams are pursuing DevOps flows, where development and operations practices, systems, and tools work in lockstep.”

    As IT infrastructures become more complex, DevOps becomes more crucial. Expansion of IT across physical, virtual, and cloud environments make even minor inconsistencies into critical issues as organizations grow, and brings additional burden to address those problems.

    Many DevOps technologies revolve around using virtualization to deploy applications inside pre-built environments, or modular blocks that can be deployed and re-used across various platforms and environments. These tools remove most of the chaos, and enable organizations to simplify deployment across virtually any infrastructure.

    Docker is a great example of the impact DevOps is having. Docker containers enable developers to package an application, and transport it anywhere Docker containers are supported without modification to the virtual environment. As Docker continues to spread influence across Azure, Parallels, and other cloud and virtual platforms, it is asserting itself as an equalizing force that eliminates the need to even care what the underlying infrastructure or operating systems are.

    Puppet Labs is another illustration of the equalizing force of DevOps. Zonca explained, “Puppet makes it easy to standardize and add consistency to your infrastructure so you can make fast, repeatable changes across physical and virtual machines, on premises or in the cloud. Because these tools create a predictable environment, you don’t have to worry about a simple update taking down an entire system your business relies on.”

    James Bottomley, CTO of server virtualization for Parallels, suggests that even DevOps has its limitations, though, and that it could eventually get in the way of advancing productivity. “However, one cannot help thinking that DevOps, while a leveler, is also a separator because it is only accessible to developers with the Ops skill, not to all developers. Additionally, the utility of DevOps depends upon the availability of Infrastructure and, while IaaS is clearly the dominant Cloud technology today; it won’t be forever. At that point, one might imagine that the DevOps technique might impede rather than assist the next paradigm shift in the cloud.

    For now at least, DevOps technologies enable developers and IT admins to take a modular, platform-agnostic approach to many aspects of applications and infrastructure to simplify and streamline regardless of the underlying environments or operating systems.

  • Microsoft determined not to get left behind by DevOps

    Microsoft determined not to get left behind by DevOps

    Microsoft is a massive company—a behemoth with a lot of inertia, and a lot of bureaucracy in place that make it difficult to adapt quickly. Under Satya Nadella, however, Microsoft is striving to re-invent itself, and—to the extent that it is possible—become more agile. For evidence of Microsoft’s new philosophy, look at how Microsoft is aggressively embracing concepts like open source and DevOps that used to be viewed as pariahs.

    I wrote recently about the partnership between Microsoft and Docker. Microsoft added support for Docker containers in the Azure cloud platform within Linux virtual machines earlier this year, but then it upped the ante by announcing that it will support Docker natively in the next version of Windows Server, and that it is working closely with Docker on an open source project to drive development of the Windows engine for Docker.

    “An important aspect of this announcement is the noticeable speed we are observing Microsoft move to implement open source technologies,” explained Al Gillen, an analyst with IDC, in a report about the relationship between Microsoft and Docker. “The company appears to be determined not to be left behind as the velocity of open source developer technologies accelerates. The company appears to be willing to integrate competing technologies, or technologies from competing ecosystems at a faster pace and with less hesitation.”

    Docker isn’t the only piece of the puzzle, either. Earlier this year Microsoft joined forces with Chef and Puppet Labs, and integrated both of those popular DevOps platforms into Azure as well. Incorporating Chef and Puppet Labs gives IT admins more flexibility to deploy, configure, automate, and manage VMs in Azure.

    What the new philosophy really represents is a realization. Nadella’s Microsoft sees the writing on the wall, and understands that it’s a new world. Technologies have changed, and the way companies and individuals use technology has changed, and the days of dominating the tech universe as a virtual monopoly are gone. The two choices facing Microsoft were to either stubbornly cling to its former glory and die a slow, painful death, or adapt and embrace change in order to remain relevant, and continue to play a vital role. Thankfully, it seems Microsoft has chosen the latter path.

    What does embracing change look like? Microsoft appears to be taking a two-pronged approach. On the one hand, Microsoft is changing internally to shed some of its entrenched bureaucracy and enable it to function with more agility. It is also striving to keep up with new technologies itself—buying Nokia to bring smartphone manufacturing in house, developing the Surface tablets, rolling out the Microsoft Band wearable health tech, and other initiatives demonstrate that Microsoft is working hard to break free from the shackles of its own past, and find ways to keep up with the changing tech landscape.

    The second—and arguably more important—prong is how Microsoft is addressing the new multi-vendor, cross-platform world we live in. Microsoft recognizes that it is not the only game in town, but it has a vested interest in ensuring that customers continue to rely on Microsoft tools no matter what platform they choose. To that end, Microsoft is doing a much better job of offering its own products and services across a wider range of devices and platforms.

    The new Microsoft represents a win-win-win scenario. It’s a win for Microsoft because it enables the company to shift and evolve to stay relevant with a rapidly-changing tech landscape. It’s a win for open source and DevOps because companies of all sizes are built around Microsoft operating systems and applications, so incorporating support for open source and DevOps tools exponentially increases the potential customer base. Finally, it’s a win for the customers themselves because they get to maintain their existing investments in a Microsoft infrastructure, while also gaining access to new tools that enable greater automation and efficiency.

  • Trolling PuppetConf: Surveying Configuration Management

    Trolling PuppetConf: Surveying Configuration Management

    Troll (Internet)

    In Internet slang, a troll (/ˈtroʊl/, /ˈtrɒl/) is a person who sows discord … by starting arguments or upsetting people,[1] by posting inflammatory,[2] extraneous, or off-topic messages in a… community …with the deliberate intent of provoking readers into an emotional response[3] or of otherwise disrupting normal on-topic discussion.[4] 

    -Wikipedia

    When I stopped by the Chef booth at Velocity Santa Clara this year Nathen Harvey was out of XL t-shirts. I happened to have a Puppet shirt on at the time too, but it was just a lucky coincidence. Nathen’s always advocated using something (anything) to automate your infrastructure, as the ‘A’ in the CALMS model for devOps stands for automation, but the look on his face when I crossed his line of sight was as delightfully awkward as it could have been. Nathen told me he’d get me a Chef shirt if I got him my address back in Atlanta. I had been meaning to do it before I made it out to PuppetConf, but a tweet from Mandi Walls ultimately ended up prodding me into action.

     cheftroll

     To Nathen’s word, he got me a shirt in under a week, and in plenty of time for #PuppetConf.

    chef

    chefShirt

     

    I think “Stirring up delight” was the right shirt to send me too. I was half worried I’d get a shirt that was much more confrontational, putting me at risk of getting jumped and having my lunch money stolen. Lucky for me pretty much everybody that noticed the shirt thought it was funny. At least anyone who said anything to me about it. A proper troll is going to get at least one person a little bit angry though, right? Anyone that recognized I was in a Chef shirt happened to create an opportunity for me to ask a few questions, so I did.

    1. Have you used Chef, Puppet, CFEngine, or any other configuration management tools?
    2. Which do you prefer?
    3. Why do you prefer that particular tool?
    4. Why does the competition between these tools help or hurt our industry?
    5. Where do you see these tools in five years?

    I got pretty consistent answers just talking to people, and the Q&A often evolved into a natural conversation, which I prefer. (Another devOps principle I like to practice is shut the hell up and listen). I also put up fliers with a link to a survey while at the conference in an attempt to gather some empirical data. Truth be told, the survey was the real troll.

    One of the best talks I saw at PuppetConf was by Lindsay Holmwood, call The DevOps Field Guide to Cognitive Biases. I wasn’t familiar with the term before the conference but I think I had the idea. I also know I still don’t fully understand it. Watch the video and check out the books Lindsay suggests at the end. It’s pretty deep stuff. I really like his advice on having engineers manage and managers engineer to help break these biases down, as it creates a lot of empathy between the two groups.

    Given my own cognitive biases, I wouldn’t have filled out the survey if I had seen one of the flyers I posted. The flyer said nothing about a home school project for twin girls, which would have had a much better chance of getting my attention. Instead, it just asked folks to fill out the configuration management survey with the promise of cookies for their trouble. I’m pleased to say the tech community will respond to surveys for cookies even when crudely marketed. While I have enough responses to do sound statistical analysis, I know my data isn’t sufficiently stratified and probably doesn’t represent the larger population of the configuration management community. In other words my data lacks diversity.

    My daughters’ home school project is to start a business, and the twins decided to launch Gemini Snacks. Their company is pretty much an online bake sale to allow them to give back to all the hospitals and organizations that have helped them and cheer up the kids that can’t go home yet.

    The deal I made with my daughters was simple. I would print out flyers for PuppetConf to get survey responses to help me write this article. I would thank as many people as we could for filling out the survey by sending them some cookies, and I would hire their company to bake and ship those cookies. If people filled out the survey my kids would be in business.

    There was a strong enough trend in the responses that I can discuss those now with confidence despite a lack of diversity in the data. Of course I got a lot of responses from Puppet users, but I also heard from people using CFEngine, Chef, Ansible, Salt, and bconfig2. A lot of people weren’t using anything and just recently picked up Puppet.  I ran into a few folks that had used three or more tools at one point or another and could give comparisons which were really interesting. Here are the main conclusions I’ve been able to draw so far.

    A Rising Tide Floats All Boats

    Most organizations are still sshing into a lot of hosts to handle things, sometimes they do it with scripts. Configuration management (CM) tools are still becoming a fully adopted standard. I have heard users of every CM application in the market say it really doesn’t matter what tool you’re using, automate your infrastructure with something. As a result this is still very much a growing sector in the tech industry.

    Don’t get into a Configuration Management Bake-Off

    Bake-Off is the word that was used, and more than one person put it that way specifically. I don’t know what you’d call setting up test environments to try different CM tools before making a decision, but I’d call it a bike shed of yak shaving myself. Bake-Off is probably more familiar people anyway. My opinion is you should either be using Puppet or Chef, but if you’re not using any configuration management tool at all it’s advised that you start.If you can’t help but take more than one for a spin before committing then keep it short and sweet. You’ll be better off taking the time to commit to one and put the work into making the transition quickly than trying to figure out the easiest way to do it. This automation will take some effort. The key thing to understand is the right tool depends on your environment.

     

    There’s a Lot of Respect and Appreciation for CFEngine

    CFEngine is a trailblazing academic project that gained a lot of popularity in the early days of CM. The adoption of the software in the open source community really proved the viability of CM as we know it. While the application isn’t seeing continued growth now that there’s competition with newer tools in this space, several people went out of their way to emphasize the immense respect they have for the product and its founder, Mark Burgess, as CM wouldn’t be as evolved without them.

     

    Puppet: Preferred by Ops Folks because it’s Easy to Configure

    The main thing the ops crowd seems to prefer about Puppet is that it’s easy to configure. I had a book recommended to me  at the conference called, “Switch: How to Change Things when Change is Hard”, by Chip and Dan Heath. The book describes a “Project Mood Chart” used by IDEO, a product design firm, to brace their clients for adversity.

    productMoodChart

    When a new project starts we’re optimistic and hope for the best. We inevitably face adversity during the course of our work, and while this adversity is always frustrating it does lead us to insight. Empowered by our new perspective on the problem we can proceed to a viable solution with confidence.

    I believe this is how Puppet evolved out of CFEngine. Luke Kanies, the founder and CEO of Puppet, was an early contributor to CFEngine. I can see Luke following this curve, gaining insight on the configuration difficulties the CFEngine community was experiencing, and going on to found Puppet, confident he had a product better adapted to industry since it was easy to configure.

     

    Chef: Preferred by Devs because it Allows them to Treat their Systems Like Objects

    I can see this same curve being followed as Chef came out of Puppet as well. With ease of use comes a lack of robustness. Puppet’s advantage over CFEngine is it’s easier to configure, but there seems to have been a point of frustration within the development community that lead to the creation of Chef. Developers like to treat things like objects, which should be obvious if you have any experience with object oriented programming. Abstracting the complexity of the software we write so it can be configured by end users that are not developers is challenging. Since Chef allows its users to treat their systems like objects, the difficulty surrounding the abstraction of any complexity into a declarative model is circumvented.

    Which is the main reason ops folk don’t like Chef as much, because they’re not programmers. If you’re in an organization with a mature operations department that’s not so interested in learning to code, Puppet ‘s declarative model for describing systems is going to be a better fit. However, if you have developers doing most of the operations work out of necessity in a small shop or an operations group that’s doing more software development for internal tools, especially if they like Ruby, Chef may be a better fit for your organization.

     

    Ansible: Symbiotic Orchestration

    The best things anyone had to say about Ansible is that it’s great for orchestration. It seems to be competing more directly with mcollective, a framework Puppet Labs supports for orchestration tasks. Mcollective can be used with both Chef and Puppet, and the same appears to be true for Ansible. Ansible is also agentless, managing nodes over SSH. Depending who you talk to this can be a good or a bad thing. By not running an agent the nodes aren’t dealing with the overhead that would come with it, but you’ve also lost the ability to monitor your systems as tightly as you could if they were running an agent. I’m under the impression CM is trying to move us away from SSH loops. While I don’t think it’s fair to write Ansible off as an overblown SSH loop, I am of the opinion that the advantages provided by agents running on each node in your environment are well worth the overhead. It will be interesting to see if Ansible’s relationship with other configuration management systems continues to evolve into a symbiotic orchestration provider or if another product, like mcollective, continues on to fill that void.

    Salt Isn’t Really Configuration Management Software, but it’s Fast

    Salt is a remote execution platform build on zeroMQ that was designed with speed in mind. Everything salt’s being used for, including CM, has been an afterthought to the fast remote execution platform at the heart of the project. The constant connection the nodes (called minions) maintain with their master definitely contribute to the speed of the system. This would lead me to believe Ansible is the slowest of the four technologies discussed here, since it doesn’t run an agent at all, and Chef and Puppet would fall somewhere in the middle since they run an agent on their nodes but it isn’t on constant communication. (This is an educated guess on my part. Please let me know if you can speak to the speed concern from personal experience.)

    I have run into a few businesses that deal with high speed financial systems that use salt for a lot of things, including CM, but only because it’s fast. If you have systems that demand split second responses to failure and change, salt may help solve a lot of problems your organization is currently facing. I’m not convinced it’s that easy to use.

    Salt seems popular with groups that want to avoid Ruby like it’s the plague or folks that are so horribly comfortable in a CLI that using a GUI just seems unnatural. I think these are terrible reasons to select a tool, since it’s directly avoiding the ‘adapt or die’ mentality that the devOps community embraces. Still, depending on your environment and the adversity you’re dealing with, these may be entirely reasonable grounds to use salt instead of another technology. I’d be less concerned if you told me you picked salt because it’s fast though.

     The Survey is Still Open

    I hope you like what you read here but I’m sure somebody doesn’t agree with me at all. The good news is you can still fill out the survey and let me know. When my data looks sufficiently diverse I’ll publish another update, and there’s a decent chance you’ll get some cookies for you time too. I’d also like to thank everyone that took the time to discuss this with me. Gene Kim, John Willis, Chris Webber, Charles Johnson, Dawn Foster, Kara Sowles, everyone at Puppet Labs, and of course Nathen Harvey and Mandi Walls, all helped to make PuppetConf a huge success and a memorable experience for me. I’ve learned way more than I expected from the experience and I’m excited to see how the competition in this space plays out in years to come. Competition is a good thing. As different challenges arise and as projects go through mood swings I believe we’ll see two things happen: The communities for these tools will adapt them to adjust to the environment they’re operating in, and at some point enough adaptation will lead us to a new set of tools all together. I for one am really excited to see what happens.

  • Harvard Business Review Survey: IT responsiveness predicts business success

    Harvard Business Review Survey: IT responsiveness predicts business success

    Increasingly, the C-suite is finding that the speed with which companies today are able to take advantage of new technology and exploit their IT operations determines the success of the entire business. A new survey out by the Harvard Business Review and Oracle this week shows that IT responsiveness stands as a huge arbiter of overall business agility, which starts with CIO leadership and trickles down into the way organizations leverage processes and technology. The results can be seen as a testament to why DevOps practices are gaining momentum in so many organizations, as respondents report a nearly one-to-one relationship between IT’s ability to react to business demands and overall business performance.

    “Leaders are focusing on speed and broadening their view of where IT begins and ends,” the report explains. “They’re rethinking roles and relationships—and their own value proposition. And they’re making sure their company doesn’t miss opportunities because the IT department was too slow to respond.”

    However, many companies are experiencing just this pain. Nearly half of respondents reported that their companies have missed opportunities because their IT department was too slow to respond to shifting business needs. And approximately 49 percent reported that a lack of bandwidth in IT is the primary obstacle of the organization to take advantage of digital technologies.

    Conversely, when IT organizations are able to quickly move on line-of-business needs, bottom line success follows. Among those respondents who identified their companies as having responsive IT organizations 42 percent strongly agreed that their IT departments were highly successful at leveraging new technology for business advantage. Only 10 percent of those with moderately responsive IT could say the same.

    This most recent finding bolsters another study out this year by Puppet Labs, which showed that high-performing IT teams using DevOps were more than twice as likely to exceed profitability, market share and productivity goals over more traditional IT shops, and they also had 50 percent higher market capitalization growth over three years.

    The advantages of IT agility comes in many forms according to the Harvard Business Review study. For example, 68 percent of high performance IT organizations reported new technology had an impact on customer engagement, 64 percent said the same about work processes, 62 percent said new tech impacted product offerings and business models, and 59 percent reported new technology positively impacting core business operations and processes.

    According to the survey, performance is highly dependent on IT’s ability to prioritize investments, a capability that a third of organizations do not have. Similarly, collaboration within IT and with line-of-business staff is another bellwether. The survey showed that only 21 percent of respondents believe that the IT organization alone primarily drives digital business innovations. Aproximately 40 percent say that it is actually an equal mix of IT and other business input, while 34 percent say the business primarily drives the technological innovation. However, organizations have a ways to go on the collaboration front. Approximately 42 percent of organizations report that they have a poor track record of facilitating collaboration across IT and business functions.

     

  • Q & A: Puppet Labs CIO Nigel Kersten

    Q & A: Puppet Labs CIO Nigel Kersten

    We recently spoke with Nigel Kersten, CIO at Puppet Labs, regarding high-performance IT and trends he sees in continuous integration in his role at Puppet Labs. Nigel joined Puppet Labs from Google HQ, where he was “Puppet Master” and responsible for the design and implementation of one of the world’s largest Puppet deployments. Kersten is also an experienced Linux and Mac deployment sysadmin.

    In case you aren’t familiar, Puppet Labs provides IT automation system configuration and management software, which is used by start-ups to very large enterprises. Here is our discussion with Nigel, where we talk the recent Puppet Labs DevOps report and high-performance organizations.

    devopsy.kinsta.cloud: In the 2014 State of DevOps Report, there seems to be a relationship between high-performance IT and high-performance companies. Can you see the causation and what correlation is there? Is it high-performance companies driving high-performance IT, or is it high-performance IT driving the high-performance company?
    Nigel: At the moment, it’s a correlation in terms of the amount of data we have. What the data actually show us from the survey is that adopting DevOps practices leads to a higher performing IT organization. This means you’re releasing code more frequently, your recovery time for failures is much, much faster, and all those sorts of things that we know go into what makes a great IT organization.
    We’ve actually been able to correlate that those organizations with higher performing IT teams are outperforming their competitors in the market, and they’re exceeding their own corporate goals. Now, at the moment, we’re just being a little bit tentative in saying that that’s a correlation rather than causative.
    To be clear, statistically we have a really high degree of confidence that it’s actually a causal relationship, not just a correlation. But we’re not quite ready to make that claim yet.

    devopsy.kinsta.cloud: We wonder how you’re seeing continuous integration change the relationships among security teams, application teams, operations, and QA. What are you seeing, broadly, when it comes to how these relationships are changing from the way they were several years ago?
    Nigel: I think that a couple of things are changing those relationships. One has been the slow death, I would say, in some ways, of waterfall approaches in favor of agile. That’s not to instantly label agile as good and waterfall as bad, because I actually think there are some development environments and problem spaces where a really strict waterfall-based approach is the right way to do things.
    But we’re definitely seeing a mass of organizations that are moving to a more agile setup. I think what this has meant is that the demands on those continuous integration systems and QA teams have changed quite a lot. Developers expect much more self-service, and they expect to be able to operate with a much tighter feedback loop. They don’t want to have to commit code and wait till the next day – until a 12-hour set of tests has finished running – before they find out whether their code is any good.
    Prior to this, I think what we saw for a couple of years was developers starting a sort of shadow QA on their own using Vagrant, using local virtualization. But definitely in the last year or two, we’re seeing a trend where QA teams are being forced to provide faster feedback and more self-service to the development organization.

    devopsy.kinsta.cloud: That must give heartburn to those who are used to having strict, steady, sequential processes in place.
    Nigel: It’s interesting, isn’t it? Some of this is a little speculative, because a lot of security folks tend to be very forward-thinking people, and I don’t think that’s necessarily true of everyone in the security industry, just like every industry. But I think it’s forcing people to look at their current security practices and work out what is just security theater and what actually improves the security of the product, or whatever it is that they’re shipping.
    So how can we automate those things that are actually improving security so that security doesn’t become the bottleneck in these fast feedback loops. agree that it can definitely give security people heartburn, especially people who are really used to rigorous code change approvals and the sort of segregation of duties that you put it in one of your pieces.
    What I see is a common compromise here, and I actually think this is a really effective way to do it. For certain environments, you really want to maintain what goes into production and have a strict change approval process. But it should be a fast and sort of appropriately light process.
    We’re seeing people giving developers the ability to run tests, change the testing environment, and have a significant degree of autonomy over the whole stack in pre-production continuous integration. People are focused on giving developers the ability to raise classes of machines, modify the tests that are running on them, changing all of those things so that they get a really fast feedback loop. But then when it comes to the actual merge to production, that’s something that your existing type change control process can take care of.
    This means that we see a lot of people segregating their infrastructure into dev, test, and prod, as far as continuous integration goes.
    And the developers do a lot of dev work locally on VMs. When it comes to test, that’s often on a shared infrastructure that, using something like Puppet, can be configured to be exactly the same as the production system. And that’s good enough for the developers. They’re getting their tight feedback loop. They’re testing code in pretty much the same way as it’s going to be tested in the official continuous integration pipeline, and yet you can still manage tight change control process around the merge from test to production.

    devopsy.kinsta.cloud: And if it works as planned like that, you would think you’d get improved code outcomes because the developers are working much closer to any defects.
    Nigel: Absolutely. And I think that’s the whole promise that this sort of system is meant to provide. I think it does fall down if you don’t have a very tightly disciplined development organization, and it’s just open season on the production infrastructure. In that case, it actually becomes counterproductive, because the feedback loops become longer and longer. Then, if the test fails, you’re never quite sure if it’s due to someone else messing with the system that’s telling you whether it passed or not.

    devopsy.kinsta.cloud: It sounds like they are self-correcting problems over time, because security often uncovers “hidden” or latent problems that you don’t see until something bad happens. But if it were a full onslaught on production, then I would imagine you would have other issues with availability and performance and related quality issues would mount up very quickly.
    Nigel: Absolutely. I think this is one of the reasons why people drive a lot of this work to the cloud, and I’m speaking from experience here, because the sys ops team I manage actually manages the infrastructure for the continuous integration system here at Puppet Labs. We have a pretty complicated testing matrix, and clearly security is a really big concern for us, because Puppet in many ways has the key to the kingdom when it’s deployed on someone’s infrastructure.
    So it’s really hard for a small company to provide enough internal hardware infrastructure to allow for that sort of elastic peak consumption. I think this is one of the big drivers toward people, at the very least, using the cloud to smooth out their internal hardware requirements so that, at times of peak capacity, they can essentially burst out to the cloud.

    devopsy.kinsta.cloud: Organizations that move this route, I assume, need to get to know their own processes very well before they move from what they were doing before – more manually driven processes – to continuous integration?
    Nigel: I think what we tend to see is that it ends up being on a project-by-project basis. If you’re a company that sells on-premise software or software as a service, you’ve got a reasonable amount of investment there, and it’s rare that companies can take the time to just pause the delivery of software. Usually, they need to keep making money; they need to keep shipping releases.
    So, what we tend to see is that people start experimenting with green field environments. This, according to my gut, is often enabled by people moving to more service-oriented architectures, where rather than shipping large, monolithic applications, they’re shipping components that have well-defined boundaries with each other.
    As part of that refactoring goes on, as they build a new component, they might do the testing and continuous integration for that component in a fundamentally different manner, and sort of step through their code base that way.
    Eventually, you have to deal with the brown field deployment. But, ideally, by that point you’ve gotten a whole bunch of huge wins from the green field continuous integration systems that make that sort of return on investment obvious.

    devopsy.kinsta.cloud: I would imagine that knowledge gained from those groups is then shared with small workgroups that experiment and then there’s a lot of knowledge sharing and expansion organically from within.
    Nigel: Yes. And with people being people, this is often where turf wars strike up. The have been a number of times when we’ve sort of walked into organizations and talked to the QA team and the developers and the operations people – and dev and ops have started working together and they can ship something much more quickly by doing things in that way – and yet somehow the QA team feels slighted that it’s somewhat bypassed by this whole process. That can be a rather complicated conversation to be in the middle of.

    devopsy.kinsta.cloud: Do you see companies make common missteps when they move toward continuous integration deployment?
    Nigel: One of the more conceptual mistakes I see people make when they’re considering it is that they confuse continuous deployment with making lots and lots of public releases, and they’re not necessarily the same thing. I often try to tell people that continuous deployment is really about enabling the ability to choose to ship something. It should be a shippable unit that shouldn’t necessarily be shipped, if that distinction makes sense.
    A lot of companies will start stressing out. They’ll wonder about their existing processes and educating their channel resellers. They’ll believe they simply can’t afford the cost of a release every two weeks, or every week. But that’s not necessarily the whole point of continuous delivery. By having artifacts that are shippable, you enable testing internally, you enable easier experimentation, and you still can choose to ship your product only once every six months if you want. But the fact that you might have 20 releases in between those public releases is important. They could all be shipped and could be a bug fix for a certain customer, or enable you to do certain types of development in the future. These are really huge wins, even if end users never actually see those specific releases.

  • Microsoft bridges the gap between Azure and DevOps

    Microsoft bridges the gap between Azure and DevOps

    DevOps can be easy for a startup. Many of the concepts and principles of DevOps come quite naturally to a fresh company just getting started. It’s a different story, however, for large, established enterprises trying to wrap their arms around this DevOps thing. For IT admins working in Microsoft’s Azure cloud platform, though, things just got a lot easier thanks to the integration of Chef and Puppet Labs.

    Many organizations are heavily invested in the Microsoft ecosystem. IT teams and administrators are familiar with Windows domains, Active Directory, Hyper-V, and Azure. They know how to write scripts and leverage PowerShell to get things done, but they’re much less familiar with the core principles of DevOps, or common DevOps automation tools like Chef and Puppet Labs.

    One of the driving forces of DevOps is the ability—or perhaps necessity—to automate those tasks that can be automated so IT resources are freed up to focus on more important tasks. Deploying and configuring virtual servers and applications is necessary, but it can also be tedious, and it is a task that can be easily automated using DevOps tools.

    The arrangement also works the other way. Smaller companies that are already familiar with, and invested in DevOps may need to scale to a platform like Azure. They should be able to build on what they’ve started, and continue to use the tools they’re familiar with as the company migrates to Microsoft’s Azure.

    One of the main attractions of Microsoft Azure is the degree of flexibility, choice and control it gives customers. Microsoft Azure’s partnership with Puppet and Chef allows customers to choose from an array of existing technologies and communities to easily deploy, configure, automate and manage their VMs.

    According to a Microsoft spokesperson Puppet Labs and Chef complement and extend Microsoft technologies such as Resource Groups and PowerShell Desired State Configuration (DSC) to offer more options for customers to migrate existing or deploy new applications to the cloud. The partnership offers Microsoft the ability to support Linux and UNIX environments, and also alleviates the concerns that organizations already using Puppet or Chef will have about abandoning their investment. Now, they can continue to use their Puppet modules or Chef recipes in Azure.

    Luke Kanies, CEO of Puppet Labs believes the integration of Puppet Labs and Azure will help Microsoft IT admins by making it easier to deploy applications across any platform or cloud environment. Kanies explained in a press release, “Our goal is to help you automate the configuration and ongoing management of your machines and the software running on them, so you spend less of your time firefighting and running simple scripts across a thousand machines, and more time getting great software in front of your users.”

    I spoke with Kanies about the difference in dynamics between a small company and a large enterprise. He told me that in a small company it’s much easier to communicate and coordinate resources, and the cost of failure of any particular project is much lower. For a large, established enterprise, however, there is assumedly a large base of existing customers, and there is more fear and worry about possibly disrupting normal operations.

    I also had a chat with Ken Cheney, VP of business development for Chef. His philosophy mirrors a lot of what Kanies described. Cheney told me that emerging companies have the luxury of starting from ground zero with a clean slate. There aren’t yet any established processes to navigate. The challenge is trying to transform an organization with established processes, and a team of existing developers over to a DevOps mentality. Integrating DevOps into Azure helps organizations to meet that challenge.

    DevOps adds an element of speed and simplicity that makes it easier for larger enterprises to be more agile. Using tools like Puppet Labs and Chef enables IT admins to add speed as a core requirement for infrastructure projects without disrupting or conflicting with other requirements—like security and stability.

    Ultimately, the new alliance illustrates Microsoft’s commitment to empower Azure customers with the best-in-class configuration and automation tools by embracing the technologies, communities and ecosystems that developers know and love. Combining Azure with powerful and popular platforms like Chef and Puppet Labs bridges the gap between traditional IT and DevOps—enabling old school IT admins to embrace DevOps, and DevOps startups to mature and evolve into larger enterprises.

  • DevOps Dirty Little Secrets

    DevOps Dirty Little Secrets

    By all outward appearances DevOps is gaining real velocity within IT shops worldwide, Duh tell you something you don’t know. I have read much on this topic as I am sure you have, but I never seem to read much with hard data results, other than we are doing great with some very broad  metrics. What are the hard numbers as compared to the legacy IT shop?  So my search begins; where in the world are devops metrics in Fortune 500/Enterprise organizations?

    What got me thinking about this were a couple of recent AppDynamics blogs “Measuring your devops success” and   “Quantifying the value of devops”.  Both blogs do an excellent job at breaking down DevOps to the implementation level. Hard data creates metrics that set a baseline with the legacy system that is the critical measurement component, without a baseline there is nothing to compare or measure against. This approach creates a meaningful way to measure the success of executing a DevOps strategy within an IT shop.  The blogs also point other factors not quite as easy to measure, particularly human elements. Many others have written on measuring DevOps, you need to adopt the approach that makes sense for you and your organization.

    At that point I figured it was time to visit Puppet Labs, when you think DevOps Puppet Labs and their survey’s comes to mind. I utilized the 2013 State of DevOps Report; the 2014 report is not available yet. I believe that in order for any new philosophy to be adopted as a true standardized practice, the C-level in Fortune 500 organizations need to embrace it. In my long experience, typically the first sign of acceptance in traditional Fortune 500 organizations is when the financial vertical embraces it.  As an example, Wall Street traders will pay good money to save two milliseconds on a trade.

    The Puppet Labs report numbers demonstrate real velocity in the adoption of DevOps as an accepted business practice. The first takeaways I had from the report were that organizations ship code 30x times faster with 50% fewer failures. The numbers paint a glowing picture for DevOps. But looking beyond the glowing numbers, I see something else. DevOps is at a critical fork in the road and needs to cross the chasm before it loses its luster and becomes just another application development tool, as opposed to a practice.

    Only 17% of the respondents were Managers or C-level, with only 22% of respondent companies having 10,000 employees or more.  It is essential to look at individual Fortune 500 companies IT shops and measure them with hard data before and after the implementation of DevOps.  I would love to read some in depth case studies of the Fortune 500 with hard numbers and metrics; I did not find anything with the legacy baseline data. A huge benefit of DevOps to the Fortune 500 is it will enable them to fail fast as opposed to continuing an implementation that will inevitably fail anyway and consume a huge amount resource in the process. Anyone who has worked for a Fortune company I am sure has witnessed this.

    In truly understanding what DevOps is and is not, certain questions need real answers; Are Enterprise/Fortune 500 companies really doing the same old thing and just need to write better code utilizing some new application development tools, does that make it DevOps or just continuous improvement? Will some IT execs change the organizations title from sysadmin to Devops Admin to protect their silo and make them seem kinder gentler Ops? Are the DevOps tools available today ready for prime time in the Fortune 500 IT environment? Is Devops about technology and metrics or really about culture? We can speculate, but time will tell us the real answers. I am looking for good metrics on what does DevOps do to a Fortune 500 company Not some so called Unicorn that was born with a DevOps spoon in its mouth, but a good old fashioned Wall Street firm who adopts DevOps. Until then call me the DevOps skeptic.

    If you would like say something; email parker@devops.com and don’t forget to comment here.