Tag: automation

  • Consumer Web Automation for DevOps?

    Consumer Web Automation for DevOps?

    These days event-based automation is everywhere from IFTTT recipes used by non-technical folks to cross post automatically to Twitter, Google+ and Facebook to email alerts for free iTunes downloads. And don’t forget the tech savvy that create even more complex sequences from Zapier and now Stamplay to track workloads and simple business processes or automate developer tools.

    But this automation is not limited to consumer uses—it can also be the answer to DevOps challenges. Simple DevOps processes implemented with services such as IFTTT and Zapier improve the workflow and boost work performance.

    (more…)

  • The Dark Side of Infrastructure as Code

    The Dark Side of Infrastructure as Code

    We spend a lot of time extolling the virtues of infrastructure as code. All of it is true. Treating infrastructure as code can add great value in terms of its ability to promote consistent, predictable and repeatable results during the application deployment process. One of the ways in which infrastructure is treated as code is through the use of programmability.

    Programmability is an integral component in software defined architectures and a part of the DevOps approach is automation through integration with traditionally development lifecycle focused tools and technologies. APIs, app templates and the ability to programmatically interact with the data path are enabling rapid provisioning of services in the network. That’s important to the business. Improving service velocity in the network is critical to achieving faster time to market and enabling the operational scale necessary to meet growing pressure on networks arising from the growth of applications driven by cloud, mobile and the Internet of Things.

    Speeds and feeds of processes are now as important to gain a competitive advantage as speeds and feeds of packets.

    Those who see DevOps and SDN as having a strategic impact on their organization view programmability as having even greater importance.

    But that does not mean that treating infrastructure as code doesn’t have its dark side. Because adopting a programmatic approach to anything means you get both the benefits and the risks.

    I happened to be cogitating on this topic when I stumbled across a post reminding us of the importance of code reviews. The content was so-so, but the premise was spot on: code reviews are a must for those embracing the notion of treating their infrastructure as code.

    Why is that (or should it be) an imperative?

    Consider some stats from InitialState, “a technology startup helping engineers capture and understand product data. Our cloud-based log visualization-as-a-service platform helps engineers make sense of the exploding number of logs being produced by “smart”, complex products and applications.”

    According to their data, developers have to find and fix 100 bugs per 1000 lines of code. And it takes 30 times longer to fix one of those bugs than to write one line of code. Overall, developers end up spending 75% of their time (about 1500 hours per year) debugging code.

    Now spend a moment thinking about what that means when you start treating infrastructure as code, writing scripts to automate provisioning and integrating with other systems to orchestrate deployment processes.

    2011.09.18_code_reviews

    I won’t lie, code reviews can be tedious. But the first time someone points out an error, particularly one that could have resulted in a security breach or an undesirable service outage, the pain is more than made up for. In the long run, code reviews are beneficial to eliminating disruptive errors. A SmartBear survey in 2012 found that “70% of organizations indicated that code reviews are very important” and “Those that do code reviews clearly have a greater level of satisfaction in the overall quality of their software – they were twice as likely to indicate that they were very satisfied with their software quality.”

    In other words, code reviews are an important part of the software development lifecycle (SDLC). Which means it should be a part of the software deployment lifecycle when it leverages programmability to treat infrastructure as code.

    Therefore it is important in the early stages of DevOps (or SDN if you’re going that way) adoption that you put into place processes to check (and perhaps double check) the scripts, templates and integrations you’re using to automate application and service deployment along the entire operational chain – from the app to the network.

  • Right-Picking the First Project to Go DevOps On

    Right-Picking the First Project to Go DevOps On

    One of the hardest things to do when starting a new initiative is picking the right project to work on. Should I start with App A? App B?  Does it really matter as long as I get started on this DevOps thing and prove its value to management?

    Yes, actually, it might.

    If you think about it, one of the reasons you’re doing DevOps in the first place is not just because it’s cool (it is, of course, but that rarely flies with the business guys) but because it’s going to add value in one or more ways by improving time to market, reducing risk and lowering operating costs. If the project you choose to go DevOps on isn’t capable of showing those benefits (or isn’t capable of showing those benefits in a reasonable time frame) then you might just be choosing the wrong one to start with.

    So how do you choose? Well, how about starting with an app that’s being developed using agile methodologies?

    According to the “8th Annual State of Agile” survey from VersionOne 88% of respondents were practicing agile development, which means you shouldn’t have a hard time finding one.

    Except, you might if you aren’t paying attention. A high adoption rate doesn’t necessarily mean a high saturation rate, however.

    number of projects using agileDig a bit deeper into the survey and you’ll find that agile isn’t used for every project. On the contrary, there’s a good distribution of agile and traditional (waterfall ) methodologies being used. While over half of the respondents are using agile to manage more than 50% of projects, the other 48% are using some other methodology.

    Which offers some interesting strategies for how to get started with DevOps on the other side of the wall (that’s you, in operations land).

    On the one hand, the projects most in need of improved velocity in deployment (time to market) are likely those being developed using agile methodologies. The aforementioned survey validates that approach, with the top reason cited for adopting agile being “accelerate time to market.” The general premise of agile is fast, frequent updates, making the need to deploy into production a more pressing priority. Choosing an app that’s being developed using agile methodologies means you’ll be able to practice, refine, and optimize the processes as you work to automate the deployment. Because of the frequency with which agile-developed projects tend to need deployment, this gives you the opportunity to practice similar techniques on your automation and optimization.

    You’ll be able to automate individual tasks and test them thoroughly before moving on to the next one and the next one and the next one, until you’ve got the process orchestrated (yes, Virginia, there is a difference). You’ll then have ample opportunity to optimize the process, finding redundancies or inefficiencies in the way information is gathered and disseminated into the process. Optimization of the process means greater efficiency, especially if you can knock out those all important wait times between tasks.

    The key here is that you have to know which projects are being developed using agile methodologies, and which aren’t. That means communicating with developers. And not just by sending an e-mail and asking for a list, but really talking to them to understand which projects the business would benefit from improving time to market. Getting an app that is the vehicle for competitive advantage to market faster seems a bigger return on investment than the app used by just a few people for non-critical purposes. You’ve got to dig in and determine which agile projects need agile deployment the most, and then decide where to dig in.

    This is not to imply that less frequently deployed apps will not benefit from agile. They most certainly will, though the benefits that DevOps brings to those apps are more about stability and consistency than they are those derived from improving time to market.

    What’s important is to decide what your goals are with DevOps and what benefits will be worth the most bang for the buck (cause it ain’t going to be cheap in terms of time investment). Getting DevOps on an infrequently but mission-critical app can reduce the risks associated with rusty processes and improve stability in general. Automating the deployment  of such applications can certainly save time and resources that can be redirected to agile projects in need of greater deployment velocity.

    But if you’re looking to show value and return on investment, you’ll likely want to choose one of the agile-developed projects that already exists and needs some attention. Not only because it’ll show value faster, but because it’s releasing frequently enough that you’ll actually get the opportunity to refine (and one hopes optimize) your automation and orchestration.

  • Time and complexity; or, Why are some things hard to do?

    Time and complexity; or, Why are some things hard to do?

    Why are some things hard to do? If we want to make complicated tasks easier, which is one of the goals of DevOps, we need to have some theory of why they are difficult in the first place.

    Richard Garfield, a mathematician and game designer, presents two factors that make tasks more difficult: time constraints and complexity. Our ability to solve a problem depends on the amount of time we can spend working on it and the complexity of the problem.

    Time and Difficulty

    DevOps discussions generally focus on time as the relevant dimension. Business value is measured in time: time to market, time to recover, and lead time for change are all important metrics for DevOps. Doing the same amount of work in a smaller amount of time is better. But an emphasis on time can also be misleading. Allotting less time for a task isn’t the same as truly being faster.

    So what makes something take more or less time? Part of the difficulty of a task is relative to the person doing it. Provisioning a new server might be straightforward for an administrator, challenging for an application developer, and impossible for a graphic designer. Hiring good people and matching them with the right tasks reduces the difficulty of the task and the time required to complete it.

    Complexity and Difficulty

    But regardless of who’s doing it, the task to be done is the same. The specifications have an objective existence outside of whatever person is doing the work. The objective part of a task’s difficulty is its complexity.

    The parallel between time and complexity is evident in the thought experiment Richard Garfield proposes. Imagine you are standing in front of two doors. One has a dragon behind it, the other has treasure. Each door is printed with a math problem that reveals whether the door contains a dragon or treasure. If you have a very short amount of time to decide, you will only be able to solve a fairly simple problem before being forced to guess which door is safe. Similarly, if you have a lot of time but must solve a very complex problem, you will again be reduced to guessing. Either insufficient time or excessive complexity can force a person to guess in a situation where a knowable answer exists.

    Automation and Complexity

    DevOps typically aims to reduce complexity through the “automation” leg of the CAMS definition. Now that we have examined the problem more broadly it’s clear that automation is a subset of the class of “things that reduce complexity.” Before automating, or if automation efforts at your organization have yielded disappointing results, a good check would be to ask “will this reduce the complexity of deploying code?”

    Automating without visibility into what/how you are automating will move complexity around but it won’t reduce it. You’ll be able to spin up new builds really quickly—once you’ve tracked down the person who pushed the latest version and forgot to mention the three little configuration settings you need to change for it to work. Configuration drift or pockets of automation reintroduce complexity. Without visibility and monitoring, these problems grow on automation systems like mold on untended fruit.

    This is not to say that automation is bad. When done in a mature environment with testing and visibility, automation is the greatest tool for reducing complexity. But without independent validation and monitoring, you’ll find that automation can also be one of the biggest sources of complexity.

  • My Little Operations: DevOps is Magic

    My Little Operations: DevOps is Magic

    Bear with me, it will all make sense in the end.

     

    If you aren’t familiar with My Little Pony: Friendship is Magic (because you don’t have a child who watches it over and over and … well, over)  let me sum up for you:

    There’s a pony who knows something horrible is coming. She wants to stop it. She’s instead sent by management to make friends, which seems to her to be ridiculous in the face of a looming catastrophe. When that catastrophe happens, she and her new “friends” (her coworkers) set off to find the elements of harmony – the only thing that can save the world. But lo and behold, evil destroys the elements. Or so it appears. See, it turns out that the friends are the embodiment of the elements of harmony and through their friendship they create the most powerful element of them all: magic. Together, they are strong enough to defeat the evil and restore harmony to the data center. Err, I mean, to the pony world.

    If we take this and apply it to operations, you should start seeing some common themes. Primarily, you might note that, like magic, the ultimate goal of DevOps – agility – can only be “achieved” by the collaborative efforts of the other elements of operational harmony, namely predictability, repeatability, composability, consistency and stability. It isn’t a tangible “thing” you can buy, it doesn’t come in a can, it can’t be delivered by Amazon.

    These elements combine and, when practiced cooperatively, create a new element: agility. Which is a significant goal of DevOps operational harmonybased on a variety of surveys. Oh, it’s always stated a different way: we want to improve the time to market, increase frequency of deploys, reduce downtime due to errors, etc… but in general, these all fall under the need to enable the same level of “agility” in operations (and the network, but that’s a different discussion) as is present in application development today. Fast and frequent change to match application (and business) changes. Agility.

    But agility, like magic, can only be achieved through friendship (or as we’ll call it, collaboration). That’s because agility requires a variety of separate concerns (qualities) to come together in concert before it can be realized:

    Agility (DevOps)

    • Predictability (time to deploy, important for planning)
    • Repeatability (of process to other applications)
    • Composability (reuse of steps in other processes)
    • Consistency (important for maintainability and trust in process)
    • Stability ( of the overall infrastructure )

    That’s why it’s important not to look at DevOps as a set of tools. There are multiple facets to DevOps that can be enabled by tools and enhanced by tools and even monitored by tools, but DevOps is not the tools themselves any more than the elements of harmony were the gems the Little Ponies sought. The real magic of DevOps is the agility created by collaboratively working together to bring each of the elements into operations. Without stability, have you achieved agility? Or is it really chaos?

    If you can’t repeat the process (apply it to other applications) have you really achieved agility or just strictly codified a single deployment? If the process isn’t consistent, and gives different results have you realized an agile environment or anarchistic one?

    These elements are necessary to achieve real agility of operations, and those elements can only come about in a cooperative environment in which they all work together.

    DevOps, so it seems, is magic after all.

     

  • Provisioning versus Configuration Example

    Provisioning versus Configuration Example

    A few weeks back I wrote about the difference between configuration and provisioning noting, primarily, that there are differences between the two tasks. It remains an important distinction to make because it’s really where the rubber meets the road (or the app meets the network) where it becomes important. As the infrastructure/network side of the house is where we’re concentrating our efforts to apply DevOps, it’s necessary to dig a bit deeper on this topic.

    So today we’ll examine  couple of concrete examples of the difference – and why it’s important.

    Your application is being deployed and it needs, wonder of wonders, a load balancer. Cause, elasticity. Scale. You know, cloudy type stuffs.

    So you grab that golden image and launch that virtual machine with Whatever Your Favorite Load Balancing Thing might be (HA Proxy, Nginx, F5, etc…). Voila! You’re ready to go, right?

    Not by a long shot. You’ve provisioned a load balancing service, yes. But is it configured? Probably not. In addition to the basic networking needed (IP addresses, VLANs, etc…) there are also a whole bunch of very application specific things that must be configured. There’s no “golden” load balancing service that can be generically applied to your application. Period. At a minimum the virtual server (that represents your application to the rest of the world) needs two things: an IP address and a pool of app instances to select from.

    simplified-lb

     

    That means there’s some configuration that must occur after the provisioning. There’s no single LB image that will work for all applications, period. Even if all other settings are the same (LB algorithm, L7 app routing rules, TCP options, etc..) you still have to tell it which pool of real servers to select from and what IP address to use so clients can connect.

    This is true for pretty much all application services (infrastructure) that support an application deployment. Because there’s no single golden image that just “works” for everything and no way to use dynamic configuration options for everything (DHCP for IP addresses is one thing, but there’s no “dynamic load balancing algorithm selection” service out there yet) there remain a set of options that must be configured after provisioning occurs. Consider application security like a web app firewall (WAF). It has to be configured to protect a specific application. It has to learn the URIs and the data expected to be exchanged so it recognizes when some interaction is out of bounds and potentially dangerous. That’s unique to the application, and thus it requires some specific configuration after provisioning.

    It’s important to note this distinction lest we get caught up in believing that we can simply create a golden image of all our network infrastructure and services and put them into production with the touch of a button.

    This little reality also helps to understand why it doesn’t really matter whether the network service in question is launched on custom hardware or some server off the shelf. The bulk of the work when automating the deployment of network services is in the configuration, not the provisioning.

    While you can minimize the amount of post-provisioning configuration necessary, you can’t eliminate it entirely. That’s why API-enabled, programmable infrastructure is a critical piece of the larger DevOps picture. It is the API, programmable templates and similar capabilities that enable the configuration to be automated and orchestrated so as to not be the bottleneck in the deployment process.

  • Enabling Collaboration Across Diverse Cultures

    Enabling Collaboration Across Diverse Cultures

    Welcome back to this series, ‘Enterprise DevOps Q&A.’ In my last post, I talked about breaking down the barriers between dev and ops. Now, I want to address another, very typical question from a global enterprise about managing cultural differences in a globally dispersed enterprise.

    Q. Being part of a large org, our biggest issue seems to be a global culture clash. How can we address the issues of geographic location and culture that are a big obstacle for us?

    Large multi-national enterprises, like CA Technologies too, often have major data centers in multiple geographies in North America, EMEA, Asia and Latin America. Despite so much data center consolidation and cloud migration, cross-geo cultural issues remain a major issue.

    For example, in Israel the holy day is Saturday, while in the US and Europe it is Sunday, so it can be hard to lock down a release window. In Japan it is common to work long, late, and on the weekends, while by contrast some French labor agreements specifically limit the workday, so ‘crunch time’ may not have the same meaning in both locales. In the USA, staff may only get ten days annual vacation, and not even take all of it, while much of Europe takes all of July or August off, so consistent teaming can be a challenge.

    To help you address these geo-cultural challenges, here are some approaches I have seen my customers adopt including:

    A local approach

    A local approach, such as co-locating devs and ops to service business users in a common geographic or cultural area allows teams to adopt DevOps practices within cultural norms. This may mean imposing a difficult structural change, but it can also be very effective. See my last Q&A for more on this approach.

    A ‘core team’ approach

    Some businesses have assigned one or two team members from each cultural area to represent their teams in a common virtual or ‘core’ DevOps transformation team. Representatives on the core team can translate (sometimes literally) for their ‘home’ teams, and bring unique culture back to the core team. This can raise process/hierarchy barriers, but when done well can also help to break them down.

    A vertical approach

    I have seen a handful of global businesses implement DevOps vertically, with devs and ops teams, often geo-located, acting as service providers to individual business units, rather than the whole company. This is best suited for organizations that are already organized by business unit, but it ensures teams have common focus on specific and shared business problems.

    A ‘known good process’ approach

    Speed the flow of software delivery by focusing on refining and documenting specific processes all teams can rely on.  Each geographical or cultural team can define and share their own ‘known good process’ (cf. the Visible Ops handbooks), and share the interfaces to those processes, so they can continue to optimize their own activity, but work seamlessly with other teams at handoff points.

    A process automation approach

    Even better is if each team can encode their processes in automation to make it possible for anyone, anywhere, using any language, to execute those processes remotely. Even small improvements matter, so even task automation can help get closer to a cross-cultural, ‘enterprise-wide,’ DevOps approach.

    A ‘pace-layered’ approach

    Different business services have differing levels of acceptable risk and tolerance to change. Mobile app dev, for example, is more tolerant of risk than, say, accounts receivable. Rather than changing the world (literally!), instead build a DevOps approach into a specific service delivery team, even in a single geography. This ‘pace-layered’ approach is working very well for some of the world’s largest enterprises.

    A tactical approach

    Similarly, you do not need to fix all your service delivery problems at once. In a global enterprise, pick one big winner at a time instead. Identify constraints in the test process, a roadblock in QA, a provisioning delay, a problematic release process, a monitoring gap, or a broken feedback loop, and connect cross-cultural staffs from similar functional teams to solve specific shared issues.

    A technology-assisted approach

    Regardless of the approach, technology can assist. Global tools with multiple language support can enable common experience and terminology. Telecommuting can give regional staff a virtual presence in team meetings. At CA Technologies, we are trialing remote-controlled robotics to give staff a physical avatar in meetings from anywhere across the globe.

    Other approaches?

    As with most of DevOps, there is no definitive prescription, and I am sure there are many other approaches that can work. If you have ideas or experience to share, please post in the comments below, and let us all know how you have or could address these cultural and geographic barriers.

    And remember, if you have questions about Enterprise DevOps, let me know and I will try to answer them for you. If I can’t answer them myself, I will find someone who can! Please leave your questions in the comments below; hit me up on Twitter at @AndiMann; or send me an e-mail to me at andi.mann@ca.com.

  • Now that you can consolidate, do not implode

    Now that you can consolidate, do not implode

    The development world is changing in a big way. We all know it. And it usually happens first with the early stage startups, who have the luxury of adopting new approaches from day one. One of those major shifts is consolidation of team members, where automation is taking over previous roles. In the modern development shop for example you are not just a front-end developer, you are a data scientist, a tester, a orchestration master, and a release manager. There are a lot of upsides to this consolidation, but a lot of downsides as well.

    (more…)

  • Enterprises, DevOps and Economies of Scale

    Enterprises, DevOps and Economies of Scale

    In infrastructure there are two economies of scale: operational and capital. DevOps should attend to the former, especially in the enterprise.

    Soon after cloud burst (sorry, pun not intended) onto the scene we all learned about economy of scale because that, primarily, is what cloud computing’s business model is based on. At great volume, standardization can be used to achieve greater output and, thus, can cost less.

    But there’s a second economy of scale at work with cloud; the operational economy of scale. If cloud providers scaled operations in the same way that enterprises traditionally scale operations, well, instance prices would be much, much higher.

    What cloud providers recognized is that in order to achieve the economy of scale required to make cloud a success they had to address both the operational and capital economies of scale.

    infrastructure economies of scale

     

    According to research, surveys and studies, operational overhead consumes anywhere from 63% to 80% of IT budgets. That’s just maintenance, management and general “keeping the lights on” kind of things. And each time a new application is pushed into production, that number gets bigger because there are more of everything – from storage to compute to network – to maintain.

    To wit, Computer Economics “IT Spending and Staffing Benchmark 2014/2015” indicates that 67% of organizations were increasing their operational expenditures versus only 20% that planned on a decrease. Because IT budgets are generally determined as a percentage of revenue, that means operational budgets are chewing up the business, slowly but surely, and leaving little room for becoming an enabler of business rather than an application assembly line.

    The holy grail, as its been described, for IT is to move budget from operational overhead to innovation. To do that means somehow magically reducing the costs associated with operations. Adding more people doesn’t help; that just adds more overhead. The goal is to do more with less; to increase the ratio of systems and services and applications per operator that can be managed. That’s operational economy of scale at work.

    To do that, enterprises need to look to DevOps and its leverage of automation tools to enable leaner, more efficient operational processes. That means not just automating, but enabling a self-service model in which common provisioning and deployment processes can be orchestrated at the behest of the app or business owner. It means applying the same “gold image” standard used for provisioning and managing virtual machines to infrastructure; leveraging templates to gather the requisite input from app or business owners and automatically provisioning systems.

    Limited variations on an infrastructure deployment template  will enable customization where necessary, but otherwise provide standardized (stable) provisioning and deployment processes that can be self-service enabled. By empowering developers, app and line of business owners to provision application required infrastructure services, operations can use technology to scale to meet demand while lowering costs.

    The goal of DevOps in the enterprise is not to enable 10 application deployments a day. It’s to achieve  operational economies of scale and a more consistent, predictable and repeatable operational environment.

  • DevOps by any other name still gets things done

    DevOps by any other name still gets things done

    My version of “A rose by any other name would smell as sweet” is not nearly as eloquent as Shakespeare’s, but the sentiment is the same. Juliet argues in Romeo and Juliet that the name of a thing such as a rose is irrelevant—it’s still a rose no matter what you call it. The same thing is true when it comes to DevOps.

    The term DevOps has achieved industry buzzword status. Most people—at least most people in IT—are familiar with the term even if they don’t have any idea what DevOps is. Even among those who think they know what DevOps is, there may not be consensus on how to define or implement the concept. It is understandable that some people don’t have a clue what DevOps really is.

    The Wikipedia entry for DevOps is a great illustration of the issue. Wikipedia is a crowd-sourced encyclopedia, and as such it is comprised of a variety of contributions. The first few paragraphs of the DevOps Wikipedia entry actually offer up different attempts at a definition. The various explanations don’t necessarily conflict directly with one another, but they also don’t agree completely and reading the Wikipedia entry will leave most people just as confused about what DevOps is or isn’t as they were before they started.

    It’s a “software development method that stresses communication, collaboration and integration between software developers and information technology (IT) operations professionals.”

    The goal of DevOps is “to maximize the predictability, efficiency, security and maintainability of operational processes. This objective is very often supported by automation.”

    It “targets product delivery, quality testing, feature development and maintenance releases in order to improve reliability and security and faster development and deployment cycles.”

    It “aids in software application release management for a company by standardizing development environments.”

    All of these are true, and yet none of them is necessarily THE definition of DevOps. The simple reality is that it doesn’t really matter.

    DevOps is not a single thing per se. It is a culture, and how that culture is embraced or implemented van vary from one organization to the next without altering the fact that both are employing DevOps. DevOps is not a product or tool–it’s a culture that has evolved organically to meet the needs of a more rapid pace of IT. Many organizations are utilizing DevOps processes or tools, and have no idea what DevOps is.

    If your organization has developers who manage operations, or IT ops staff who do development, or developers and IT operations staff who work closely together to get things done, you might be doing DevOps.

    If your organization uses scripts or other tools to automate the provisioning and/or configuration of servers, virtual servers, or entire virtualized environments, you might be doing DevOps.

    If your organization rapidly develops new iterations of an application, or continuously rolls out new features and tools, you might be doing DevOps.

    If your organization continuously monitors for suspicious activity or potential malicious attacks, you might be doing DevOps.

    If your organization makes extensive use of the cloud in order to be more agile, and be able to scale quickly to meet demand, you might be doing DevOps.

    If your organization lacks rigid structure between departments, and functions more like a startup where everybody does everything, and just does what it takes to get things done, you might be doing DevOps.

    If your organization automates simple tasks and functions with technology rather than wasting valuable man-hours doing everything manually, you might be doing DevOps.

    It doesn’t matter if you understand what DevOps is. It doesn’t matter if you call it DevOps within your organization. All that matters is that you take the elements that make sense for you, and adopt them in order to function more effectively and efficiently, and remain competitive.