devopsy.kinsta.cloud

Tag: deployment

  • Container Considerations on Your DevOps Journey

    Container Considerations on Your DevOps Journey

    Containers are things for your things. That is true in the development world, too—containers are things for your software, and, like containers for your things, software containers can help you be organized, efficient and secure.

    Chris Morgan (@cmorgan_cloud), the Technical Director, OpenShift Partner Ecosystem at Red Hat, delivered a keynote at the 2016 All Day DevOps conference. Chris lives and breathes containers and recently discussed how they are transforming DevOps pipelines.

    Chris started with an encouraging figure: DevOps has a 74 percent adoption rate (at least on a path toward full adoption) at the enterprise level. The reason is that DevOps helps solve issues around:

    • Slow service delivery
    • Poor service quality
    • Frequent down times

    However, the needs of development and the desires of IT can often hamper full DevOps adoption, which is where containers can help. Containers allow apps to be easily shared and deployed with:

    • Consistent environment and tools
    • Predictable building tools
    • Faster deployment

    In short, containers can act as a common language between development and IT operations. And when configured properly, the operations team can utilize containers to assemble environments without compromising security.

    The survey that revealed a 74 percent adoption rate for DevOps in the enterprise also shows an 18 percent adoption rate of containers in the enterprise. Why the difference? Containers sometimes can be difficult to implement across multiple hosts.

    If you are gearing up to consider the use of containers across your IT environment, Chris offered up some things to consider:

    • Where do you store the container image?
    • Which container on which host? Which has enough capacity?
    • How do you monitor the health of the containers?
    • What do you do if a container crashes?
    • How can I address scale?

    When you have large apps that span multiple containers?

    • Which containers can access others?
    • Which containers should be deployed together?
    • How do I limit certain containers?
    • How do they find each other?
    • Is there malicious software on any containers?
    • How do I handle persistent storage?

    The bottom line is that you need more than just containers. You need a host, a platform, management and persistent storage.

    Is your organization ready to at least consider containers for DevOps? Chris reminded us, “Technology should not dictate the process. Identify the ideal process first, and then ensure the technology is flexible enough.” Start by finding some low-hanging fruit, assess skills and capabilities on your team and engage in a discovery session.

    Chris’ full keynote went into more detail, notably some solutions to implementing containers at the enterprise. You can watch Chris’ full All Day DevOps conference session (just 40 minutes) or you can view the other 56 presentations from the 2016 conference online and free of charge here.  This blog series is reviewing sessions from the 2016 All Day DevOps, which hosted over 13,500 registered attendees. Next week, look for “Ops Happens: DevOps Beyond Deployment” with Damon Edwards.

    Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free, online Oct. 24.

    — Derek E. Weeks

  • What Happens Post-Deployment: The Rotating Roster

    What Happens Post-Deployment: The Rotating Roster

    DevOps has become that celebrity who’s always the talk of the town but is largely misunderstood. It’s even quoted in the wrong context.

    A few days back, while speaking at a conference, I was interacting with audience and one of the attendees said DevOps is like the backbone of Agile. I think that’s an excellent way of describing DevOps. In an industry where we are still struggling to implement Agile—and even more so to keep people from drifting—is DevOps ahead of its time? Maybe not.

    DevOps is the set of practices necessary to sustain the fast pace of delivery. The best part of DevOps is there are no levels or certifications one needs to achieve. It’s the stack of practices that you pick and implement and reap benefits. The 7Cs—continuous planning, continuous integration, continuous delivery, continuous testing, continuous deployment, continuous monitoring and continuous operations—help maintain sanity, though.

    One teams had to sustain fast-paced releases with a big customer base. We saw our users increase from .1 million to 2.4 million in three years. Agile, continuous integration and continuous delivery are not just fads for us, but needs for survival.

    Using the Rotating Roster Method

    When we talk about continuous delivery but then focus on just delivery, it’s not enough. It’s equally important to work on what happens post-deployment, such as continuous monitoring and continuous operations.

    Rotating Roster is one practice we use in our organization and this has helped us pull together the whole ecosystem of:

    • Resolving customer issues,
    • Feedback system pumping into out technical pipeline/debt for next releases,
    • Knowledge transfer and learning ecosystem to address attrition, and
    • Broken silos.

    Here’s how we do this. First, we define the scope of the Roster. We picked the following:

    • Production Outage
    • Customer issues/P1
    • Ad hoc requests from stakeholders
    • Data extraction (only tasks out of scope of a separate Data Analytics team)

    Next, we publish a schedule of every two to four weeks, designating at least one person to be on the Roster. It’s essential that the Roster rotates to all or most in the team to build a bus factor or a knowledge transfer ecosystem in the team.

    For each request received, we asked the Roster person to log it in JIRA. This was not just for bookkeeping sake, but as an essential step in line with the Roster objective.

    After every Roster period, we sat down and reviewed these JIRA requests for possible targets to:

    • Automate the task or the effort involved, such as account creation requests automated by writing php scripts, for example.
    • Delegate the task to the right team to cut the effort. For example, for a new data extraction we set up SQL slaves and replicate tables on analytics machines, then delegate the task to them for all future instances.
    • Mitigate the reason for the request, such as the root cause of an outage.

    The mitigation/automation steps had to be picked by development team in their Sprints. Otherwise, usually they become technical debt and lay around in the backlog.

    Since the development team was taking turns on the Roster, they understood the priority and necessity of doing the task. Thus, the technical pipeline was chosen and fed actively.

    Another advantage of the Roster was it showed the team the “big picture” and helped catch early trends.

    The Rotating Roster system turned out to be a big success for our teams; thus, we replicated the model through the rest of the organization. It’s a small but essential step to adopting DevOps culture.

    In one definition, Jez Humble explained that DevOps is “a cross-disciplinary community of practice dedicated to the study of building, evolving and operating rapidly changing resilient systems at scale.”

    I could not have put it in better words.

    About the Author / Kiran Chhabra

    Kiran Chhabra is Vice President of Engineering at Infoedge India Ltd. She has 14 years of industry experience designing and implementing highly scalable and available architectures with optimized continuous integration and delivery in technologies including LAMP, Java and mobile technologies.
    Kiran holds a U.S. Patent and has published technical papers. She is certified scrum master and certified scrum professional, and speaks at various Scrum Alliance & technical conferences in India and the United States. Connect with her on LinkedIn and Twitter.

  • The Perfect Combination: Imperative Orchestration, Declarative Automation

    The Perfect Combination: Imperative Orchestration, Declarative Automation

    There are two fundamental approaches of how to design automation processes, declarative (also often known as model) and imperative (also often known as workflow or procedural).

    The purpose of this article is to explain the principles of each, explore a little of how they work, try to highlight some of the strengths and weaknesses and explore how they can be used together. I don’t think one size fits all—based on your requirements and some of what is explored below you may use a combination of the two, even if some of the tooling you use leans toward one way or the other.

    Declarative/Model-Based Automation

    The fundamental of the declarative model is that of desired state. The principle is that we declare, using a model, what a system should look like. The model is “data driven,” which allows data, in the form of attributes or variables, to be injected into the model at runtime to bring it to a desired state. A declarative process should not require the user to be aware of the current state; it will usually bring it to a required state using a concept known as idempotent.

    Idempotent

    What this basically means is that if you deploy version 10 of a component to a development environment and it is currently at version 8, changes 9 and 10 will be applied. If you deploy the same release to a test environment where version 5 is installed, changes 6 through 10 will be applied, and if you deploy it to a production system where it has never been deployed, it will apply changes 1 through 10. Each of the deployment processes brings it to the same state regardless of where they were initially. The user, therefore, does not need to be aware of the current state.

    Maintaining State

    Once we have a model for our infrastructure or applications, any future changes required are made to the model. This means that as a model changes over time—and may have a current, past or future state. This raises the question of how can a model be applied to a target system, how can it know what changes to make? This is not always easy to answer and is not the same for every platform and runtime you are managing.

    State Management

    It’s hard to cover a broad subject like this without making some generalizations—and there are some in this section. There are broadly three methods used to keep an environment or system and its desired state models in line:

    Maintain an inventory of what has been deployed. This where we maintain the state of what has been deployed, and what is in the desired state of a release, and only apply the difference.

    For example:

    1. My release or ‘desired state’ contains database updates: 1.sql, 2.sql, 3.sql, 4.sql, 5.sql
    2. My example test system has had 1.sql, 2.sql, 3.sql already deploy (recorded in my inventory)
    3. Automation determines to only run 4.sql and 5.sql

    Use Case: This type of state management is particularly useful for things such as databases or systems that requires updates from a propriety API where the state of the target cannot always be easily determined.

    Validate/compare the desired state with what has been been deployed. As a simple illustration, we may want to determine if important_file.xml from my desired state deployment exists on the target system and whether it looks the same as the one in my package.

    1. If it doesn’t exist – create it / copy it, etc.
    2. If it does, determine whether it is different
    3. If it is different, how do I update it (converge, copy over, etc.)?

    Use Case: Probably the most widely used implementation is to simply manage the files that many cloud-native runtimes use to store configuration definitions, such as ssh keys for a server or xml files to manage a java runtime such as tomcat. This also can be used to maintain runtimes that are managed via an API; for instance, What is my heap or config item for my WebSphere Server? Is this what it should be? If it is doing nothing, set it to correct desired state value.

    Just make it so. There are occasions when we don’t really care about what exists. The most obvious example here is anything that is stateless, such as a container. There are probably very few example of why you would want to run any automation within a container to change its state; you would just want to instantiate a new one that has the new configuration you require.

    Imperative/Procedural/Workflow-Based Automation

    The fundamental of what often is referred to as workflow, procedural or imperative approach is that a series of actions are executed in a specific order to achieve an outcome. For an application deployment, this is where the process of “how the application needs to be deployed” is defined, and a series of steps in the workflow are executed to deploy the entire application.

    A standard example might include:

    • Some pre-install/validation steps
    • Some install/update steps
    • Finally, some validation to verify what we have automated has worked as expected.

    The often-cited criticism of this approach is that we end up with lots of separate workflows that layer changes onto our infrastructure and applications—and the relationship between these procedures are not maintained. The user, therefore, needs to be aware of the current state of a target before they know which workflow to execute. That means the principle of desired state/idempotent is hard to maintain—and each of the workflows are tightly coupled to the applications.

    The reality here is that the situation is not black and white. Puppet is an example of what is seen as a Declarative automation tool, while Chef is said to be Imperative. Do they both support concepts of desired state and are idempotent? The answer is, of course, Yes. Is it possible to use a workflow tool to design a tightly coupled release that is not idempotent? The answer is, again, yes.

    What are the Benefits of Workflows?

    The benefit of using a workflow is that we are able to define relationship and dependencies for our units of automation—and orchestrate them together. In the example above, we can create a workflow to perform pre-install, installation and post-install steps, perhaps adding additional conditional steps if it’s in a particular environment such as production (disable a monitoring system, additional security configurations, etc.).

    Procedural workflows also allow us to, for example, deploy component A & B on Server1, then deploy component C or Server2 and then continue to deploy D on Server1. This give us much greater control in orchestrating multi-component releases.

    It is often suggested that the choice is between Imperative- and Declarative-Based Deployment. However, these two approaches are not mutually exclusive.

    Workflow for Applications

    One of the issues in the world of DevOps is the lack of a consistent lexicon or terminology. Just ask someone who works in the industry what their definition of an environment is or what the term application means to them. My guess is you will get differing answers.

    To me, an application is made up of components and a release is generally in the form of an application that is created from multiple versioned components. Now, depending on the organization you work in, you may or may not recognize the definition I have given—and if everything in your world is a stateless loosely coupled microservice on a cloud native platform, then this may be the case. If you work  in a more traditional enterprise organization, you will more likely recognize the requirement for multi-component release orchestration, requiring procedural or workflow processes to handle the coordination and dependencies between the different components.

    Model for Components

    When it comes to deploying a specific component, I think it’s hard to argue against the benefits of using a Declarative or model-based approach. The concept of something being Idempotent—I can deploy any version to any target system if it has never been deployed to before, if we want to introduce a new change to an already existing environment or even if we want to roll a unit of automation back—is hard to argue against for its consistency and simplicity.

    Imperative Orchestration & Declarative Automation

    The future of IT Operations is doubtless based on things such as stateless microservices running in fungible containers, using Kubernetes watchers to trigger event-driven automation, much as the future of retail is for Amazon to deliver milk to my IoT-connected refrigerator using a drone.

    That being said, most of us still live in the present and have the need, for some time to come, at least, to address some of the automation and orchestration challenges that exist in today’s IT landscape. With this in mind, a combination of Declarative (or model-driven) units of automation, coordinated by Imperative (or workflow-based) Orchestration can be used to achieve this. An imperative/workflow-based orchestrator also will allow you to executive not only declarative automation, but also autonomous imperative units of automation should you need to.

    My recommendation is that an application workflow defines the order and dependencies of components being deployed. The Declarative Model for a component determines what action needs to be taken to bring a target system into compliance with the model.

    It is hard to underestimate the importance of desired state—it has changed the way we look at system and application updates and drastically improved the visibility, audit and control of IT systems. Any progressive organization looking to move to a continuous delivery/deployment model or implement things such as autoscaling or “deploy and destroy” for development or testing purposes will benefit from the use of declarative/model-based automation.

    However, a model-based approach does not answer all the difficult questions around orchestration. This, in my opinion (for the moment, at least) is best achieved using Imperative, Procedural or Workflows processes.

    About the Author / Dave Sayers

    Dave Sayers is a co-founder and technical services director of MidVision, a provider of Release Automation and Continuous Delivery Software. He works closely with customers and partners on DevOps and Automation initiaitves throughout Europe and North America and is responsible for devising and managing the partner and channel strategy. Connect with him on LinkedIn.

  • Lessons Learned: We’re Still Killing Ourselves with Innovation

    Lessons Learned: We’re Still Killing Ourselves with Innovation

    Since the widespread acceptance of virtual machines (VMs), we’ve been on a racetrack with changing technology, with little opportunity to stop and digest lessons learned. That’s not to say organizations haven’t done a good job of learning from experience and mistakes both; rather, the lessons learned are too frequently obsoleted by The Next Big Thing.

    While I have an abiding interest in several areas of DevOps, we’ll focus on one I’ve spent a lot of time around recently: provisioning.

    All Roads Lead to Rome

    Road signs for Rome
    Image Credit: Melanie Bateman via Flicker

    At this point in time, you can provision to VMs, “the cloud” (note that from an automated provisioning perspective this is anything but homogeneous, but the fiction helps here for a moment), containers, clusters and clustered container managers. While all end up giving you a working machine that has IP access, how each goes about it is different.

    That is a problem for automation, unless a company goes all-in on one of the above. Containers were going to save us from custom configurations based upon target—and largely have—but in DevOps they still leave us with an underlying deployment infrastructure that must handle where that container will land. The infrastructure doesn’t define itself, unless we teach it to, and that is still very much tied to the target. Anyone who has used both AWS and Azure knows that even two cloud environments can look very different to get up and running with automation.

    Server deployment tools are steadily growing their capabilities and are adopting the ability to spin up cloud instances alongside their ability to handle hardware/VMs. That’s a good thing, but it’s still relatively new.

    The thing is, we in DevOps are trying to do the exact same thing in every case: get a server configured far enough that our application provisioning tool (Puppet, Chef, Salt, whatever) can pick it up and get the server populated with our applications. One goal, many paths, and it is a rare organization these days that is taking only one of those paths.

    Plan Ahead, Forge Ahead

    The rate of change is not showing signs of slowing down—the re-emergence of clustering software in the light of containers is proof of that. I’m stoked about the idea of clustering, simply because it can take a layer out of an increasingly complex environment, but it changes the way we deploy yet again, offering new software, configurations and APIs.

    The defense mechanism I recommend is to look at what is available, consider how it does or will fit into your architecture and plan for it. DevOps is the area of IT most affected by these changes, so that’s where the planning should take place. But like all topics DevOps, this requires a lot of communication with Dev and stakeholders to make certain the platforms being considered are the ones that will best serve the needs of the company. But supporting all of the possible deployment mechanisms in a DevOps environment means investment in scripting and apps that do the same job for different targets. While that may be doable today if you are blessed with tons of DevOps man-hours, I cannot say often enough that it is building technical debt. Each of those different deployment mechanisms will need to be maintained moving forward, and while the idea of DevOps is always relatively simple, it has been my experience that the reality means some pretty complex code/script feats are necessary to make it work successfully and reliably.

    So build a list of platforms you aim to support, start working toward that list, then revisit in six months or so to make sure nothing has changed. Considering that the needs of most organizations change continually in light of technical advances, and the markets are changing from those same technical advances, six months is a good timeline for revisiting. Any longer than that and you may find yourself with a list of things to support that isn’t accurate and misguides DevOps automation efforts; any sooner than that and you will find you are shifting priorities too frequently to adequately support large installations.

    It’s About the Business

    In the end, for the vast majority of IT, the goal is simply to support the business in an efficient and secure manner. Find out what your organization needs and then go out and make it work. Our industry is lucky enough to have a metric ton of smart people, so making it work is not the issue; pinning down the goals is.

    And keep rocking it. For every error, there are a bazillion successes we don’t jump up and down celebrating, so do the same with those errors. Learn the lessons and move along. Pretty soon you’ll have fully automated deployment on the newer platforms most-suited to the organization, serving up the apps the organization needs.

    — Don Macvittie

  • How to Avoid Trouble when Deploying Apps to Multiple Clouds

    How to Avoid Trouble when Deploying Apps to Multiple Clouds

    Deployment is where Dev meets Ops. The glue that binds the two is orchestration, which places operations guidelines in the application itself. Here’s how to avoid trouble when deploying your apps to multiple clouds.

    Things move faster when your infrastructure is the cloud. Applications go from concept to production in hours rather than weeks or months. Everything happens at the same time: development, testing, deployment, updates. Your tools have to be as quick and nimble as your virtual environments.On the open-source side of orchestration, two scripting tools dominate–Chef and Puppet. As Tom Nolle explained in a July 2016 article on TechTarget, Chef is considered the more “programmer-friendly” of two, while Puppet traces its roots back to the operations side. In practice, however, the imperative or prescriptive Chef model and the declarative Puppet model are “growing closer together,” according to Nolle.

    Chef and Puppet: Two Paths to the Same Destination

    With Chef, you create “recipes” and “cookbooks” in a programmatic manner; each deployment step is described separately, and the independent recipes are combined into a repeatable app deployment process. By contrast, Puppet has you build a model that describes, or “declares,” what the end state will look like. Puppet models are intended to simplify these deployment descriptions. The Puppet approach is a natural complement to application lifecycle management (ALM) because it accommodates a complete ALM function.Other orchestration tools that take a purely declarative approach are CFEngine and Juju, while Ansible and SaltStack support both declarative and imperative models. The Organization for the Advancement of Structured Information Standards has released the Topology and Orchestration Specification for Cloud Applications (TOSCA), a declarative-based model intended to support both the definition of end-state application deployment and specific modeling of virtual resources and pools.

    DevOps as an ‘Easy Button’ for Cloud App Deployment

    Today, the components of any application may reside in-house, in a public cloud or anywhere in between. Microservices, virtualization and similar innovations have turned the concept of “infrastructure” on its head. Now the infrastructure adapts to the needs of the applications and services running on it rather than the software adapting to the hardware it runs on. DZone’s Mark Jackson wrote in an Aug. 1 article that IT organizations are evolving from builders and managers of infrastructure to “security-driven cloud service brokers.”

    The ultimate goal is to adopt a new approach: a hybrid-cloud stack model based on a programmable, policy-driven infrastructure. In addition to the stack’s infrastructure, management layers are hybrid cloud orchestration and cloud-native stacks that combine to support nearly any type of data-driven app, whether mobile, IoT, hyperscale or vertical use cases.

    Chart - Deploying Apps to Multiple Clouds
    In a next-generation data center reference architecture, a hybrid-cloud stack accommodates deployment of apps whose components may exist in-house or in public clouds at any given time.

    Take the Sting out of Deploying to Multiple Clouds

    Organizations that rely on one single cloud service are the exception rather than the rule in today’s IT operations. Flexibility is the name of the game as companies deploy to public, private and hybrid clouds using a range of services: OpenStack, AWS, Google Cloud Platform, HPE and Microsoft Azure, among others. Cloud Technology Partners’ David Linthicum wrote in a March 28 article that deploying to multiple clouds presents three challenges:

    • Taking advantage of each service’s native capabilities using generic, relatively new DevOps tools and processes;

    • Ensuring your DevOps tools keep pace with the rapid changes being made to the platforms; and

    • Maintaining security and governance when each service’s rules for these vital operations are platform-specific.When deploying to multiple clouds, that same code set—usually teamed with data—has to be marked for the target platform at the point of staging. As the app runs through the platform-specific testing, it is checked by the tool or process to verify it will work with the platform’s native features. While this feature-checking can be automated to a degree, it often entails manual rechecks as potential problems are identified and corrected.

    Workflow - Deploying Apps to Multiple Clouds
    Deploying apps to multiple clouds requires platform-specific testing that may not be possible to automate completely.

    Once the platform-specific testing is completed, the actual deployment to production on each platform can be automated by placing it on a machine instance in a public or private cloud, and then spinning up the “continuous operations” process. The five components of the process are monitoring, management, resource governance, service governance/service catalog and security:

    • Monitoring the application during execution reports any stability or performance glitches when preset thresholds for various subsystems are exceeded.
    • Management acts in response to alerts of potential problems by interpreting the monitoring data, usually via a cloud management platform for resource management, as well as primary application and subsystem components.
    • Resource governance abstracts the performance data received from individual cloud services through a single console, or dashboard, linked to all the target platforms. User policies can be set once and applied to the various clouds. This is the approach taken by the Happy Apps performance and uptime monitoring service, which provides status and real-time visibility across all your apps and business services.
    • Service governance lets you create policies for the use of APIs and services that serve as common services for all the clouds you use, allowing apps to leverage the service on whichever cloud they’re deployed to. A service catalog is used to track infrastructure-level and application-level services in all your clouds as well as on premises.
    • Security attempts to minimize complexity in a multicloud model by reducing the number of security services involved, which usually include user authentication, access management and breach detection.

    The nature of individual cloud services ensures that deploying to multiple clouds will remain complicated into the future regardless of your preferred set of deployment tools. One way to minimize the complexity is to plan your DevOps strategy from the start with a multicloud environment in mind. There will be times when getting an application to pass data seamlessly across many distinct cloud platforms will require a good deal of manual coding.Adopting a service orientation can mitigate the costs associated with deploying to multiple clouds by sharing services among apps. This way, a single application could run hundreds of remote services from any number of clouds. The service and resource tracking may be daunting at first, but microservice apps prove their business value by making developers more productive and opening doors to new sources of revenue.

  • DevOps Days London: The Human Factor

    DevOpsDaysLondon16logo

    Day 2 of DevOpsDays London 2016 kicked off with a presentation from the legendary Gene Kim. The theme that dominated today’s proceedings was the “human factor,” (fitting, as we finished Day 1 with #humanops) nicely started by Gene’s quote from Nathan Shimek: “We need DevOps to make our work humane.”

    The second day delved beyond what makes people happy, posing questions regarding what it is to be human, and why automation is not everything when it comes to DevOps. As Gene said, “DevOps transcends the technology stack we are working with.”

    DevOps aims to supplant pain with joy and help people find autonomy, mastery, purpose and deep satisfaction in their working lives in technology. Werner Vogel, CTO at Amazon, is often quoted as saying: “You build it, you run it.” But Gene took this further as he explained how a developer had described his most satisfying working experience as coding something and deploying it and monitoring it and fixing it, too. He also touched on the power of shifting left, using Test Driven Development—and its demand that we write the test script in before we even start to code—as an example. That’s building quality in. That’s cross-skilling and cooperation.

    It resonated with the audience when Gene said there was one single question that, when answered, tells us everything we want to know about an organization’s DevOps maturity: “To what degree do you fear deployments?”

    We came back to this question later in Jeromy Carriere‘s talk about his experiences of scaling at Google. Going the gym for the first time is painful—we have to build muscle memory, he said. We need to practice these processes until they are like breathing.

    Gene made a book recommendation:  “The High Velocity Edge,” by Steven J Spear. When we do DevOps we are aiming to transform into a dynamic learning organization. We should not expect but be required to talk about problems. No fear, no blame. Everyone in an organization should be trained to run a blameless post-mortem—or a healthy retrospective, if you prefer. As Claire Agutter put it: “Improving daily work is more important than doing daily work.”

    Gene then outlined the DevOps Enterprise Summit London event coming up at the end of June and the extraordinary stories of experiences people share about their DevOps journeys during DOES events. He said: “Often the transformation leader was putting themselves in personal jeopardy.”

    I can testify to that. It takes courage, optimism, conviction and tremendous amounts of energy to be a change agent.

    The human theme continued into the panel discussion as the COO of Technology at Barclays explained how its command and control structure, waterfall-ruled development and volume of changes resulted in total gridlock. He said that the big challenge is trying to re-engineer the organization to behave differently: “Changing HOW we do our work is the hardest thing.”

    Changing human behavior is very challenging indeed; as Joanne said, measurements have to change and we have to focus on customer value. Gareth Rushgrove also majored on the human angle in his talk on DevOps and microservices. He said: “Microservices are as much about people as technology,” and, “Which comes first: the software or the team?”

    Essentially, DevOps and microservices help us to make lots of small bets—spreadbetting, if you like—spreading risk and accelerating feedback. And we need small teams to be able to do microservices and move away from monolithic applications.

    One particular Ignite talk stood out from the rest, and won the hearts and minds of the audience: @coldclimate from Server Density. Through a personal story of a back injury brought on by prioritizing work over looking after himself, the #humanops message: “You are not immutable, you are not highly available!” struck home hard.

    So there we have it. My key takeaway from DevOpsDays London 2016? DevOps is less about technology and automation and more about happy, healthy humans.

  • Ultimate Checklist for Successful App Deployment

    Ultimate Checklist for Successful App Deployment

    Deploying software releases is a mixture of planning, testing, late hours, and celebratory beers. This checklist is intended to be a guide to help improve how your team achieves this complicated and sometimes hairy task.

    Part 1: Planning and Deployment Preparation

    Collaboration Between Development and Operations

    It is important that operations be in the loop when it comes to planning any software deployment. They need to know when the project starts, not at the last minute about your new deployment. Ensure that operations is aware of any infrastructure changes needed well in advance. If the operations team is your front-line defense for monitoring and troubleshooting your app, help them help you by collaborating on how best to monitor the app and discuss early warning signs of trouble to be alerted to. If you’re thinking of setting up DevOp team you should probably read this.

    Build & Release Automation

    Removing the human factor is important to duplicating software deployment success. How you deploy the code doesn’t have to be state of the art. Utilize a basic build server like Maven, Bamboo, or TFS to automate the process or even a simple script that copies the files across the network can be sometimes sufficient. The key here is to make it easily repeatable by anyone on the team and eliminate the possibility of skipping any manual steps.

    Minimize the Amount of Change

    When things break in IT, typically it is because  something changed. When you introduce small changes in each software update it is easier to roll back those changes, or know exactly where to look for the source of the problem. As each release takes a certain amount of work in planning and testing, the key is finding the right balance of frequency and size of your releases that is optimal for your team to handle.

    Create and Test SQL Change Scripts

    Prior to your app deployment make sure you use tools like Red Gate SQL Compareto know what SQL schema and data changes have to be moved to production. to ensure there are no environmental differences that will cause potential headaches, I recommend never to forget to take a backup of production and run the test scripts on the backup copy before the final release. Production data differences or different database software versions can commonly cause hiccups. Also, have a plan to roll back to a previous state if the release has issues. Code is easy to roll back, but SQL requires some planning.

    Know your KPIs

    Every app should have some form of key performance indicators or metrics that can be used to know if everything is running smoothly. These will vary wildly from one app to another. They can be as simple as server CPU, page load times, or database performance metrics. The best metrics are more custom though. Knowing how many orders are happening per minute, or how many messages are being processed off a queue and how long it takes, or how long a certain piece of code takes to execute is a critical part of your app. Make your KPIs the heartbeat of your application and monitor them at all times.

    Setup Synthetic Transactions Tests

    Create a few tests that test things like your login page, key pages within your apps, APIs, etc. This will allow you to feel very confident that all systems are operational after the app deployment. I highly recommend Selenium, the leading open source framework for web automation.

    Notify Your Users and Colleagues
    Don’t forget to notify your users and others within your company about the upcoming changes. Make sure you have time to update documentation, train internal staff about new features, and coordinate with other departments in your company.

    Part 2: After the Deployment

    Monitor Server Utilization

    Make sure servers are all online, in the load balancer, and still receiving traffic. Monitor server and app CPU, memory, network, and disk usage. Noutilization can be as bad as sudden high utilization.

    Monitor Exception Rate

    After a new release your software is likely to throw some new exceptions you have never seen. Utilize an error tracking program so you can immediately identify new errors introduced in the release and to ensure previously resolved errors haven’t been reintroduced. A comparison of your overall error rates before and after the deployment will give you good indication of the release health.

    Check your KPIs

    Keep an eye on your performance indicators that let you know if your application is running properly.

    Watch your logs

    After an app deployment make sure logging is still working and the volume of logging hasn’t changed dramatically. No logging at all can be as bad as sudden high logging rates. A centralized log management tool makes this easy to monitor. You might also want to check the logs of new features that were released to ensure they are behaving properly.

    Monitor Page Load Times and HTTP Error Rates

    Watch the overall page load times of your application and keep a closer eye on 2-3 requests that are mission critical to your app. Also watch out for the rate of 400 and 500 level HTTP requests to ensure they haven’t dramatically changed.

    Monitor Database Performance

    After your deployment make sure database CPU, IO, and overall traffic look normal. Nothing is scarier than a sudden drop in database traffic after a release… unless it’s a sudden spike in database traffic.

    Monitor Key Database Queries

    Setup monitors in your monitoring system to test key queries to ensure they are loading quickly and returning proper results. For example, if some important background processes work based on SQL queries, run similar queries to make sure it isn’t missing work to be done and is working properly.

    Monitor Application Queues

    Monitor how many messages are in your application queues to ensure everything is flowing properly and not getting backed up.

    Cache Server Changes

    Don’t forget about caching. Depending on how you use caching, you may need to clear your caches or make other configuration changes. Data type changes can commonly cause some weird serialization type issues in cache.

    Have a celebratory beer!

    At Stackify we prefer a local joint with some good craft IPAs or the one brewed by one of our developers. Make sure your team has a good tradition for after releases.

    Post Mortem Meeting

    It can be very valuable to stop for a few minutes and discuss how the last release went and how things potentially can be improved. These kinds of conversations can help identify team weaknesses, allow people to vent, and take note of issues that can try to be prevented in future development cycles.

    Summary

    Deployment success over a long period of time depends on being able to automate and create processes that minimize issues for all stakeholders. Creating some automation or scripts around your build and release processes pay big dividends in the long run. Utilizing various monitoring tools or looking for new ones that  can help monitor your server and application health from every angle. Tracking trends over time can help you show off the improvements you’ve made with each release in availability, performance, KPIs, and user experience.

    About the Author

    mattMatt Watson founded Stackify in 2012 and as CEO provides the vision and leadership for the direction of the company. Prior to founding Stackify, he was the founder and CTO of VinSolutions. VinSolutions’ innovative product offering led to rapid company growth and was even featured on the Inc. 500 (#447) in 2010. Matt is an entrepreneur at heart and excels at product and software development.

    Matt studied Computer Information Systems at DeVry University. He started programming when he was in middle school to automate playing video games while he was at school. He hasn’t stopped programming since and loves anything technology related. Other hobbies include working too much, travel, movies, golf, cars, and of course his family. You can reach Matt at  @Stackify, @mattwatson81,  www.stackify.com, https://www.facebook.com/Stackify

     

  • Time-Sharing to Unicorns: The DevOps Problem Facing Enterprise IT

    Time-Sharing to Unicorns: The DevOps Problem Facing Enterprise IT

    How did we in Enterprise IT get to this point? There’s a lot of talking and writing about the value of adopting DevOps but very little analyzing the problem domain we’re trying to correct. There are some schools that will say we can move forward without looking back, but I believe a key element of the dysfunction that faces many enterprise IT organizations can directly be attributed to their history and how they were formed.

    One of the clear lessons we can learn from businesses that we call “unicorns”–termed this because of their mythical capabilities as an IT organization that can deliver high quality software very quickly–is that they have very little friction in the process from ideation to operation. Startups and greenfield initiatives are practically operating in a vacuum relative to the amount of friction they encounter compared to operations around entrenched decade-old legacy mission-critical applications. In retrospect, this should not be surprising given frictionless delivery is a mandate for many of these efforts and has been from the start.

    You create what you focus on!

    Some will say we don’t require the capability to deliver to production ten times a day (or even a year) for our SAP Enterprise Resource Planning (ERP) system. So they will conclude that perhaps DevOps isn’t right for our needs. This shows the skewed thinking that current literature on DevOps is fostering. It is not the need to deploy changes quickly into production that is at issue, but the ability to deliver business capabilities when they are required as quickly and with little impact as possible. Current processes around changes to the ERP are usually so fraught with friction that when the time comes to fulfill a need for the business so that the business can continue to compete effectively, it takes weeks or months instead of days or hours.

    I researched how the division between development and operations was formed and interestingly, there’s very little history about this division of work. We can trace this division back to the early days of time-sharing systems in which the division of work actually represented the exact advice DevOps prescribes today. Deliver and operate a platform on which developers can build and deliver their applications. In the days of time-sharing systems, operations was not responsible for what developers put into the box, just ensuring that the box was available, secure, and backed up. If developers applications had bugs or crashed, it was the developers that were responsible for repair.

    Then came the era of client/server. Application pieces ended up distributed across application infrastructure like a kid that dumped their Legos all over the floor. Managing these applications became as painful as stepping barefoot on a Lego as well. Developers didn’t work on the complete application, but instead worked on modules and these modules were split across various services each requiring their own levels of expertise to keep running in an optimal manner. The DBA was responsible for ensuring the database could meet transactional requirements and support data design and management, the network professionals dealt with protocols and traffic issues caused by these modules communicating with one another. Each layer of the application had it’s own capacity management and operational concerns. The only chance we had to keep the ship from sinking was to put it all in the hands of the one team that had access to all the components — IT Operations.

    Things have not progressed much since this divergence away from time-sharing to client/server either. Today’s applications are more complex and more distributed than ever before. We have clusters of the same server working in unison to handle scalability exponentially increasing the complexity and difficulty for operations. A single application may now require anywhere from 5 to 25 servers to deliver the required capacity.

    On top of all this we’ve never modified the division of work from the days of time-sharing and continue to push the burden for monitoring and uptime to the operations staff. We have turned IT operations into the Jigsawyer’s–someone who does jigsaw puzzles–and developers into the puzzle makers. Moreover, the puzzle is never the same; the puzzle makers change the picture all the time as they add new piece forcing the Jigsawyers to figure out how to retrofit the new piece into the whole.

    While researching the history of IT operations and trying to figure out the path from computing’s earliest organization until now, I came across a great piece that really helps identify the problem we have created. So What Is A Deployment Really? by Robert van Loghem illustrates what it takes to move from ideation to operation with today’s application platforms. What I really like about this piece is the illustration that even if we want to go back to earliest division of work, it may not be possible. IT operations cannot simply provide a black box for developers to deliver their applications on and expect them to work as planned. Firewalls need updating, web servers need configuration changes to support the new domain requests and the security architecture must now allow these new requests to be routed appropriately based on rights and source.

    Deployment is a team activity!

    So, what have most of us in enterprise IT done? We broke down the tasks of deployment further into divisions of work so that we can apply the necessary level of specialization to ensure the application can be delivered to the end user at the cost of adding a ton of friction into the process slowing things down by a logarithmic factor when compared to our unicorn friends.

    What is surprising is the number of pundits that put forth that the answer to this problem domain is empathy and culture change. Clearly, we have organizationally written ourselves into a corner because this is how large enterprises solve problems. Here’s a simplification of how this decision was most likely made:

    Question: There’s a lot of issues attempting to monitor and operate these distributed applications and the developers don’t have the means or transparency into all the components.

    Retort: Then who does have access and insight into all these components?

    Answer: IT operations can see what’s running and has the tools to monitor the health, let’s give them the task.

    At a time when IT was viewed as an expense versus a strategic advantage, there was no offsite planning to assess what the impact to this answer means to the business ten or twenty years hence. It’s solve the problem and keep moving because that’s what enterprises need to do in non-strategic areas. At the time this answer was formulated most businesses equated this decision with “which office supply vendor should we use?” Moreover, the leaders in client/server barely had any experience with IT operations, they were primarily comprised of application developers. Hence, there was no experience to guide businesses away from this decision. IBM, Unisys, Honeywell, etc. the vendors that were a prime source of guidance at the time were still leading with a time-sharing mentality.

    This entire trip down memory lane was with one purpose in mind, figure out what we created and how it was created so we can consider the means to unwind it and provide a path for enterprise IT to move forward in an agile manner. The history provides the answer. We need to move away from the traditional time-sharing division of work that was formulated before the age of the Internet and realign along frictionless surfaces. What does that mean? Well, that, my friend, is an entire book that happens to be in the works! 😉

  • 5 ways APM can solve your DevOps worries

    5 ways APM can solve your DevOps worries

    This week at the InterOp conference in New York I had the chance to talk to a lot of IT managers transforming their organizations to adopt a DevOps culture. One of the key questions I heard repeatedly was – are there some best practices to break down the wall between Dev and Ops?  How do you get to a point where we can deploy code multiple times a day like Facebook , Amazon and Netflix without impacting our users’ experience ? How do we minimize the risk of deployment failures and the blame game that ensues? With the need to innovate and close the customer feedback loop quickly Ops is being asked to deploy code more rapidly than ever before.  The complexity of hundreds of developers updating code makes it a very sensitive environment. The last thing Operations wants is an outage leading to financial loss and the last thing development wants is to get a call in the middle of the night to figure out what code change caused it.

    To have tight control over this fast paced environment we need Application Performance Management at every step. APM can help break down the Chinese wall between Dev and Ops and eliminate the finger pointing that happens during outages. Here are 5 APM best practices to enable a smooth DevOps working environment

    1. Use the same APM tool in Dev and Ops: APM should not be an afterthought , should not be a tool that only Operations looks at to monitor apps. APM is essential in every step along the way , from dev/test to staging to production. As a part of test deployment, developers should use the same APM tool as Ops to see the impact of the code deployment on the application’s performance and catch issues before hand. With the same tool everyone is on the same page and there is less confusion on the metrics being tracked. Developers can’t say “Sorry Ops, we are not seeing the problem”
    2. Isolate and fix performance problems sooner with Code Level Diagnostics:  Today’s APM tools come equipped with deep diagnostics that lets you drill down to the line of code that may be causing a performance slowdown. When code deployed in the Test or Staging environment results in a response time spike, developers can use deep dive code diagnostics to detect the line of code, the SQL query, the method call that may have caused the spike and fix the problem before it is deployed on production. For example, you can detect that a SQL query is taking way longer than it did last week for a specific mobile device platform.
    3. Detect changes with Deployment Detection: When new version of code is deployed in production, the change should be detected automatically without having to reconfigure the APM tool.
    4. Analyze Pre and Post Deployment Performance Trends: Get before and after snapshots of the end to end transactions from your APM tool and compare performance trends over time. With topological visualizations you can, at a glance, see if any particular business transaction slowed down post deployment. With historical data you can see how the application’s performance and resource consumption is evolving over time. This will help fine tune your apps’ resources as well. For example you can determine whether an AJAX call has been slowing down over the last 2 weeks due to the slow down of a third party API call or because a disk consumption is increasing over time.
    5. Automate actions on failures: Configure your APM tool to take actions on failures. For example configure an action to automatically increase memory if a memory spike alert is generated post-deployment while you take the time to diagnose the root cause.

    Using APM let your developers focus on writing code rather than spend time fighting deployment catastrophes and enable a healthy collaborative working environment between Dev and Ops.

  • Walgreens reminds us to keep our database healthy

    Walgreens reminds us to keep our database healthy

    It’s clear that the super app you developed with your team, is really something, and it has high chance to be the “Bread and butter 3.0”, but are you prepared for the worst? There is a good chance that some combination of data, events and circumstances out of your control will eventually collude against you, to cause a failure or a software defect. You can try to anticipate and reason about all of the things that can go wrong with your application to prepare, but in a complex system it’s impossible to predict everything.

    (more…)