devopsy.kinsta.cloud

Category: Enterprise DevOps

  • Application Monitoring in the Age of Big Data

    Application Monitoring in the Age of Big Data

    Operations (or ops, as it is known colloquially) has been necessary since the first shared systems came online in the ’60s. While the job title didn’t necessarily exist, the same questions needed answers then as they do now:

    • Is it up and accessible?
    • Is it responding in a reasonable time frame?
    • Is it performing better or worse now than previously?

    In a simple system, say a static web site, these questions are answered easily by requesting the page. In an automated fashion, of course.

    However, when you look at any “late model” technical infrastructure, you’ll find many, many moving parts. These range from border routers to web servers, work queues to databases and data caches to big data clusters. There are thousands upon thousands of computing instances, software processes, network storage devices and network links that can go down, up or just plain wonky. This means potentially millions of places where things can go wrong.

    How do you keep an eye on all of these potential issues? How do you know if they’re responsive? How do you know if things are getting better or worse?

    Some Monitoring Progress

    The “old” method was to write some sort of monitoring script that polled some data that you thought would represent the overall health of that component. Through the years things have gotten more granular. Our monitoring script libraries have grown and we’ve even instrumented our applications with counters (statsd, librato, etc.) and come up with big data storage (opentsdb) to house all of these data points.

    The question remains, Do we have any more clarity than before? We unarguably have more data points. So we must know more about the performance of our systems, right?

    Turns out, in practice, we just have a lot of data and a lot of people attempting to decipher what that data means. If you’ve worked in operations you’ve been up at 2 a.m. looking at status dashboards with green, yellow and red icons and many, many graphs, all in an effort to answer the question: What is going on?

    Sure, some incidents are easy: An instance was retired, or a fiber line was cut somewhere in Kansas.

    I’m talking about the ones that aren’t so clear cut, so to speak:

    • Site feels slow
    • Median response times are up 25 percent from yesterday
    • Usage is down week over week compared to some relevant time frame

    How do you answer those questions in a haystack of data points, up/down status and, in some cases, just plain noise? Application performance managers (APMs) came along and gave us tooling in the app frameworks and the browser. Now we have visibility into which calls are performing poorly and how our app “feels” to the end user.

    This has helped by breaking down the haystack into smaller haystacks and provided a way to search and sort potential performance issues. However, most APM tools depend on instrumenting code such as Java, .Net or Ruby, and are not able to provide monitoring insights for the rapidly growing open source frameworks such as Grails, Akka, Netflix OSS and the like. We’re still left with the haystack problem: What is going on?

    When I’m up at 2 a.m., stumbling over to the computer to find out what issue PagerDuty has alerted me to, I don’t really want to pore through page after page of graphs and up/down status. What would be really helpful is a system that can aggregate all those monitoring data flows and statuses, actually apply some intelligence. Then it could tell me, “Look at this response time of this component, and this metric is reporting something anomalous. This is most likely causing your issue right now.”

    In other words, it would be helpful to have a system that can show both the symptoms and the root cause of the issue. That system also would provide high-quality, high-value information about my infrastructure and combine anomaly detection with root cause analysis to provide real root cause detection.

    One new startup, OpsClarity, is hoping to make troubleshooting more accurate and efficient by combining data science and anomaly detection with root cause analysis.

    Perhaps this is the long overdue beginning of the next age in monitoring.

     

    About the Author/Michael Hobbs

    Michael Hobbs HeadshotMichael Hobbs is the director of Technical Operations at Expa, a startup studio based in San Francisco. Significant experience in building, operating and automating highly scalable application infrastructures has allowed him to contribute to the success of Infusionsoft and Stumbleupon, among others. Michael continues to put that expertise to work at Expa by advising and assisting the next generation of Internet companies. Michael is also a significant contributor to many open source projects, including dokku and herokuish.

     

  • SolarWinds Powers Instacart’s DevOps Efforts

    SolarWinds Powers Instacart’s DevOps Efforts

    Instacart is an Internet-based home delivery service for groceries. Its mobile app and website enables consumers to leverage its service, which operates in 17 cities across the country. Customers order through the app, and Instacart’s personal shoppers fulfill those orders at local stores. Forbes magazine named Instacart the Most Promising Company of 2015, citing its growth since the company’s 2012 inception, notes Nick Elser, director of Engineering at Instacart.

    DevOps touches critical business applications at Instacart, including the apps and APIs for both iOS and Android mobile operating systems. It also supports back-end systems such as processes rooted in machine learning that assign grocery lists to shoppers and a pipeline that retrieves retail partner product data about available grocery items nightly.

    Key Mobile Development Challenges & Goals

    The Instacart DevOps shop is tasked with maintaining infrastructure in software, expanding the current business apps through innovations, and scoping and building future software for the company and its services. “We needed better infrastructure-monitoring solutions in place to save time and help scale parts of the infrastructure as the company continued to grow,” says Elser.

    Elser and his team looked at SolarWinds‘ capabilities, and were enticed by the Librato cloud-based monitoring tool’s potential to safeguard the performance and uptime of the company’s mobile APIs. “Our customers demand a high level of reliability from their consumer services,” says Elser. “Librato ensures the reliability of our mobile APIs by keeping a close eye on the health of our internal/external services.”

    Instacart hoped to use SolarWinds to support its infrastructure-scaling efforts. By enabling Instacart to leverage aggregates of logging data and statistics reports, Elser and his team could focus on creating an increasingly innovative and fruitful grocery delivery service. “We wanted to concentrate on scaling our operations, fleet of shoppers and metropolitan presence, as well as identifying processes that would enable us to move faster as a team,” explains Elser.

    Applying SolarWinds Tools

    With the software libraries they needed already present in the programming languages and environments they had on hand, implementing the SolarWinds tools to serve Instacart’s intended goals was relatively pain-free. One tool, Papertrail, required only a minor adjustment to existing config files to forward log records.

    Papertrail log management tool eased logging duties for the entire Instacart infrastructure. “I could have Papertrail on all my boxes, give it an endpoint and it would be aggregated into different buckets based on host names,” says Elser.

    Instacart also added SolarWinds’ Pingdom solution to notify the team about website downtime issues. This enabled the team to gauge real-world user experiences with uptime and to assess synthetic transactions for issues that could concern customers. “Pingdom was the most flexible uptime monitoring tool my team could find,” says Elser.

    Librato also proved worth the effort, thanks to its many value-adds. “We received a lot of free advantages with Librato because it came with Amazon CloudWatch integration,” says Elser. This made it fairly trivial to add instrumentation to our apps.”

    Using Librato ultimately enabled Instacart to create graphs to identify correlations in event data—instances where some relationship exists but falls short of cause and effect. Instacart could then use that data to pinpoint and resolve security breaches. “We also use the Librato CloudWatch integration to monitor database loads, as these affect the infrastructure’s health,” says Elser.

    By integrating Librato with Heroku and AWS CloudWatch and applying other SolarWinds apps, Instacart gained visualizations of important infrastructure elements and transparency into its entire software stack for monitoring stack health.

    Other Tools

    Instacart also uses other tools to help its DevOps, including Crashlytics mobile crash and usage tracking application, Urban Airship mobile push notification tool, Branch.io deep-linking application and the Rollbar exception tracking tool.

  • The Big DevOps Question: How Do We Get Started?

    The Big DevOps Question: How Do We Get Started?

    Every conversation I have with clients regarding DevOps eventually reaches a point where the client wants to know my recommendations for getting started with a DevOps program. Some have already embarked on a DevOps program, but want me to critique their approach against what I see others doing. Others simply don’t have a clue how to make the necessary changes in their organization.

    Ultimately, I sense in all these conversations a lack of confidence that their intended outcome is achievable and that they’re on the right path.

    The first questions I ask clients are, “What are you looking to accomplish, and what are your business drivers?” Interestingly, 90 percent of the time clients are unclear why they want to embark on this journey. Sure, DevOps makes sense academically, but if you don’t have a clear understanding of your expectations for improvement most likely you’ll end up with more questions.

    Sometimes it helps if I give prompts:

    • Are you looking to improve quality in your production releases?
    • Do you need to release updates and patches into production more quickly?
    • What metrics are you currently looking at, and what makes you believe these are not optimal for your business?

    This last question is really important. Quite often data represents invalid perspectives. Other industries’ and services’ metric of how often they deploy to production or how long it takes execute a release cycle might not be right for your business or industry. I hope medical device and airplane equipment manufacturers take a lot longer to bring something to production and test it more robustly than Netflix does its website.

    In an earlier post I wrote that the need for DevOps is effectively the admittance of a problem; to wit, the cure is continuous delivery. Hence, I often tell clients that a very realistic place to start is not with the actual work, but with the communications methods and pathways that will affect implementing continuous delivery within the organization.

    So many clients want to start their DevOps journey in application development and talk about what they’ve done with continuous build. That’s fantastic, but it has more to do with Agile software development than it does DevOps. If all that work creates bottlenecks in testing, you’ve accomplished very little. Now you must fix the flow to balance out development and testing. Trying to achieve continuous development through serial analysis and correction will take a long time and show little progress along the way—and often leads to abandonment.

    However, if all the leaders of all the various areas involved in the continuous delivery cycle discuss how their efforts are impacted by what the other leaders’ groups do, everyone can see the bottlenecks and constraints and they can prioritize changes. Management often fosters much of the cultural impairment, as these changes can impact everything from workload to bonuses. Getting everyone involved in working together to eliminate the bottlenecks often buys credibility for the program and minimizes some of the individual adoption issues.

    Don’t jump into the deep end of the DevOps pool just because everyone else is. Before embarking on your DevOps journey, do a deep analysis of your organization’s goals and baseline your current levels of performance. Discuss what needs to change based on real business goals and drivers (e.g., it takes us too long to issue regulatory changes to the system). Identify bottlenecks and constraints that exist in the current environment today and put together a plan to eliminate as many as possible.

  • Only DevOps and Taxes

    Only DevOps and Taxes

    As software eats the world, it’s hard to have a conversation about anything that isn’t directly related to or driven by modern mobile, web and IoT applications. DevOps describes the modern approach to building these applications. But like all new movements, there is a cost. And while some of the costs associated with modern development are not new, it’s becoming more and more clear that DevOps organizations have to pay their tech tax or prepare for trouble.

    (more…)

  • End-User Analytics: A Reality Check to SLAs

    End-User Analytics: A Reality Check to SLAs

    Everyone understands that a great user experience is vital for customers, but few organizations give the same level of consideration to their own employees. This lack of insight into the internal end-user experience leads to frustration, inefficiency and waste. Detailed analysis can lift the curtain, exposing the gap between your service level agreements (SLAs) and the reality of your current systems and software.

    Some of your employees may have more applications than they really need, while others may lack the tools to do their job effectively. It’s also likely that some of your employees are making do with outdated software that’s no longer fit for purpose. Before you can set about changing that, you must gain an understanding of the overall picture.

    What does your user experience look like?
    Many organizations will have trouble answering this question. They may not know how many applications they have, which ones are being used, and how well they’re performing.

    The rise of shadow IT—and the fact that 40 percent of spending no longer controlled by the IT department, according to Gartner’s “How to Manage the IT Budget Wisely Through Cost and Value Optimization,”—are further muddying the waters.

    To determine your company’s user experience, ask yourself the following questions:

    • How many users do you have, and how are their roles defined?
    • What devices are accessing your systems on your daily basis?
    • Where are all the access points located?
    • What content is being accessed?
    • What applications are being used?

    By trimming the unused apps and consolidating common apps through a virtualized XenApp or similar environment, you can redirect resources to focus where they can deliver the greatest benefit.

    SLAs are failing
    It may be tempting to take the SLA model and apply it internally, but SLAs are too simplistic and one-dimensional to deliver the insight that’s required here. Cloud service providers might focus on availability and uptime, but what do those metrics tell you about performance? Issues such as slow-loading pages usually aren’t included in the metrics, but they really should be if you want to assess the user experience.

    There’s also a trust issue here. Can you validate that your service provider is meeting expectations? What about security? Relying on provider self-reporting and a set of metrics that don’t reflect accurately the performance or the user satisfaction with the platform or application are inadequate if you really want to understand what’s going on and act to improve things.

    End-user analytics
    You need insight to real-time and historical data on all your hardware, users and applications across multiple environments inlcuding traditional, mobile and virtual. Good end-user analytics will flag security risks and suspicious behavior, record when an application hangs or crashes, and identify whether specific issues are widespread or localized. Records should be comprehensive enough that you can audit areas of concern and run root cause analysis to find ways to improve. Turn your end users and their devices into roaming sensors, which paint a factual view on usability and performance.

    Finding out how your employees really work is invaluable insight that can streamline many emotional conversations. Finding out what applications and devices they’re using—and how they’re using them—can unlock many potential benefits for different areas of your business. You can ensure costs are kept down and you maintain regulatory compliance while maximizing agility. A fresh insight into risk and productivity will help you to make informed business decisions.

    The end user knows best
    The only real measure of software quality that’s worth its salt is end-user satisfaction. Forget about what applications have been designed to do; forget about developer and service provider promises that don’t include an end-user analytical strategy. Cut to the chase and look at what’s actually happening; analyze every user session and employ business intelligence to extract actionable insights that will have a tangible impact.

    Give users the flexibility they want and the tools they need to do the best job possible. Happy and productive staff are the foundation of a good business. Seek out service providers that provide transparent end-user analytics; this will develop a foundation of trust and collectively you can provide guidance based on factual performance. Hone in on boot times, application performance or network speed: You dictate the metrics that are important to your business and metrics that make a difference to your customers.

    By acting without analyzing and putting your faith in SLAs, you’re basing the company’s future and its future path on pure guesswork. With end-user analytics, you can find out exactly what’s actually needed and ensure that you’re on the right track.

    About the Author
    crop_Nicholas Lee of FujitsuNicholas Lee is head of Global Digital Programs for Fujitsu, the Japanese information and communication technology company.

  • Machine Learning, AI Driving DevOps Evolution

    Automation is the fuel that drives DevOps. Automating routine, repeatable tasks is one of the defining characteristics of DevOps culture. As artificial intelligence (AI) and machine learning improve, the scope and complexity of the tasks that can be automated increases, which raises the bar for all of DevOps. The humans behind the DevOps will be freed from even more mundane tasks and able to focus on more innovative and creative endeavors.

    It makes sense, if you think about it. The trend from physical services in an on-premise data center, to virtualization, to cloud services, to DevOps has been a steady march of raising the bar for automation. The more applications and entire infrastructures can be developed, deployed, monitored and managed programmatically and through automation, the more efficiently and effectively an organization can function.

    As tasks are offloaded to automation, humans can address other challenges. That focus inevitably leads to identifying other new tasks that can be automated. AI and machine learning represent a quantum leap in this evolution, however—with AI and machine learning, the automation can begin to identify and improve itself. When the automation can identify bottlenecks and inefficiencies and automatically adapt to overcome those obstacles, human input and interaction could become almost obsolete.

    In a recent interview, Christian Beedgen, CTO of Sumo Logic, shared some thoughts about the impact of machine learning on DevOps. “Machines will enable greater productivity among teams and businesses through the advances that machine learning and automation can offer,” he said. “A necessary component to augment human contributions, machine learning will finally give humans the power to review millions of bits of data.”

    Automating routine tasks is crucial, but there’s another factor that plays into the role that AI and machine learning have with the future of DevOps. The reality is that there are some things humans simply can’t do as well or as fast as machines—especially at scale.

    Steve Burton stressed in a blog post that humans are capable of observing, digesting and interpreting only so much information at one time. Some feel like the Holy Grail solution is some sort of application performance management (APM) dashboard that can aggregate and correlate real-time data so human IT personnel can monitor metrics and identify issues manually. “This all sounds good in theory, except it’s completely unmanageable in reality when your applications have thousands of components, billions of metrics and hundreds of changes every day,” he wrote. “Humans can no longer cope with this complexity, they now need machines to do the leg work, process this Big Data and provide operational insight into what the f**k is going on.”

    Beedgen agrees. He suggested, “Increasingly, businesses will adopt machine learning technology for its usefulness in augmenting human understanding of complex interaction and large data sets by uncovering the unknown unknowns.”

    On some level this all sounds a bit scary. It seems as though the more we succeed at DevOps and the more DevOps evolves, the less humans are even a necessary part of the equation. Conceivably you could be so good at DevOps that you DevOps your way right out of a job or inadvertently launch SkyNet—or both.

    That scenario is simultaneously feasible and far-fetched. I suggest that you don’t fear that future or look at it as machines stealing your job or taking over humanity. Embrace the future of DevOps driven by AI and machine learning and find the silver lining. Seek out the new frontiers and horizons, because there always will be tasks that the machines simply can’t do without us—and that is where innovation happens.

  • Why You Need to Embrace Automation

    Just in case you hadn’t noticed: The robots are coming.

    Well, you probably had noticed. It’s hard to miss. Newspaper articles and blogs everywhere herald their arrival. The stories usually follow the same general formula: A catalogue of amazing things that robots (really, AI systems) can do. Debate about forced unemployment vs. people freed up to do more interesting work. And then some comments about whether it’s for the good or bad. You can generally infer the general tone from the choice of image accompanying the story: a cute Japanese robot with a plastic smile or a more sinister face with the Terminator’s jawline.

    Of course, workplace automation has been around for almost as long as we have had workplaces. Each new wave of technology disrupts the employment landscape in its own particular way. However, this time it’s serious: It’s not just manual labor that’s under threat, but increasingly, we’re told, white-collar workers are being supplanted by machines. A recent McKinsey report has generated a wave of headlines: “Even the CEO’s job is susceptible to automation,” wrote Patrick Nelson at Network World; and Ana Swanson at the Washington Post reported, “How robots will even affect the jobs of people we thought were immune.”

    Not surprisingly, a closer read of the McKinsey material gives a more nuanced appraisal than a headline can offer. Few of us will actually be replaced by a robot, but many workers will lose some part of their current jobs to automation. The McKinsey report states that up to 45 percent of the activities that employees perform today could be automated, but very few occupations will be automated entirely in the near or medium term. The things that require creativity and emotion remain uniquely in the domain of humans, and they are likely to remain there for quite some time.

    It’s against this background that we see a steady increase in what is often referred to as robot journalism: machines writing news stories. So far, these applications rely on scenarios where the general ‘shape’ of a story remains pretty consistent because of the regularity of the data underlying those stories. For example, a report on stock performance is always going to mention the current stock price, what the stock price was at some particular point in the past, the price highs and lows, and perhaps something on volume or analyst sentiment. These reports often are found in the back pages of a newspaper, and are the stories that few journalists actually want to write, since they allow almost no space for creativity and emotion.

    The reality is that today’s news-writing software is still a long way from replacing human journalists. Which is not to say that what a robot journalist produces isn’t useful; it surely is. Last month I spoke about robot journalism at the news:rewired conference. It was fascinating to see how most of the major news media are exploring how to get robots on staff. So while we’re going to see more robot journalists, it doesn’t necessarily mean fewer human journalists. A recent study by researchers at Oxford University and Deloitte explored how susceptible various jobs were to automation—journalists ranked 285th out of 366 jobs, with an ‘automation risk’ of just 8 percent.

    But if you really want to see where the action is regarding automated writing, just look at it from a different angle. Rather than Natural Language Generation displacing people from their jobs, it’s offering a solution to the problem of shortage: A shortage of workers or expertise. And that story isn’t about news; it’s the Big Data story. You can take issue with the astounding numbers that are thrown around, but even if they’re out by a factor of 10, they’re still significant: by 2020, about 1.7MB of new information will be created every second for each person on earth. But less than 0.5 percent of all data is ever analyzed.

    In 2011, McKinsey noted, “The United States alone faced a shortage of 140,000 to 190,000 people with analytical expertise, and 1.5 million managers and analysts with the skills to understand and make decisions based on the analysis of big data.” And subsequent reporting points to massive year-on-year leaps in demand.

    Regardless of how few or how many people are displaced by robots, they are not all going to become data scientists. The only way this volume of data will be analyzed and explained is if we have machines doing it for us. And that’s the real calling and promise of Natural Language Generation (NLG). Interactive visualizations and dashboards don’t cut it; the meaning in all that data will be lost unless we call in the robots. But you need more than robot journalists, tirelessly churning out one formulaic story after another. You need articulate robots that use sophisticated NLG to construct fluent narratives from idiosyncratic data sets. So, accept the data reality, embrace automation, and find an NLG robot to explain the insights buried in your data.

    About the Author:

    2,41 CWDr. Robert Dale, Chief Technology Officer and Chief Strategy Scientist at Arria NLG, is recognized as one of the world’s foremost experts in Natural Language Generation (NLG) research and Development, having authored or edited seven books and 160 papers on computational linguistics. He was a Professor in the Department of Computing, Director of the Centre for Language Technology at Macquarie University.  He co-authored the seminal textbook “Building Natural Language Generation Systems” with Arria NLG Chief Scientist and co-founder Professor Ehud Reiter.

  • Rugged DevOps: Good, Bad & Ugly

    Rugged DevOps: Good, Bad & Ugly

    The Good: DevOps has been a success for Dev.

    With higher expectations, better processes, more management focus and new tools like advanced PaaS and containers, developers are churning out more and more software faster than ever. Success! Or is it?

    The Bad: DevOps has been a success…for Dev.

    For Dev teams, this may result in pats on the back and high fives. But for Ops, this can look like one thankless tidal wave of work after another. The same software velocity that is praised in Dev is currently overwhelming their counterparts in Operations. And frankly, many DevOps tools released in recent years happen to benefit Dev much more than their counterparts in Ops. (My pet theory is that developers at IT automation firms are acting and thinking like…developers – focusing their creative energies on solving Dev problems.) A similar problem holds true for IT Security teams who want to build quality in by “shifting left” – see  https://devopsy.kinsta.cloud/2015/12/10/getting-rugged-devops-right/ . From the vantage point of IT Operations and Security teams, they are doing more work than ever but without any more human or IT automation resources. To add insult to injury amid their increased workloads, many IT Ops teams have to invest countless hours (away from family and personal time) in training courses and migrations to reorient their processes to fit into shiny, new Dev-centric “DevOps” tools and paradigms. 

    The Ugly.  

    Work. Overwork. Glitch. Search. Find. Rework. Repeat.

    A quick glance at IsItDownRightNow.com will show you which IT Operations teams are in a living hell at the moment. The other thing you will notice is that these IT Ops teams are from some of the most reputable companies on the planet. They are talented teams but stuck in rework cycles of web performance problems, security vulnerabilities and outages.

    These problems get the attention of everyone. Operations…and Dev…and Security…and customers…and Finance…and auditors and the press.

    No one likes rework. And no one likes the panicked fire drills that cause panicked rework. So what causes those rework cycles?

    Drama-free IT. Visibility Cures Chaos.  

    Much of this chaos can be tied to miscommunication. Boring? Maybe. Simple? No. Part of miscommunication is cultural. (Hint: OrcaConfig’s “How to Screw Up DevOps” white paper covers the cultural aspects of miscommunication). But miscommunication is also cured by automation solutions that promote transparency and visibility.

    If we want drama-free IT we need IT automation solutions that eliminate miscommunication and missed assumptions between Dev, Ops and Security.

    What better way to do this than with full transparency and visibility?

    Bird’s Eye Views:

    Many IT automation tools cater to SysAdmins who are interested in managing configurations of individual servers. That’s great if your focus is on individual severs. But IT Operations, eCommerce teams, Application Owners, Compliance & Security teams and IT Managers often care more about the health and performance of their application ecosystem (including related applications, middleware, databases and configurations that tie them together). So there is often a mismatch between the tools that SysAdmins and Developers have versus what IT Ops needs.

    Operating at Enterprise Scale Requires at-a-glance Visibility:

    IT Ops teams need bird’s-eye views of applications, databases, middleware and ecosystems to promote ease of ongoing operation. They need to be able to see, at a glance that a configuration compliance problem exists and they need to know right away the location and nature of the error. Large scale IT Ops teams do not have the luxury of toggling back and forth between multiple reports and multiple screens searching for that non-compliant, broken needle in an ecosystem haystack. If they are faced with a glitch or an outage, they need to find and fix it right away. One size does not fit all. Dev, Ops and Security each have their own interests and their own need to view their application through a lens that is most meaningful for them. Multiple ecosystem views eliminate guesswork between Dev, Ops and Security and save precious time in the find-fix cycle.

    Forrester Research spells it out well in their 7 Habits of Rugged DevOps white paper, Habit 1: Increase Trust And Transparency Between Dev, Sec, And Ops:

    “I&O pros pride themselves on understanding the interconnectedness of the hardware and software environment, and they are unnerved by complexities that poorly governed delivery practices create because the firm rates I&O pros on their ability to improve the performance and stability of the production environment. To maintain the environment, they must reduce the chances of an outage or interruption of service, and they must reduce the MTTR when an issue does occur.“

    Understanding that interconnectedness depends greatly on how much intuitive ecosystem visibility we provide to our colleagues in IT Operations. But it’s not just IT Ops that benefits from visibility, Dev and Security do too.

    Developers are able to write more secure code when they get early, in-context feedback. As David Mortman, distinguished engineer and Dell chief security architect, said, “Security issues are product quality issues, and no one wants to write buggy code.”

    Even better than efficiently finding and fixing errors is of course to prevent them in the first place and to proactively eliminate the damage they cause. Here too, the answer is better visibility. With true application ecosystem visibility, Dev and Ops teams would be able to model relationships between applications, middleware and their configurations so users will know that “changing this, affects that”.

    My teenage son has better visibility via his smartphone as to his driving route and weather conditions than I have for my multibillion dollar IT operation.

    Visibility into the Future:

    Ecosystem visibility is a key to communicating between Dev, Ops and Security and running complex application environments in the present. But what about the future?

    Release automation tools and custom scripts are excellent at deploying changes across large environments. Unfortunately they can be just as efficient at deploying typos and other mistakes across large environments, causing the chaos that we discussed earlier. Here again Operations teams would benefit from improved visibility – the ability to preview and pre-validate proposed changes across the entire ecosystem well before those changes are deployed. This is the real promise and savings of ecosystem visibility. Better visibility, with more use-case lenses cuts out the miscommunication that causes too many performance glitches, security vulnerabilities and outages. This chaos takes a toll on our IT Operations teams, our employers and our customers.

    What Next?

    We are soon entering a new season with new budgets. The temptation to “buy something” will be great. In the spirit of Rugged DevOps my recommendation is to take a full, bird’s eye view of your SDLC and to shift Ops and Security leftwards, building their critical considerations into your processes and into your IT automation tool selection.

  • Never Mind the Bollocks, Here’s the DevOps

    Never Mind the Bollocks, Here’s the DevOps

    What is DevOps?

    “DevOps is a cultural and professional movement” –Adam Jacob

    [movement (n): a group of people working together to advance their ideas.]

    So, who are these people and what’s the big idea?

    The people are system administrators, developers, product managers, QA technicians, security analysts, network engineers, and anyone else involved in the business of developing software.

    The big idea is to increase the value your product or service delivers, more quickly with less effort.

    (Having a product or service that is designed and built well isn’t inherently valuable. The value comes when your customer is able to use your product or service to solve a problem. DevOps is all about delivering value quickly and effortlessly.)

    DevOps practices and technologies are evolving, but here are some of the core concepts paraphrased–be sure to read the links in the footnotes for the full explanations:

    The Three Ways

    • Systems Thinking
      • Learning how everything is interconnected, from concept to customer
    • Amplifying Feedback Loops
      • Cutting the time between action and reaction, failure and success
    • Culture of Continuous Experimentation and Learning
      • Recognizing failure is important data about how and where to improve

    CAMS

    • Culture
      • It’s about how people relate and work, not something you can buy
    • Automation
      • Faster, less error-prone processes help amplify feedback loops
    • Measurement
      • You can’t get where you want to be until you know where you are
    • Sharing
      • Effective teamwork ignores organizational boundaries

    ICE

    • Inclusivity
      • Everyone is responsible for (and benefits from) these practices
    • Complex Systems
      • Because simple systems are simple
    • Empathy
      • Real collaboration requires real understanding of another’s point of view

    That’s the theory from a very high level; the following Q&A provides some broad explanations of related concepts. These are actual questions I was asked in a recent discussion of DevOps.

    Q: What specific skills or experience do I need before I can put “DevOps” on my resume?

    A: It depends. At this point in time when recruiters and/or hiring managers list DevOps in a job posting they are almost always looking for a “DevOps Engineer” (i.e. a sysadmin who knows just enough coding to be dangerous), or a “full-stack developer” (i.e. a developer who knows just enough about system management to be dangerous). The reality is that (as of today) there is no DevOps certification, organization or even manifesto, so anyone who pretends to authority on what is and isn’t DevOps is overstating their case at best, a charlatan at worst.  

    Q: What is source control?

    A: Source control is a repository for files that make up a modern software application during development. It maintains a historical record of changes so it’s easy to review (or rollback to) changes made to an application’s source files. The most popular source control technology today is Git, and GitHub is one of many online services used to manage Git repositories.

    Q: What is Jenkins?

    A: Jenkins is to task engines or “build servers” what Kleenex™ is to facial tissue. It’s currently the most widely used software product for automating various functions of software build and deployment. For example, a system might be configured such that any code change pushed to source control triggers a job in Jenkins to build, test and deploy that code.

    Q: What is configuration management (e.g. Salt, Ansible, Puppet, Chef)?

    A: Configuration management is a means of scripting server configurations so it’s easier to track, deploy or make configuration changes to large groups of systems.

    Q: What are containers?

    A: In the same way that server virtualization allows you to have many isolated “virtual machines” on a single physical machine, container technology allows you to have many isolated application environments on a single operating system. Containers have been around in Unix/Linux for a long time, they just boomed in popularity when Docker came along and made them easy to use. The next version of Microsoft Windows Server (2016) will introduce Windows containers.

    Q: What are microservices?

    A: Microservices are small, independent components of a software application that are built, tested and operated in isolation from the core product. Using microservices enables development to be faster and more nimble than possible when working with a large, monolithic application.

    Q: What is Continuous Integration/Delivery/Deployment?

    A: Continuous Integration is the practice of continuously integrating (and testing) new code with your existing source code. Continuous Delivery is the practice of automating as much of the build/test/release process as possible while still requiring a manual push to production. Continuous Deployment is the practice of fully automating build/test/release all the way into production.

    Q: Are there any conferences or meetups I can attend to learn more?

    A: Yes! The original DevOps conference is DevOpsDays and there are many of these events around the world annually. A popular conference with large enterprises is the DevOps Enterpise Summit.  Various DevOps-related meetups (#CoffeeOps on Twitter) are springing up around the country. Check the CoffeeOps.org website for details, or search for DevOps at Meetup.com.

    Q: [Do I need to understand everything above | Is this all I need] to be or do DevOps?

    A: Absolutely not, on both counts. You can successfully implement a DevOps way of thinking/working by incorporating all or just some of the above, and there are many other tools and practices around delivering value more efficiently. Naturally there are extensive resources available (a few linked in the footnotes) to do a much deeper dive on all these subjects.  

  • Multi-Tiered Escalation Policies are Paramount

    Multi-Tiered Escalation Policies are Paramount

    As any team charged with keeping systems up and running will tell you, having one person on-call isn’t enough. What happens if your on-call engineer sleeps through his alert? What happens if his phone’s battery dies without him knowing, or if he gets an alert at an inconvenient time, like when stuck on a bus or in traffic? It will happen.

    In today’s always-on world, companies can’t afford for that to happen without a backup plan. One or more people should always be waiting in the wings, ready to spring into action if a primary on-call is unable to perform his or her duties to the best of his or her abilities at any given time.

    Engineers who perform backup duty don’t need to be as “on-call” as a primary engineer — no pagers needed here — but what backup engineers lack in readiness they can make up for in numbers. It often makes sense to have multiple backups or an entire team of backups.

    Operationally mature companies often group the currently-on-call engineer and all of his or her backups into an escalation policy. Escalation policies set the order in which each person is alerted and the delays between alerts. While there are many ways to organize these escalation policies, here are some of the patterns most often used:

    Primaries Aren’t Enough; Determine a Secondary, Too
    Most companies have a primary on-call engineer, usually determined by a calendar that designates when members of an ops team are on call.

    Operationally mature companies take their on-call practice one step further and supplement this rotation with a rotation of secondary on-call engineers. Why a secondary rotation instead of just paging the whole team when the primary falls through? If a large team is paged, you get a ‘tragedy of the commons’ effect— it’s urgent that somebody respond to the issue, but not clear how urgent the issue is that each individual person respond. Having a designated secondary removes that confusion.

    The secondary rotation is set up to shadow the primary rotation and generally has the same members as the primary rotation. The difference is that the order of the engineers in the secondary rotation is usually offset from that of primary rotation— if your primary and secondary are the same person, you’re going to have a bad time. For example, if a primary rotation contains Alex, Bob, then Charlie, then a secondary rotation may contain Bob, Charlie, then Alex in that order. Companies often find that if last week’s primary is this week’s secondary, the it helps make sure there’s somebody with recent context who can help if the primary runs into trouble. It is becoming more commonplace for organizations to have both a primary and secondary on-call rotation; but, in an environment where customers expect zero downtime, secondary rotations are no longer an optional tool.

    Enable a Strong Support System
    Larger companies and companies with significant numbers of operations tasks often deploy a separate support team that handles all basic operations tasks before the organization elevates the tasks to an on-call engineer. This team — the first tier support team — usually knows how to handle all of the smaller, repetitive problems that crop up often and have clear resolution procedures. These first-tier teams often offer their support to multiple engineering teams and are placed first in an escalation policy.

    If first-tier teams cannot resolve the issue, escalation policies typically dictate that primary then secondary on-call engineers must complete the task. However, there is one more support network operationally mature organizations must implement: management. Management is ultimately responsible for a company’s systems, and needs to be in the know when severe problems happen— both to hold their teams accountable and to muster additional support. At PagerDuty many of our EPs have a rotating manager schedule at the top of multiple escalation policies, and many of our customers use this pattern as well.

    Put Your Tiered Teams to Work
    In practice, an organization policy for a hypothetical “Database Ops” team might look like this:

    • Assign the incident to the user who is on-call in the First-Tier Ops Team schedule
    • Assign the incident to the user who is on-call in the Primary DBA schedule
    • Assign the incident to the user who is on-call in the Secondary DBA schedule
    • Assign the incident to the team lead
    • Assign the incident to the dev manager

    This escalation policy would unfold over time, with timeouts of between 10 and 30 minutes before each subsequent escalation, depending on a team’s needs.

    Implementing an escalation policy that involves multiple tiers of resolution ensures that devops teams address all alerts — whether business-critical or minor — in a timely manner. Limited downtime is paramount in today’s fast-pace business environment. Customers expect their applications to run smoothly, or they expect to switch providers if devops teams cannot resolve issues fast enough. Organizations will be much less likely to see their processes fail, and much more likely to see happy customers, if they implement a multi-tiered escalation policy.