Author: Dominic Wellington

  • Want a Successful Software Build? Just Say No

    Want a Successful Software Build? Just Say No

    It is a truth which should be universally acknowledged that the most important skill for a customer-facing job in what we laughingly call “information technology” is a cast-iron ability to say no—and make it stick.

    Empty suit

    The misguided desire to please other humans by attempting to deliver on their every request has sunk more projects than any units mismatch or catastrophic misconfiguration—even more than the justly feared Friday afternoon deployment.

    The key to success is an agreed definition of what success is, and how we will all know and measure whether we have achieved it. Let’s take the car as an example.

    French Cars Have Always Been Weird

    These days, we would consider a car to be woefully unfinished if it were lacking upholstered seating for at least four adults, a roof and windows to keep the elements out and some form of in-car entertainment—an FM radio as a bare minimum, probably a CD slot or USB socket as well. This is on top of seat belts, airbags, anti-lock brakes that are probably power-assisted and more lights and safety features than you can shake a stick at.

    Nicolas Cugnot did not bother with any of these features. His design didn’t even have four wheels. It not only didn’t have an airbag in the steering wheel, it didn’t even have a steering wheel. It was controlled with a ridiculous tiller arrangement that unsurprisingly completely failed to catch on. And yet, his vehicle was a success by the stated criteria of the project: It made it much easier for the French army to haul ammunition around.

    Nicolas Cugnot's steam dray from 1769
    Nicolas Cugnot’s steam dray from 1769

    Admittedly, he was also lucky enough to be constrained by the technology available to him. Even Louis XIV might have balked at adding a four-piece chamber ensemble to provide an appropriate musical soundtrack for the driver. But the key point is this: Monsieur Cugnot stuck to his brief, and concentrated all his efforts on the features necessary to achieve it.

    Bringing Apple Into It

    In somewhat more recent times, we have Steve Jobs (emphasis mine):

    People think focus means saying yes to the thing you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas that there are. You have to pick carefully. I’m actually as proud of the things we haven’t done as the things I have done. Innovation is saying no to 1,000 things.

    One reason why Steve Jobs is so famous is that he actually said those thousand nos—often loudly, rudely and profanely. Steve stands out in IT because most technologists have an irrepressible instinct when faced with a question to start to speculate about it, bikeshed possible solutions, build a quick minimum viable prototype and, before you know it, you are armpit-deep in feeping creatures, all of which you are now somehow responsible to nurture and feed. Not being a technologist himself but more of an ideas person, Steve was perhaps better able to take a step back and focus on the end goal.

    Let’s Get Concrete

    Now, this should not be taken as a license to adopt all of Steve Jobs’ methods (Lesson One: Be a complete arsehole to absolutely everyone in your life), if for no other reason that even Steve got fired for it, and from the company he founded at that. When someone comes to you with a request for just one leeeeetle feature, you and I probably can’t get away with screaming “NO!” and slamming the door in the requester’s face. The polite and career-prolonging way to deal with the problem is to ask, “Why?”

    Ideally, in the process of explaining the rationale for whatever the request is, some other solution to the problem will be identified. A good way to improve the odds of this happening is to involve other people in the discussion. Extending the audience in this way creates opportunities for someone to interject helpfully with something along the lines of, “but Carol, we have a perfectly good hammer over there; why are you trying to use a screwdriver to batter a nail in?”

    Peter Falk as Columbo

    Sometimes, you may need to adopt an iterative approach, which sounds impressive, but simply means waiting until the end of the first explanation, nodding thoughtfully, and then asking, “Yes, but why?” Among people who watch more television than me, this is known as the Columbo approach: “Help me understand, what is it you are trying to do here?” and just keep doggedly repeating the question. Eventually enlightenment will arrive—or the requester will get tired and go away, which is also a perfectly valid win condition.

    This method will also help with the tricky situation where the most important question is the one which is not being asked. There is often an implicit question behind that one, where the solution is not to give a glib answer, but to think critically about the proposed or effective solution and ask the questions yourself: “Why are we trying to solve for this?” Or, “What is this thing you are proposing actually meant to do?” And finally, “What was the original problem, anyway?”

    People have a tendency to focus on the issue that is right in front of them, even when the real problem is in an upstream step of the process. They observe or experience something about X as the identified problem, and so default to solving for X directly without considering the broader landscape. This leads to convoluted, inefficient and often high-friction solutions—Rube Goldberg/Heath Robinson machines (delete as appropriate to whichever side of the Atlantic you are on), where a simple connecting pipe is all that is required. An artillery piece will make holes in the ground, but there are a couple of collateral impacts you may want to consider—and maybe offer a spade instead.

    Man standing in front of giant crater
    All I needed was somewhere to plant some flowers!

    Unfortunately even the Columbo Method is not entirely guaranteed to work, because on some rare occasions the underlying request is actually valid and cannot be summarily dismissed. This is when you bring out the big guns and start ranking the urgency of what was already agreed as the scope of the project (a nail hammered into the wall sturdily enough to support that lovely picture) as opposed to what is being requested (a fully motorized picture rail, capable of supporting up to 10 paintings on the scale of Guernica and turning them to optimize lighting conditions throughout the day while also keeping them out of direct sunlight). Nobody is saying that the second option would not be nice, but it certainly won’t be done in time for supper.

    And that, children, is how you say No. Go forth, and do ye likewise, and may all your projects be huge successes. And, most importantly, not require me to come in and clean up after you.

    — Dominic Wellington

  • A Vendor Guy Goes To Monitorama

    A Vendor Guy Goes To Monitorama

    I attended my very first Monitorama this year, in Amsterdam, and I have thoughts!
    Monitorama Amsterdam
    Monitorama Amsterdam

    First of all, it’s deeply weird speaking at Monitorama when you work for a vendor in the monitoring and observability space, but you’re not there as a sponsor. This would be pretty odd at the best of times, but the crowd at Monitorama are next-level. They are very comfortable with highly technical content, so they get irritated very quickly with (what they perceive as) marketing fluff. “Down in the weeds” doesn’t even begin to describe it; most people had never heard of the sorts of things that are part of life in enterprise IT (AIOps, Netcool, Patrol or any of our usual suspects), but, on the other hand, everyone I met could engage in a multi-hour argument over which monitoring tool is worst.

    In fact, I would say that one of the highlights of Monitorama was the conversations in the hallway track. This is where all the Twitter loudmouths—I mean, “influencers” and “thought leaders”—who drive this industry congregate. If you manage to get into a conversation with them over coffee or beer, both of which were flowing freely, there can be a real exchange of ideas. I know I learned probably as much from those small-group chats as from the presentations on the big stage. I hope I was able to share some perspective in return.

    Heck, as ridiculous as it sounds, even the dress code was a challenge. At the events I attend as a vendor, there are basically two speeds: full suit-and-tie battle dress or a T-shirt with my employer’s logo. Neither case requires much thought when packing. Monitorama, though? I’m no longer credible in my old Thinkgeek shirts from my sysadmin days, but I’ll be laughed out of the room if I show up in a tie.

    Preparation

    I had answered the Call for Papers with a pitch about AI for distributed systems. Here’s my abstract:

    The IT industry is in the middle of one of its regular swings between centralization and decentralization, driven by increased automation of the network fabric itself, as well as new use cases such as IoT. With more and more processing and autonomy devolved to the edges, old assumptions about how to manage and operate a network have to change. It no longer makes sense to try to forward all the edge alerts to a central location for analysis, but central visibility on the health of the network is more important than ever.

    New techniques in AI-enabled observability hold the promise of helping NOC teams deliver better experiences for users of their networks, without requiring excessive manual work or heavy impact on on Operations teams’ personal lives. Machine learning enables knowledge to be captured and made available in context when it is most useful, accelerating incident resolution and improving user experiences.

    My first hurdle was that, although I have public-speaking engagements at least once or twice a month—and have done for some years now—I normally appear under the flag of my employer. This fact has both pros and cons. On the positive side, I can usually rely on a substantial library of slides that I (and others) have created in the past. I have some known high notes that I will be aiming to hit, and the call to action at the end is pretty explicit (“try our stuff”). The other side of the coin is that, because the audience is aware of these factors at least at some level, I have to work that much harder to earn their attention, as they assume they are being sold to. As a vendor spokesperson, I am also somewhat constrained in how wacky and off-the-wall I can get—although I do always try to push the boundaries. At one speech last year, I talked about bicycles for several minutes, to the consternation of the organizers, until they worked out what I was building up to.

    So I slaved away for some time over a hot Keynote, watched videos from past Monitoramas, consulted people in the industry whose opinions I respect and rocked up in Amsterdam pretty pleased with myself—to the point that I stayed out at In de Wildeman perhaps later than I should have on the night before Monitorama kicked off.

    Not Enough Preparation

    As speaker after speaker took the stage on Tuesday, I sank deeper and deeper into my seat, realizing that I had completely missed the mark—and as if that were not bad enough, several key analogies in my presentation had already been used by the first-day speakers. I did go to the party at Delirium Café, but left early to sleep the restless sleep of the nervous speaker. In the wee hours of the morning, I gave it up as a bad job, and spent several hours basically rewriting my entire talk in the light of what I had seen. Goodness knows what my neighbors thought of me rehearsing my lines, although I tried to keep the noise down.

    Here is the result in Slideshare, or you can watch the video here. That link should take you to the correct time stamp, but if not, scroll to about 5h30′ in.

    Katja Budnikov was also doing amazing sketchnotes (live!) of all the presentations. Here is the one from my session:

    Relief

     

    In the end it seemed to go down pretty well. I did my level best (honest!) to call on my own now-obsolete experience as a practitioner and on conversations with people who are more current than I am with the nuts and bolts of modern IT, rather than relying on the crutch of vendor boilerplate, and I think I was mostly successful in that.

    If you are working for a vendor and considering submitting to the Call for Papers at this sort of event, whether it’s Monitorama, DevOpsDays or Config Management Camp, I would definitely suggest test-driving your content as widely as possible. You probably have a lot of subconscious reflexes as a speaker that are not appropriate for the sort of audience these events attract. Even when you are conscious of them, it’s hard to eradicate them completely, but do your best.

    In the other direction, while I have enormous respect for the technical level of Monitorama attendees, some of them seemed to be at risk of losing track of the big picture. I would not want Monitorama to change its character as a techie’s conference, but I am grateful that we had a few presentations outside that mold, and I hope that I was able to provide some context for how all the wonderful technology gets used out there in the big bad world.

    — Dominic Wellington

  • Ops: The Other DevOps

    Ops: The Other DevOps

    Much of the conversation around DevOps, including right here on devopsy.kinsta.cloud, is about helping developers get easier access to production systems, increasing agility by shortening release cycles. Most of the time, this focus on developers leads to a concentration on so-called “Day One” tasks, the Dev side of the equation, to the exclusion of “Day Two” operations—the Ops side of things.

    Cows in a field with farmer
    Photo via Damien Kuhn – https://unsplash.com/@producteurslocaux

    Recent moves in the DevOps market underscore the fact that the weight is still very much on the Dev side, and not so much on Ops. Microsoft buying GitHub is huge news, don’t get me wrong, not least because many people are still stuck in a very Nineties conception of Microsoft, and this acquisition should help clarify that this is not your parents’ Microsoft (full disclosure: my employer, Moogsoft, has a strategic partnership with Microsoft). News on the Ops side of the house, meanwhile, rarely achieves the same prominence.

    The reason for this imbalance is to be found in DevOps’ origin myth. The story is all about developer frustration with risk-averse operations procedures which prevented them from deploying as rapidly and frequently as they would have liked to. As with most good stories, there is a kernel of truth here, but the truth is more complicated. Ops people are risk-averse for good reason: When they mess up, they don’t have a safety net, and they get a lot of very public blame.

    That said, it is certainly true that a reasonable aversion to risk can be taken too far. Couple that with a very human tendency to define oneself by one’s job, and you wind up with people putting hands to keyboards for even basic, routine tasks, becoming bottlenecks for huge processes.

    What Does DevOps Mean To Ops?

    The biggest benefit of DevOps from an Ops perspective is that it has forced that hard conversation about moving from artisanal, handcrafted IT infrastructure, to industrial and scalable models that don’t require a human in the loop for routine operation. This cultural shift began with the famous pets-versus-cattle analogy, but the transition is far from being complete.

    A big sign that there is still work to be done is the relative lack of sophistication around tooling  in IT Operations. There is a lot of talk about software-defined infrastructure and desired-state automation, but relatively little practical uptake, especially when it comes to operating such frameworks day-to-day. These new models will require a wrenching transition from traditional Ops models, with pervasive automation working to augment human sysadmins’ capabilities.

    This is not the old style of automation, with cron jobs and shell scripts, which any BOfH worth their etherkiller has been doing for decades. The new serverless frameworks do away with the last vestiges of persistent coupling of infrastructure components to individual business services. What vMotion promised in 2003 is now coming to pass, with workloads moving, growing and shrinking to support changes in user demand, and infrastructure reconfiguring itself top to bottom in response.

    Key Questions To Ask About Operational Automation

    In this context, operational automation can no longer work on the old model of “set it and forget it.” Every process set in motion needs real-time contextual awareness of what is going on around it and how that is constantly changing. Without that visibility, backup scripts may go on faithfully backing up what used to be a high-priority server, but is now a backwater of the application infrastructure. Meanwhile, the cloud services where the bulk of the application’s logic runs are not meaningfully integrated into the disaster recovery (DR) plan.

    All of these facets need to be reviewed regularly, in terms of need, function and ownership.

    Who Needs This Automation?

    Anything that runs should be linked to some kind of business requirement, however indirectly. “This ensures that our infrastructure can scale and flex seamlessly in response to user demand”: Good. “This keeps our operating systems patched”: A valid goal, but do we need to do it in production if the average lifetime of provisioned operating systems is shorter than the patch cycle? Just patch the templates and you’re done. “We’ve always done it this way, and I’m not sure what might happen if we stop”: Time to set some chaos monkeys loose, and see what happens.

    What is the Function of This Automation?

    We are in the middle of huge technological transitions, and the waves of innovation keep coming closer and closer together. As an industry, we had just about come to terms with virtualization by the time cloud computing came along, but that was far from settled when containerization got underway, and it looks like that may be undermined by serverless architectures before it even comes to its full fruition. Watch out for hidden assumptions in terms of the way you go about a task, and check back in regularly. Ops types should take a leaf from Dev’s book, and regularly consider refactoring their automation to remove cruft and take advantage of new developments.

    Who is the Owner of This Automation?

    Someone has to own every piece of automation, so that they can maintain it over time according to the previous two points, and so that everyone else knows who to call if (or realistically, when) it breaks. Again, developers have been talking about technical debt for a long time: the impact of older, unmaintained code. Meanwhile, sysadmins used to pride themselves on the uptime of individual servers. That attachment needs to move from the individual server to the overall infrastructure and the automation and management that develop over time to ensure its availability and performance. They’re cattle, not pets, remember?

    Ops is the Other DevOps

    As Ops becomes more and more about managing the automation, the distinction between Dev and Ops starts to blur. With Dev roles “picking up the pager” and taking responsibility for production operation of their software, Ops roles develop more and more code to fulfill the promises of those applications. With traditional Ops tasks evolving or even disappearing, new roles will appear. The job of automation architect will become much more relevant than any job that requires someone to ssh into an individual server or router. In this transition, old-school Ops people can learn a lot from what our Dev colleagues have been up to over the last few years.

    One day, the term DevOps will no longer describe developers and non-developers, but two different types of developer, one focused on business logic and one on architectural underpinnings.

    — Dominic Wellington

  • From One-Size-Fits-All to Custom-Fit for IT Ops

    From One-Size-Fits-All to Custom-Fit for IT Ops

    The downfall of the all-in-one IT management suite and the rise of the customized IT Ops toolchain

    Here is how enterprise IT Ops used to work: you would make your choice among the Big Four vendors and buy in (quite literally!) to their strategy. This meant a suite of products that came preintegrated (more or less) out of the box, and—at least on the sales proposal—covered all of an IT organization’s needs across a huge spectrum of activities.

    selection of tools hanging ready to use
    Photo by Barn Images on Unsplash

    Broadly speaking, each of the Big Four’s suites included some flavor of the following: monitoring, ticketing, configuration management and asset tracking and automation and orchestration. Some vendors were stronger in one area than in another, but unless you had pretty specific needs, each vendor could be your one-stop shop.

    Development and improvement inside these suites was mainly incremental and occurred at a fairly leisurely pace. Meanwhile, startups with bright ideas would come along occasionally and challenge the incumbents in specific areas, only to be bought by one of the mega-vendors if they showed enough promise.

    There was a symbiotic relationship between vendor and IT buyer, as IT professionals built decades-long careers as experts in this or that vendor’s technology stack, with the wall of framed certifications to prove it. CIOs were reassured by the fact that there was a readily available pool of experts to hire, and continued reinvesting in the next iteration of their chosen vendor’s stack.

    The Old Model No Longer Fits Modern IT

    In the last decade or so, this comfortable status quo has broken down. The cracks were already showing before then, but only the most agile and forward-thinking organizations were moving fast enough to trip over them. As more and more businesses began to expect greater and greater responsiveness and speed from their IT departments, the huge sprawling suites of the mega-vendors were no longer able to keep up with the required rate of change. Leisurely 18-month release cycles no longer cut it in an agile world, where that time span may encompass several entire strategies.

    In the old world, to call something “best of breed” was a sniffy dismissal of a one-trick pony, as opposed to the more supposedly versatile offerings of the Big Four. In the new world, it became obvious that aside from a handful of tentpole products that were genuinely good and fit for purpose, most of the individual components of the big suites was distinctly unexciting. Worse, the main value proposition of the suites—that they came preintegrated out of the box—started to break down as well. Many IT professionals were still fiercely loyal to individual products, but they became more willing to pick and choose, rather than buying into the whole vision.

    In parallel, the rise of simple integrations—primarily REST APIs—made it easier than ever before to connect disparate solutions together into a single IT Ops toolchain. Worse (from the point of view of the incumbent vendors), sometimes it was easier to connect modern point solutions than their own component pieces, weighted down by the baggage of decades on the market.

    How to Deliver the Flexibility and Agility the Business Expects?

    These days, it is becoming commonly accepted even among the most staid and risk-averse organizations that the way to go is no longer to align to a single IT vendor and trust them to deliver on their vision, but rather to design custom-fit, multisolution toolchains, including home-grown, open source and commercial tools.

    agile jump
    Photo by Max Bender on Unsplash

    But what to do if you find yourself dealing with the consequences of this sort of IT monoculture?

    Look Back At How You Got Here

    The first step is to re-evaluate past buying decisions and strategic alignment. It is very important to do this without assigning blame; it was a different time, and those choices may well have been the right ones for the prevailing environment at the time, when integration between tools from different vendors was legitimately difficult.

    Don’t Just Throw Everything Out

    In many environments, too, the correct solution may well not be a clean sweep. One or more of those tentpole products that kept the overall suite looking good past its sell-by date may well still be perfectly fit for purpose and not worth the hassle of swapping out. However, other components are probably legitimately lagging behind the state of the art, in which case it is worth replacing them with something more fit for purpose.

    Count Costs – All Costs

    Include cost in these calculations, but remember to include all costs. Maybe you or a predecessor signed a big enterprise license agreement (ELA), so Product X is “free” in that context. But what does it cost to keep Product X running, in terms of ancillary licenses, hardware requirements and, last but not least, skill and effort on the part of the IT administrators? I don’t want to name specific products or vendors as examples, to avoid the discussion devolving into particulars, but it’s worth bearing in mind that many legacy products have shallow talent pools—not that many people are available on the market, and many of those people are coming to the ends of their careers, which makes them both expensive and reluctant to change employers. Instead, up-and-coming technologies often have much deeper and wider talent pools, with many more experienced specialists available at all salary and skill levels. If you don’t believe me, try hiring half a dozen COBOL programmers and half a dozen Node.js coders, and then we’ll talk …

    IT Ops hard at work
    Photo by Giu Vicente on Unsplash

    Your IT Ops Toolchain, For Your IT Ops Needs

    Moving from one of the big all-in-one suites to an assemblage of different tools, all flying in loose formation, is not easily done. Any technology transition of this sort is a Project with a capital P, and needs to be planned and managed with due care and attention. However, there is a large payoff at the end in terms of increased agility and flexibility within IT Ops. In turn, this affords IT the ability to offer their business counterparts the sort of responsive and even proactive IT support that moves IT from being a cost center, on a par with Facilities, to a strategic differentiator that enables new business initiatives.

    Business as usual doesn’t cut it any more, and assumptions that used to be valid have fallen apart. Decisions based on those assumptions need to be re-evaluated. New, more flexible approaches will set up IT Ops for success in the newly flexible world we are already living in.

    — Dominic Wellington

  • Sneckdowns In IT Operations

    Sneckdowns In IT Operations

    What IT operations professionals can learn from the actual paths people take through tools and processes, and the differences from the expected usage model.

    It has been a snowy winter across much of Europe, and although the snow is now beginning to melt in most places, there are still substantial sneckdowns to be seen across our cities.

    If you’re not familiar with the term “sneckdown,” it describes those areas of streets that are not actually used, as revealed by the unmarked snow that settles on them.

    Sneckdown example

    Transportation planners have even been known to use sneckdowns successfully as planning aids.

    The world of IT should learn from this example. We may not have anything as whimsical as snow to help us, but on the other hand, we do have enormous amounts of data. Let’s investigate some examples, starting with IT operations.

    Restate My Assumptions

    IT operations has a bunch of assumptions baked in, but just like with street layouts, some of them may be obsolete, leftovers from a previous generation of technology. My own hometown dates its founding back to Roman times-218 B.C., to be precise-but the sorts of design choices that Roman colonists might make to facilitate chariot and ox cart traffic have had to be updated significantly-first to accommodate motor vehicles, and then again to ensure that the motor vehicles did not crowd out other forms of transportation such as bicycles.

    Similarly, much of IT operations is based around the notion of a ticket, which describes and encompasses an atomic problem. This ticket has a single owner at any one time, and is processed sequentially through a series of steps. This is the operations equivalent to the waterfall model of development.

    These assumptions used to be reasonable in previous eras of IT technology, when a simple failure of a single piece of equipment was enough to trigger an incident. That incident ticket could easily be paired with the specific piece of equipment that was impacted, and from there a simple lookup would return the responsible group, who could then be assigned the ownership of the ticket.

    These days, that model is largely obsolete. Hardware, especially virtual hardware, is cheap enough to be made multiply redundant. Automated functions such as load balancing ensure that single failures are rarely enough to cause an actual incident. Instead, multiple failures have to occur in the right sequence—or the wrong one, depending on your point of view. Meanwhile, operators are drowning in a sea of red alerts, unable to find and focus on the few actionable ones because processes require them to respond to everything. Finally, when a real ticket does make it through the noise, it gets bounced around from team to team, as harried operators try to work out whether it’s their problem or somebody else’s.

    Operators are human, after all, and over time they develop their own paths through and around this rigid process, just as people walk and drive their own paths through the snow.

    Examples of IT Operations Sneckdowns

    Users Working Together

    One big IT operations sneckdown is collaboration. Instead of using the ticketing system and its rigid assumptions, operators work around it, using alternative methods to work together and share information with each other. The goal is to deliver full-stack operations for the entire complex business service that is being supported by the team.

    The problem with sneckdowns is that if there is no city planner watching and measuring, eventually the snow will melt and leave no visible trace of the paths that people take around their city. In the same way, the risk to the wider organization is that as people work and collaborate in alternative, unofficial tools—whether IRC or Slack—information gets stuck there and does not make it back into the official knowledge base, where it is available to be consulted and reused over time.

    Sneckdown Recommendation: Recognize that the nature of modern IT and network architectures means that there may be no single owner of a problem. The rapid rate of change, too, makes it almost certain that any owner will not be documented in a static database somewhere. Instead, modify processes to recognize this multiowner approach, and enable users to work easily with each other without having to play ticket ping-pong.

    The Right Tool for the Job

    This overlaps with another sneckdown: unofficial tool adoption. In the same way that operators might start using an unofficial collaboration tool to route around limitations in the official service desk process, they might look at Kanban to share task status, or start building libraries of automated diagnostic and remediation actions. The problem is that the rigid official policy often does not accommodate these extensions, and indeed has no mechanism to do so.

    Sneckdown Recommendation: Modern operations tools need to offer flexible interfaces to enable integrations and extensions to be done in a safe way. This way, if a particular integration does not work out or needs to be upgraded, that can be done without disruption to the wider process and to any other integrations that are working well.

    Look into the Shadows

    The final sneckdown—at least for this snow storm—is shadow IT. In the same way that much of our universe is theorized to be made out of dark matter, not visible to our current tools, a growing proportion of IT spending is happening outside the “official” IT budget. This also means that whatever is bought with that budget is outside the official IT operations and support process. That’s all very well on paper, but if the tool Sales or Marketing were relying on to do their jobs goes down, somehow, IT operations is still going to get the call to fix it.

    Sneckdown Recommendation: There isn’t an easy short-term fix for this one—no way to shovel it out of existence. Instead, IT operations professionals need to work on this issue from two angles. The first is to address the causes of shadow IT, those rigidities and frictions that make people look for outside solutions rather than engage with their in-house IT departments. When everything is properly aligned, the IT department has a lot to offer in terms of understanding of How Things Are Done inside the organization. The other angle is—yes—to make it easier to accommodate solutions that might have been procured originally as shadow IT. Figure out how to engage with support processes for SaaS solutions that people adopted on their own; if it’s working for them, it’s up to you to keep it working! The benefit is that this alignment will reduce the need for constant injections of energy by sysadmins and operators to keep things running: the Second Law of IT Ops.

    Conclusion

    Compared to city planners and transportation activists, we IT professionals actually have it pretty easy. We’re not reliant on particular weather conditions for our data, but rather, we have enormous amounts of data at our fingertips all the time. It’s up to us to look at the data with open minds and read the conclusions that are already there. The hard part, of course, is putting those conclusions into action, especially when that requires changes to widely accepted processes, but that is also an opportunity to revisit those processes and make sure they are still as fit for purpose as when they were first adopted.

     

    Game Day Chicago

    If you’re interested in what that process looks like, there is an interesting event series: Game Day ITSM & Incident Management Workshop. The next Game Day is in Chicago on March 21, but there is a whole series running, so there is probably an event near you soon. It’s a fun time, and also a great opportunity to rethink some assumptions and look at where some sneckdowns might be that could be incorporated into your planning.

    — Dominic Wellington

  • Focusing on the Real Value of AI for IT Ops

    Focusing on the Real Value of AI for IT Ops

    In all the hype over AI, let’s concentrate on the real value it can provide, especially for busy IT Ops teams.

    2018 is already shaping up to be the year when AI and machine learning (ML) go mainstream—at least in terms of public awareness, if not yet of actual usable products. As I write this, the annual CES event—the show that is, Prince-style, “formerly known as “The International Consumer Electronics Show—is in full swing. The usual suspects are out in force, with Amazon’s Alexa getting embedded in more and more products and Google also showcasing its own consumer AI capabilities.

    More unexpected AI adopters included Ferrari, but there were also some notable failed demos, with an AI-enabled suitcase repeatedly running away from its owner or AI refusing to answer questions during an on-stage demo. All of this led the BBC to question, “When will AI deliver for humans?”

    What Happens in Vegas, Stays in Vegas

    Of course, much of what comes out of CES is frivolous, but there are all sorts of other applications of AI and ML. The problem is that whenever there is this much hype, disappointment and disillusionment are not far behind. Gartner’s famous Hype Cycle model can help here.

    Gartner Hype Cycle - from Technology Trigger, via Peak of Inflated Expectations, through Trough of Disillusionment, up Slope of Enlightenment, to Plateau of Productivity.
    By Jeremykemp at Wikipedia, CC BY-SA 3.0

    The conventional concern is that overly optimistic projects initiated at the Peak of Inflated Expectations risk failing, leading to excessive caution and fewer projects starting during the Trough of Disillusionment, before a more realistic understanding sets in and the technology achieves true mainstream adoption. Due to the nature of AI and ML, though, there is a second failure mode that is far more insidious.

    AI and ML generally operates as a black box. Results are provided, very quickly and without the need to develop deterministic rules, but it is generally not possible to determine why a specific result was generated or troubleshoot failures in detail. As these techniques are used in the real world, potential problems that might not have been obvious in the lab begin to appear. If your smart fridge misreads the expiry date on that chicken (and you don’t think to check yourself), you might be in for an uncomfortable few hours. If your smart thermostat decides for some reason that your house should be as hot as physically possible, you’re still on the hook for the heating bill.

    Those are the “nice” failure modes. If a facial recognition algorithm was trained on a data set that didn’t have the right racial balance, you might wind up with some of your employees not being able to get in the door, or some of your customers not being able to unlock your device, or whatever.

    It gets worse. Law enforcement has an obvious interest in facial recognition—but what if a faulty system places you at the scene of a riot? https://www.liberty-human-rights.org.uk/news/blog/misidentification-and-improvised-rules-we-lift-lid-mets-notting-hill-facial-recognition Even if you can prove your innocence, you might still have to deal with some difficult conversations with your family and your employer, and your neighbours may never forget the time you were hauled off in handcuffs.

    Don’t Throw The AI Baby Out With The Hype Bathwater

    So should we just forget about this AI nonsense? Far from it. In fact, I am personally convinced that AI is very much a key part of our future. It just needs to be used in the right way.

    I mentioned training data above, and that is the key. AI systems need a lot of good data, and they need to be trained to make good assessments. Out there in the real world, both sides of that problem are actually very hard. Gathering enough data, making sure that the data gathered are actually representative of the problem space, dealing with privacy issues around personally-identifiable information, and formatting the data correctly for ingestion by your machine learning system are already not trivial issues. However, even assuming that you have successfully dealt with that hurdle, the initial results will not be very good or reliable. Intelligent assessment by humans who understand the domain that you are training for is key if your deep learning model is not to go off the rails.

     

    Come Up to the IT Ops Lab

    Seen in these terms, IT Ops is an ideal domain to which to apply these techniques. First of all, IT infrastructure already generates huge amounts of event data – Big Data, if you insist – and all of those events are already, by definition, formatted to be easily machine-readable. Sure, there are many different formats out there, from SNMP, via various proprietary product-specific formats, to modern REST APIs, but it turns out that computers are pretty good at translating information from one representation to another.

    Once the event data are in the AI system, it can apply whatever algorithms to try to make sense of them: which events are even significant, which ones are related to each other, what other relationships might exist, and so on. When it comes to evaluating the results, again, IT Ops already has people, tools and processes working on precisely that question: is this a real issue, and if so, what to do about it — or if not, why not. This evaluation can be fed back to the learning system to improve its results for next time, iterating over time to provide ever more valuable and reliable results.

    This learning can even be applied to the evaluation process itself, watching what users find most interesting and valuable, and extrapolating from that to present the most significant data first, or looking for patterns in human interaction to assemble the most effective teams to work on particular types of incident.

    This is what AIOps is all about. Don’t assume that the sort of consumer-grade AI that you see at CES is all that is going on in this field. AI has huge potential to help streamline enterprise IT Ops, by presenting human specialists with actionable events, helping them collaborate more effectively, and learning and improving over time.

    Once the CES hype has died down, take a good look at AIOps and start to think how it can fit into your plans for 2018.

    — Dominic Wellington

  • Now is the Time for Full-Stack Operations

    Now is the Time for Full-Stack Operations

    The idea of “full-stack developers” is becoming a fairly common one, although it is still widely contested. But what about full-stack operations?

    There are cycles in the IT industry, and if you wait long enough, you can see the same ideas swing in and out of favor. Business applications used to be fairly monolithic, and so it was expected that their programmers would have a good understanding not just of the application’s code and business logic itself, but of its underpinnings as well.
    Stacked stones

    Over time, different parts of those underpinnings were separated, breaking out data silos in a process of modularization. With the increasing complexity of architectures, the expectation became that specialists would be in charge of each module and simply ensure that standard interfaces were available to teams whose modules needed to interact with each other.

    While modularization at the architecture level is here to stay, there is now some push back against it being extended too aggressively to the hum

    an developers in charge of building the system. This takes the form of a quest for “full-stack developers,” who are able to understand all the nuances of the entire application stack.

    There is a similar need on the IT operations side of the house. Different specialized support teams used to be able to focus on one particular area (server, network, database, middleware and so on) and use their own specific tools to analyze and understand the performance of one particular module of a complex enterprise architecture. The assumption was that if each module was up, then ipso facto, the entire stack was up, and therefore the application as a whole was also up.

    As it turns out, it’s not that simple.

    Let’s Do Some Root-Cause Analysis

    These days, failures of business services rarely are caused by a catastrophic outage of one particular component. Hardware is cheap, resiliency is built in and experience from web-scale environments has trickled down to become common best practices. What actually causes problems—such as getting you embarrassing news coverage or angry customers demanding refunds—is the unexpected: many seemingly minor and isolated issues that suddenly add up to a big problem.

    When this sort of cascading failure occurs, modular operations teams and processes are not able to cope. Multiple tickets are opened for each affected module and routed to that module’s support queue. Each of those tickets contains only partial information, by definition, and may even describe only a symptom of a failure in another module. Because of this, precious time is wasted diagnosing, reassigning and escalating those tickets, until eventually a complete picture of the actual problem is laboriously assembled.

    At this point, representatives of all of the various affected support teams assemble, either in person or on a conference call, and try to determine what to do to restore service as quickly as possible. All too often, these efforts are further delayed by unproductive blamestorming and finger-pointing, but eventually, the issue does get resolved.

    Given the complexity of modern-day enterprise architectures, it is impractical for any single IT operations specialist to understand the entire stack in depth, including all the interactions between different modular components. Even if it were possible to do so for a snapshot in time, these days most environments are self-modifying to a greater or lesser degree, provisioning or decommissioning capacity in response to demand, balancing load across available resources and performing routine maintenance tasks. This sort of dynamism is impossible for humans to model accurately in real time.

    How Is It Possible To Do Full-Stack Ops?

    No single person can be a full-stack ops specialist, but that does not mean that the whole team cannot work to deliver full-stack support. The key is to break down the walls between the different siloed teams supporting each module, and make it easier to build an overall picture of, yes, the full stack that is supporting a particular business service.

    This bringing-together must happen in real time, because that is the pace of business today. An outage that has a technical duration measured in a handful of minutes may cause hours-long business repercussions. Ops teams need tools that can relate signals from different modules into a comprehensive picture of what’s going on in the IT environment, proactively inform the right people, and give them the support they need to pool knowledge and work together effectively to resolve the outage and restore service to affected users.

    These tools cannot be simple incremental improvements in the monitoring of one particular module, important as that is. The responsiveness that the organization as a whole expects from its IT operations requires that silos be broken down and information and expertise be pooled. Real-time collaboration between specialists in different domains is the only way to deliver truly full-stack operations.

    The rapidly maturing discipline of AIOps aims to do just that. The beauty of this approach is that, unlike rip-and-replace approaches in the past where moving to the new thing meant ripping out the old thing, AIOps takes advantage of what was done in the past, simply joining together what were previously disconnected islands of knowledge and making information available when it is needed.

    Gartner Market Guide for AIOps PlatformsTake a look at some of the players building out these capabilities in the AIOps Market Survey from Gartner. Normally these documents cost $1,295, but you can get it for free courtesy of Moogsoft at this link.

    — Dominic Wellington

  • Measure What’s Important: DevFinOps

    Measure What’s Important: DevFinOps

    Lately there seem to have been quite a few DevSomethingOps terms being coined. However, I do think we need one more: DevFinOps.

    Dev-What-Ops?

    One of the biggest problems in IT is working out the cost of IT. Left unchecked, IT will expand to fill whatever budget it is given. This may mean evermore involved business continuity and redundancy, reinventing evermore elaborate wheels, irrational versus buy decisions and so on.

    In fact, the whole idea of cost estimates for IT projects has become something of a joke. Most IT projects are reputed to exceed their budgets.

    There is an entire #NoEstimates movement that rejects the idea that it is even possible to deliver a sensible estimate for a software development project. As is often the case, the basic idea is reasonable:

    it is possible to do small chunks of work incrementally, leading as rapidly as possible to a desired shippable product, and that when you do that there is no need to do much of anything in the way of estimating stories or the project.

    However, the simplistic interpretation that is all over Twitter is that because there is no estimate of the  overall project, there is no estimate at all. That is not the idea! It is simply to break down the overall estimate (which is probably wrong) and instead budget for much smaller chunks of work, which can presumably be estimated more precisely and accurately.

    So What Can We Do?

    Maybe what is required is integration of financial concerns directly into the fabric of IT development operation. We might call this DevFinOps, in line with other extensions of DevOps such as DevSecOps or DevNetOps.

    This is not such a leap; after all, we already integrate costs in areas such as SLA calculation, estimating the TCO of a tool or the ROI of a proposed purchase. However, these are generally calculations that are made after the fact, and may even be retconned to fit external assumptions.

    What I am proposing, though, is to build this information right into the IT metadata and processes to enable us to make better decisions.

    This is not IT Asset Management, although it would build on those data and extend their reach. A particular device or piece of hardware has its purchase price and current amortised value documented in the ITAM database, but the actual value it has at any given time might vary depending on which business function it is currently supporting. This calculation should change over time to reflect the real current value it is delivering to the organization at any given time.

    An idle server has no particular value beyond the intrinsic value of the hardware, back-end support (floor space, network, cooling, etc.), and perhaps any associated licensed software. However, the same server will take on a much higher value to the organization when it is provisioned as part of the infrastructure supporting a major business-critical application.

    Performing actions on these systems would therefore have different potential impacts, and should be treated as costs deducted from a limited budget. Live-patching a business-critical server is an expensive operation, not to be treated lightly. The cost of doing so should be deducted from an operational budget.

    This calculation has the benefit of making it easier to communicate the different impacts of various proposed courses of action outside the IT department. Deciding between cloud and on-premises deployment options, or between different flavors of cloud, can be a fraught process. Technical options and pricing models are rarely easily comparable, and then there are other costs such as compliance (yes, and GDPR) or vendor risk. Quantifying and normalizing these disparate factors would make it that much easier to compare options up front and calculate benefits after the fact.

    Is This Even Possible?

    Google already does something like this in its Site Reliability Engineering (SRE) program, with the notion of an error budget:

    The business or the product must establish what the availability target is for the system. Once you’ve done that, one minus the availability target is what we call the error budget; if it’s 99.99% available, that means that it’s 0.01% unavailable. Now we are allowed to have .01% unavailability and this is a budget. We can spend it on anything we want, as long as we don’t overspend it.

    There is a long and sordid history of attempts to make programming projects comparable and their estimates more reliable. Often this devolves to simply counting lines of code, according to the age-old management syllogism: “We need to measure something; this measurement is easy to gather; let’s measure this!” and of course the Mythical Man-Month.

    Fortunately, there is a way to calculate the complexity, and therefore the potential cost impact, of a piece of code: the Halstead volume. This is a measurement of the complexity of code, described in terms of the number of distinct operators, the number of distinct operands and the total number of operators and operands. While it is not guaranteed to be perfectly accurate in every case, it is at least accurate enough to be comparable, and therefore can be used to estimate the risk of a change, in terms of the relative Halstead volumes of the code to be changed and the patch to be applied.

    But What About the Squishy Meatbags?

    Connecting the human costs would also enable other metrics to be gathered. It has been historically difficult to differentiate between what is urgent and what is important, and how to prioritize each. If we connect salary data (including overtime) to changes, it becomes much easier to make the call on when enough time has been wasted on a problem and the cost-effective solution is to start from scratch.

    It sounds like a lot of work to gather all of this information, but these data already exist; they’re just not connected up in one place. When the business wants to launch a new service, there are (usually) pretty rigorous calculations behind that decision. The expectation is to acquire or retain a certain number of customers who have a certain average lifetime value and break-even points and other criteria are all mapped out ahead of time. For whatever reason, though, these islands of information are rarely if ever connected up.

    This, incidentally, is just one reason why it can be worthwhile to work with reputable and experienced vendors. I have written before about what this process looks like. The good vendor sales teams will help to unearth this sort of information—but even in the most positive sales processes, the kimono is rarely fully open. There is far more information inside the organization that could be enormously useful if only it were available in the right place and at the right time.

    Longer term, this would mean that business cases and ROI calculations would finally be concrete and factual, rather than extended performance pieces in the medium of spreadsheets. On that note, there is a wonderful little book, titled “How to Lie with Statistics,” which is even available as a free PDF. It’s one of those books that is fun to read because you start spotting the things it was talking about simply everywhere.

    If you wanted to be maximally Zeitgeist-compliant, you could probably verify all of this with blockchain, although I worry that the transaction costs (including thermal costs!) would be prohibitive. The good news, though, is that you could use this DevFinOps methodology to work that out too …

    All That Effort, for What?

    Bottom line, the goal is to be able to calculate (at least within reasonable limits) how much IT is costing today, and what the impact of any proposed changes would be—and crucially, to do so in real time.

    The goal is agile cost estimates. There is a reason I chose the DevFinOps coinage. The way we do cost estimates today is very “waterfall“: large teams have extended arguments over spreadsheets, and the process is so painful and expensive (in person-hours and disruption) that it is only undertaken for the largest changes.

    Agile’s faster release cycles do not align at all well to this process, which is where #NoEstimates is coming from. However, if we start gathering and processing all the cost data in real time, and make it available via a simple API call, it becomes possible to calculate both ends of the cost of a proposed change: the complexity of the code itself, and therefore the probability that it contains bugs, and the infrastructure impact that the deployment will have. This cost can then be compared to the team’s current SRE error budget to determine whether the change can be made automatically or whether it requires signoff as an exception.

    The result looks a lot like ITIL’s notion of a “standard change”, but without the inherent conflict between ITIL’s centralization and SRE’s decentralization. The key is a common accepted measurement of what a standard change actually is, which would otherwise require the ITIL Change Manager to make the determination ahead of time.

    What Does This Look Like in Ops?

    Let’s not forget Ops, either. Proper cost estimates like the ones described above would enable Ops to prioritize alerts correctly and estimate impacts accurately. Two outages with equal technical severity may have very different business impacts. Today, that reconciliation relies on tribal knowledge and systems that are often inaccurate or out of date. Given a central shared repository of data that describes the value of deployed systems, it becomes possible to calculate those impacts much more accurately.

    While we work to get to full DevFinOps, you might want to consider AIOps, which helps gather a contextual view of what is actually going on in IT. Gartner just released a Market Survey for this emerging space. Normally these documents cost $1,295, but you can get a free copy here, courtesy of Moogsoft.

    Dominic Wellington is Chief Evangelist at Moogsoft. He has been involved in IT operations for a number of years, working in fields as diverse as SecOps, cloud computing, and data center automation.

    — Dominic Wellington

  • AIOps: Managing the Second Law of IT Ops

    AIOps: Managing the Second Law of IT Ops

    Or, why IT systems keep breaking

    The tension between the two halves of DevOps is long-standing, going back long before the term was even invented. What underlies the split between Development and Operations is a fundamental difference, like an actual physical law. Let’s explore that analogy together.

    IT Ops is fundamentally a constant fight against entropy, meaning disorder. Sysadmins expend significant energy trying to keep everything up and running, in a process which can sometimes feel like trying to shore up a sandcastle while the tide is coming in. Systems are constantly breaking down; users need changes made that introduce risk; and, as if that wasn’t enough, the whole shebang is constantly under attack from malicious outsiders. It’s no wonder that sysadmins have a reputation for being a bit grumpy!

    Homer: In this house, we obey the laws of thermodynamics!
    It’s not just a good idea, it’s the law.

    In contrast, Dev actually increases the level of entropy. Every new release, be it as small as a one-line fix or finding that errant semicolon, is a change that has to be made, reducing the order that Ops fights so hard to maintain. Dev is entropogenic, embracing change as a positive force.

    The interaction between Dev and Ops is analogous to the second law of thermodynamics: Entropy can only increase over time, unless energy is introduced to the system from outside. In other words, unmanaged change will eventually cause something to break, unless people put in continuous effort to prevent those breaks.

    Thermodynamics: Lots of Heat, not Much Light

    These conflicting goals, taken to extremes, can get unhealthy, with Ops actively fighting change or Dev going around Ops to increase their velocity of change. Left unchecked, this tension can turn into a vicious cycle, as frustration with the “Department of No” drives developers into unmanaged “shadow IT.”

    The good news is that the analogy with hard laws of physics is not perfect. The process of increasing entropy is not irreversible, and does not require constant inputs of energy to prevent. New agile approaches to IT Ops are emerging that are light and flexible, taking their structure from that tension. In the physical world, this is called “tensegrity,” and was originally formulated by Buckminster Fuller, one of those annoying polymaths who seem to make breakthroughs in every field they touch.

    Tensegrity structure
    Tensegrity structure

    IT Ops needs its own equivalent to tensegrity, a flexible, holistic structure that is able to embrace change while maintaining its shape. Structures that try to resist change through rigidity are brittle and ultimately fragile.

    Today, this flexible strength is enabled by AIOps, a term coined by Gartner to describe a new way of operating IT, based on augmenting human skills with automated analysis enabled by modern algorithmic and machine-learning approaches.

    Complexity Breeds Complexity

    Modern IT is too complex for any single person or even team to keep track of, instead requiring multiple specializations. In turn, each of these functions requires its own specific support tools that enable that particular role. Beyond a certain point, it becomes very difficult to combine these partial views to gain a holistic understanding of what is actually happening in the environment.

    Because of the scale of modern IT environments and the sheer rate of change that businesses require, old-style rules-based “Managers of Managers” can no longer keep up, requiring more effort to build and maintain their worth. Especially as infrastructure becomes more and more self-modifying in response to changing conditions and demand, IT Operations needs to accelerate just to keep up.

    AIOps brings developing issues to the attention of sysadmins before they become visible to customers and end users, and gives them the tools to collaborate effectively between different teams and specialist roles to avoid outages or restore service quickly and efficiently. In this world, automation is used to augment the capabilities of the human operators to make them more effective, letting people concentrate on adding value, not just fighting fires.

    This is how Ops can get out of the business of fighting Dev, and instead become a supportive partner. AIOps can catch issues that are affecting services and flag them to the right people early on, including the DevOps engineers that are responsible for production support.

    For developers, one of the fears around DevOps is that in “picking up the pager,” they are opening themselves up to a constant stream of irrelevant alerts. Old Ops hands nod wryly at this, or maybe try to share their personal approach to filtering the noise. Instead, AIOps ensures that only valid, actionable alerts get as far as being shown to a human being and taking up their valuable time.

    Eyes on the Prize

    It’s not all roses. As the AIOps market begins to mature, there is a certain amount of “algorithm-washing” going on, with more and more vendors trying to claim that either their legacy tool was AIOps all along, or that their one-trick pony is actually a whole three-ring circus. What gets lost in the excitement is what all of those algorithms are actually supposed to achieve.

    Me too!
    Me too!

    In considering a shift toward AIOps, the goal should not be simply to build a better mouse trap, meaning a smarter filter for monitoring events. The success of the project can be measured not so much in the mathematical results of the algorithms, but in the smoother exchange between the human users. This easier collaboration is especially important across team boundaries, whether these be different specialist units within the Ops function or developers and software engineers who need to work with Ops to understand the performance of their code in production.

    The benefit of transforming operations this way—moving from sequential, waterfall Ops to abile, collaborative Ops—is to turn DevOps-enabled teams into a sort of perpetual-motion machine, able to continue operating without requiring constant massive inputs of energy from outside the system. Modern applications and the complex infrastructures that support them cannot be managed with the sorts of rigid, cobbled-together, industrial approaches that were appropriate in the mainframe days. AIOps is the agile, flexible approach that can not only accomodate that complexity, but thrive on it.

    — Dominic Wellington

  • From Waterfall Incident Management to Collaborative IT Operations

    From Waterfall Incident Management to Collaborative IT Operations

    What can Ops learn from Dev in the transition from sequential, reactive IT support to real-time DevOps?

    One of the signature ideas of DevOps is right there in the name: the blurring of the boundary between the historically separate worlds of Development and Operations.

    On the face of it, this seems like a great idea. Developers have always grumbled about how Operations is preventing them from deploying as rapidly or as frequently as they would like. Meanwhile, Operations people have harbored a dark suspicion that developers might code with more attention if they were the ones getting paged at 3 a.m.

    Another of the great things about DevOps is that it is pretty easy to adopt in one area or project without needing to get the entire company onboard before you can get started. This means that you can quickly produce some interesting results in terms of accelerated development times and release frequencies, faster resolution of problems and more time spent innovating instead of firefighting.

    Armed with these results, DevOps evangelists spread through the organization, telling their story and recruiting other departments and teams to join them in the bright future.

    This is Where the Problems Start

    Complex enterprise IT environments are, well, complex. Different teams have adopted different tools and methodologies to do their jobs, and are justly resistant to throwing out something that works for them. Specialists in different areas require their own specialist tools to be able to do their jobs. Because of this, when an incident occurs, all of that agility and collaboration that DevOps brings goes out of the window in favor of something that looks much more like a classic “waterfall” approach:

    • When an event is received, it is routed to one particular team
    • The event is placed in a queue
    • Eventually, someone from the relevant team takes a look at the event and makes a determination of whether it falls in their area or not
    • If it’s in their area, they begin working to resolve the issue within their team
    • If it’s not (believed to be) in their area, they reassign it, either back to the front-line team, or to whichever team they believe to be responsible (often, this is the network team)
    • Repeat until either someone takes ownership and fixes the issue, or it becomes a Major Incident and everyone gets paged at 3 a.m.

    Just to complicate matters, there may be multiple instances of this process going on at any one time, as two or more different teams grapple with what eventually turn out to be different aspects of the same problem.

    Why Does This Happen?

    In health care, there is only so much that can be done by way of prevention. There is always the need to react to patient’s issues, starting from their symptoms. In the same way, the starting point for Operations is monitoring—gathering the symptoms of the IT infrastructure to be able to return it to health as quickly as possible.

    To understand problems with the Ops part of DevOps, we needed to start at the beginning, with those symptoms. We wanted to do this the scientific way, so we polled attendees of Velocity and Monitorama to understand their top monitoring challenges.

    The interesting fact was that across both populations, the main challenge listed by attendees was “alert noise/fatigue/volume.” Simply put, Operators were drowning in alerts to the point that they had completely lost the ability to understand what was actually important, let alone an overall picture of how those alerts related to the service that IT was supposed to deliver.

    This is another parallel between IT and health care, where alert fatigue has long been a concern, and with potentially far more serious consequences than in IT.

    A big part of the cause of the alert fatigue on the IT side is the proliferation of specialist monitoring tools. In the poll, 66.67 percent of people we surveyed have five to 10 monitoring tools in place, and yet 61.90 percent of them are still struggling with that problem of alert noise/fatigue/volume.

    To be clear, those different tools are each very important. Trying to monitor the network with an APM tool is probably not going to go well for you. They only become a problem when it becomes too difficult to build up an overall picture of what is going on, and sharing that between different people to solve problems faster.

    So What Can Be Done?

    Dedicated Operations organizations have long worked to assemble tools and techniques that can give them a complete view of the environment that they are supporting, and of the services that run in it. The accelerated pace of release that comes with DevOps is causing the old models to break down, but fortunately, it also carries in itself the seeds of a new approach to the problem.

    As development moved from the sequential waterfall model to a more rapid, collaborative, and agile approach, operations teams can learn from that transition to come up with a more agile approach to IT Ops.

    What would this look like? The first thing to do is to get rid of the blame-storming and buck-passing: “It’s the network!” “No, it’s the database!” I don’t know about you, but personally, I’ve been in enough of those meetings to last me for the rest of my life, thank you very much.

    Let’s All Go Fishing in the Data Lake

    Instead, a common perspective is needed, so that everyone can quickly agree on what is going on and what needs to be done.

    This does not mean throwing out all of those specialist monitoring tools! They are there for a reason. Rather, gather all of those symptoms in one place where the whole team can collaborate to make sense of them more easily. These days, it should be easy to get information in and out of the various components of your toolchain (and if it’s not, that’s a glaring warning sign that maybe it’s time to migrate off that particular tool).

    This will, of course, produce a huge pile of data, so it’s not sufficient to stop here with the creation of a “data lake.” Things disappear into lakes and are never seen again—in which case, why did you bother gathering them in the first place?

    What you need is a way to make sense of what is going on in the lake, and show that to the right human operators. In the same way that developers resent sitting around waiting for the Change Advisory Board to meet, operations people get frustrated and bored just staring at a screen waiting for something to happen. If instead you can use automatic techniques to assemble all the related symptoms of a single issue together in one place, that will give you a good starting point to resolve that underlying issue quickly.

    Brainstorming, not Blamestorming

    This is where the collaboration comes in. Again, people are doing this already, but they are doing it in a disconnected way. The tools have evolved, from email and IRC, to Slack and Google Hangouts, or even (heaven help us all) WhatsApp. What has not evolved is the usage model; it’s still ad-hoc and reactive, and requires lots of copying and pasting.

    Operations people can learn from what their developer colleagues have built or adopted. Kanban boards are a great technique for tracking open issues and who is working on them. ChatOps is a fantastic way to integrate human-to-human communication with human-to-machine communication. Integrated collaboration tools capture knowledge that otherwise is stuck in people’s heads or inboxes, unavailable to their colleagues.

    I’m an Ops guy at heart, but I can recognize a good idea when I see it. Our Dev colleagues have come up with quite a few that we could stand to adopt in Ops. Let’s learn from what they already know to work, instead of trying to reinvent the wheel.

    — Dominic Wellington