devopsy.kinsta.cloud

Author: Peter Waterhouse

The Whole Nine Yards with DevOps and Agile Operations
Growing up as a boy in Manchester, UK, football (ok, soccer to US readers) was everything to me. I vividly remember being herded into the stadium stands on cold rainy Saturday afternoons to watch my team (#MCFC) win, draw or lose. Back then, football was for the die-hards – no frills or fancy tech, just hot tea and a suspect meat pie at half-time.

How things have changed. Now sporting events have undergone a complete transformation – all digitally-driven.

High Tech Sports

Take Levi Stadium, home of the San Francisco 49ers for example; considered by many to be the most high-tech stadium in the world. There’s too much tech to describe in detail, but you can be sure that all the Wi-Fi beacons, 400 miles of cable, photo voltaic energy and a funky mobile app have been designed with one thing in mind – enhancing the fan experience. So no ‘cattle-herding’ at a 49ers game – nope, a personalized digital experience that’ll find you your seat, food concessions, the closest bathroom — and of course the shortest beer line (#1 feature in my opinion).

Levi Stadium isn’t just another example of technical wizardry. It’s more an exercise in an organization deeply connecting with its customers through the delivery of highly experiential services. Understanding the needs, motivations and desires of their loyal fan base and servicing them accordingly.

It’s a pity this same thinking isn’t always applied in IT Operations – that is, deeply understanding what Ops ‘fans’ need if they’re to ensure a good business user and customer experience across new highly dynamic agile environments.

High Stakes IT Operations

Similar to the Manchester game day experience years ago, IT Operations has often done it tough. They’ve worked in isolated NOC’s and service centers, been placed on 24/7 call; working weekends and missing their kid’s birthdays. They have been served specialized ‘team merchandise’ in the shape of complex tools and processes – always counting on their loyalty to save the day when the next mega release comes over the wall. No great surprise then that Operations in many organizations is seen as uncommunicative and uncompromising – almost, well – tribal.

But IT tribalism doesn’t win matches in the software-driven economy game. Now, digital business realities means IT Operations must supplement its traditional fixation on reliability and resilience with a willingness to drive change – Operations must itself become more Agile. But how do you start? Thinking back to the innovation at Levi Stadium, here are three good ways.
1. Gaining Loyal Supporters – Development and Operations aren’t enemies, but like rival UK football team supporters the teams are often segregated. Dev and their tools to the left, Ops with theirs to the right. From a business perspective this is plain dumb – especially when it comes to getting the most value from technologies like application performance management. Rather than frantically monitor applications as problems manifest in production and then call Dev for help, wouldn’t it be better to incorporate operational intelligence much earlier in the software lifecycle? So when code is released, monitoring could immediately provide development with an early warning on operational issues. Conversely, Ops practitioners gain accurate and timely insights into production performance requirements and KPI’s – that’s not only a DevOps style win-win, it’s an agile like best practice to increase quality and avoid technical debt.
1. Filling the Business Stadium – The goal of Operations has always been keeping the business lights on, but Agile Operations now means making them shine brighter. Cloud, Microservices architecture and containers all give businesses the technical clout needed to support speed at scale, but it comes at a cost. Today’s sophisticated applications rely on specialized technologies, so it’s perhaps understandable that Operations chase the technology down the proverbial rabbit hole with more specialized tools. But the real truth is that any monitoring system which doesn’t deliver true business value by providing unified insight into how customers are actually experiencing IT services is of no use to anyone. Remember too that technology scale is about future proofing your business, so look for scalable monitoring to match it.
1. Delivering a Complete Experience – The digital experience for a 49ers fan doesn’t start and end on game day. It is delivered by a linked series of services – a digital journey comprising seat selection, ticketing, parking, entry, action replays, and player stats reviews – every one of which can positively or negatively impact the fans experience. Whatever your industry, IT Operations will be called upon to support similar services, meaning deep and uninterrupted visibility across the myriad of front and back technologies underpinning every customer interaction, engagement and transaction will become essential.
Successful digital transformation needs more DevOps and agile operations, agile development, faster release cycles and higher quality software. It’s time for IT Operations to stop fixing production problems and become an active participant in supporting these goals. Without Agile Operations, your role in the software-driven business game ferociously being played out will be like that meat pie I ate many years ago – suspect.
July 15, 2015
The Heavy Cost and Burden of Shrink Wrapped DevOps

I’ve worked in the enterprise software game for quite a few decades now and for most part it’s been a good gig. Ok, there’ve been a few road bumps that beset anyone working in this oh so fickle and volatile business, but what really keeps me motoring isn’t the software at all – it’s the joy that comes from helping folks learn and grow – even in the midst of all the crap, chaos and confusion that surrounds us.

Paradoxically and sadly, most of this “triple-C” burden comes from over hyping and sensationalizing the real IT goodness. Goodness in the shape of best practices that can help us grow both at an individual level, and in context of supporting team, departmental and business goals.

DevOps is a classic example.

Here we have a set of principles and guidance to facilitate better communication across an often fractured IT function. All simple goodness you might well say. So why is it that clever folks are waxing on at great lengths to proclaim DevOps is the best IT thing since sliced bread? Why are vendors hurriedly washing just about every product (from servers to thermostats) with the DevOps soap? Why is it that our industry is taking something that’s really common sense and shrink-wrapping it in hyperbole and nonsense?

Well, the short and simple answer is there’s a buck to be made. Hey, nothing wrong with entrepreneurial spirit, except of course when the cost and burden of suspect knowledge and overstated capabilities (all packaged in glossy DevOps wrapping), is passed on to you – and becomes – well, another “box” that sits and gathers dusts on the shelf of good intentions.

So whether you’re a working at the sharp end in development and operations or in the management treadmill how do find and put to work all the DevOps business goodness and not end with another trendy IT white elephant? I have some surprising, but simple advice that starts and ends where it’s most effective – with you.

Garbage in, Garbage out – Go on, Google DevOps — 10,900,000 results in 0.23 seconds– isn’t the Internet just wonderful. But in all the noise, how much is actually useful and applicable in context of your daily grind – Not that much, I’d say. Nuggets are especially hard to find when you’re distracted by all rants and raves – like, ITIL is dead – long live DevOps, or DevOps solves Global Warming. Ok, I jest, but for the sake of your sanity, seek out folks who apply common sense thinking as it applies in your world. Personally, I enjoy (although I don’t always agree with) the work from the likes of Kim, Spafford, Haight, Bloomberg, Edwards Willis and Mann (even Rob England for a good dose of Skepticism), but depending on your own craft, look for solid, meaty, practitioner-baked material that makes sense to you.

Greeks bearing Gifts – or anyone for that matter. The good or bad thing (depending on which way you look at it) about DevOps is that being a collection of principles it lacks a formal framework and isn’t a standard. Unfortunately this provides an opportunity for scurrilous types to fill the void with suspect claims. So if a vendor tells you that they have the best DevOps product, or that their product/service is DevOps compliant, they should probably be selling steak-knives on one of those late night promo TV channels.

Don’t Play the Fool – In slang we often use the word “tool” to depict someone that goes along with something without question and never questions the greater purpose. How often do we see the other form of tools (great software products) used in this manner. All too often the business invests in damn fine open source and smart enterprise software, only to have its true potential diminished because of tribalism, parochialism and plain stupidity. So the next time a tool lands in your eager hands, think beyond your silo and see what automation and integration exists to weave the DevOps collaboration fabric between you and you colleagues. Like for example Ops guys using performance management data to auto-generate realistic virtual services that Dev and Test can use to prevent defects reaching production.

Apply the KISS Principle – it’s easy to get carried away in DevOps folklore with grand tales of continuous deployment, microservice magic, containers and design for failure. But back on your planet, old creaking servers still need to be patched, the help desk sucks, and you’re being asked to deliver mobile apps on the smell of an oily rag. Sure, there are great lessons in the DevOps hall of fame, but think practically about what’s applicable in your organization. Like any best practice, DevOps isn’t an all or nothing approach, so cherry pick what works and think big as you progress. Personally, I think the best place to start is by applying lean thinking – that is, clinically identifying and unashamedly owning up to the waste and technical debt that exists in your own department.

DevOps is great, but it isn’t a perfect package. Your role is to take it, shape it, and apply it in ways that work for you and your organization – which always involves, time, cost and commitment.

Shame on anyone who tries to shrink-wrap DevOps in outrageous claims that just add more burden and cost to an organization.

May 11, 2015
Can your DevOps and APM Initiatives Increase Customer Loyalty?

I’ve reached a point where I can rate most services based on the quality of my digital interactions. So much so in fact, that any human interjection into my daily digital consciousness has become, well, almost quirky.

It was with some surprise that after a prolonged period of digital engagement with my bank, I received a call from a customer service representative. After the initial shock, I was brought back to earth by a pretty standard question:

Based on the service you’ve received recently, how likely is it that you would recommend our company to a friend or colleague? Please rate on a scale of 0 to 10.

Now with a background in marketing, I understand what’s in play here. It’s an attempt to build what’s called a Net Promoter Score (NPS); a simple management tool to gauge the loyalty of consumers. If I respond with a 9 or 10 I’m considered enthusiastic – a promoter. On the other hand if I think the service sucks and score 0 to 6, then I’m a detractor. The metric is calculated by subtracting the percentage of detractors from the percentage of promoters. Yep, management theory doesn’t have to be complex.

NPS as an indicator of business performance is important because most leaders understand they’re competing primarily on the basis of the customer experience. And, since this experience is predominantly based on a journey of linked digital engagement, maintaining the quality of these interactions is vital.

So what has all this to do with Application Performance Management (APM) and DevOps? Well, lots – let me explain.

If we think about digital engagement, there’ll be many points across a customer journey (from app on-boarding to service renewal) that’ll help determine the overall experience and whether or not customers become promoters. I call these “magic moments”; points of interaction where businesses have an opportunity to deliver an experience that transcends beyond the purely transactional into a more powerful emotional connection. It’s vital therefore that these moments become the focal points for our APM strategies – only this requires some new thinking. Here’s my take:

Monitor the Magic – IT teams must identify all the key magic moments (especially those that have the highest correlation to customer loyalty) and implement automated performance collection and measurement. Some of these may form part of a linked value chain (e.g. an airline flight experience = seat selection + mobile boarding pass + baggage collection + loyalty rewards), or be services designed to drive a recurring revenue model (e.g. free shipping / discount options), or even be delivered by partners through API’s. Whatever the model, APM must be capable of providing a unified end-user view into every contextual, situational or transactional interaction – mobile app and web-based – because it’s here that loyalty is determined and promoters are formed.

Make Metrics Matter – when a prospect navigates to your application (perhaps via some promotional web-advertising) that’s great, but it’s not yet magic – that is until the visit converts into a sale. It’s at these points that we should be connecting our “listening” strategy (via APM) to the customer and determining effectiveness in a business context. This can be tricky because traditional (but still important) metrics like MTTR are introspective, whereas what’s really required are external customer-centric indicators. To address this, DevOps practitioners should consider applying newer APM analytical methods and instrumentation in order to learn the relationships between service elements (e.g. web site sign-on / authentication time, response times) and customer experience – using the intelligence gained to drive improvements where they’re most effective – yep, you guessed it – at those “magic moments”.

Share the “Profits” – an advanced experience-based APM strategy must also include working with groups who can profit from rich information insights. From a DevOps perspective, this will involve development leveraging the data as the operational feedback needed to steer both functional and non-functional improvements – for example, mobile app usage and behavior analytics are critical early indicators of functional usability. But given the businesses reliance on digital services, many groups beyond IT will also profit. For example, if it’s reliably determined (and presented) that application performance is a key driver of Net Promoter (or any other business metric), which in turn is linked to improved financial outcomes, then this could serve as the basis upon which to prioritize future investment.

Instrument Culture – aligning with metrics like NPS is valuable because it helps shape collaborative, customer-centric behaviors. However, this requires tools that can embed these behaviors into everyday operational practices. Improving customer experience is a shared goal, so APM processes must facilitate the feedback of intelligence between Dev and Ops, while also meeting the needs of every function (Developers, Application admins, Level 1 and 2 support etc.) within each group – especially in context of the role each plays in improving customer experience.

As companies increasingly engage customers via digital channels it’s a given that IT will be measured on how it increases customer loyalty. While business metrics sounding like NPS may at first seem foreign, they present us with a unique opportunity to demonstrate what DevOps and APM is really all about – winning more customers and getting them to stick with you.

March 17, 2015
Digital Transformation with Agile and DevOps – Something “Wicked” This Way Comes
Our world is full of wicked problems, like climate change, urban planning and social injustice. They’re called wicked because they’re hard to recognize, define and solve. Furthermore, complex interdependencies mean solving one aspect will just expose or create other problems.

Today’s digital transformation imperatives are creating wicked problems for IT. So wicked in fact, they make managing technology in the traditional sense look like child’s play.

Suddenly, businesses are faced with complex dilemmas. How can they compete in a world where their customers place less emphasis on physical products and more on the total experience? How can we adjust marketing from driving consumer behavior to responding to it? How can they adapt business models in context of the customers’ world? – a world that’s increasingly dependent and driven by digital technology.

The short answer is of course software, but the problem is our approach to delivering it just isn’t suitable in a wicked business world – why? Well its simple – we keep working on the assumption that we understand the problems we’re trying to solve, when in fact we don’t understand the problems of our customers at all – that is, until we have a solution.

Head scratching stuff huh, but consider this –

Did you know you had an entertainment problem before Apple gave you iTunes? Was booking hotel accommodation such a pain before AirBNB introduced the whacky notion of people renting out their own rooms?

So faced with such wickedness what can we do? Well, we can start by accepting our current IT failings, develop some wickedness of our own, and mitigate against what can derail your efforts.

Small is Beautiful … really it is!

In the past we’ve tried to solve business problems with what I call IT “bigness” – big iron, big software applications, big teams, big projects, and big expectations – oh, and let’s not forget big funding. The results, however, haven’t set the world on fire have they? – Big IT white elephants, big budget blowouts and big executive fallouts (literally and through the revolving door).

But surely we can’t lay the blame squarely on IT. After all, traditional models of software design and project management where we diligently survey the client’s needs are baked in common sense. Add to this the rigor and procedure around sign-offs, budgeting, design, development, testing and deployment – doesn’t this guarantees success?

Of course it doesn’t. Apart from the big fat fails, we also end up with what I call a big “sucking culture”– your software sucks, IT operations sucks, this job sucks, our business sucks – we’ve all been there, haven’t we?

All of this isn’t due to bad people and technology. It’s because we work on the premise that the business understands what it wants – it doesn’t. Much better then to put aside “bigness” and start listening (maybe insisting) that the business first lays all the wicked cards on the table, like – what are our markets and how is this changing? What are our target demographics? What are our barriers to entry?

After all this ‘warts and all’ revelation, it’s time to get busy. Now, with a shared understanding, we can start quickly prototyping elements that perhaps only solve one small problem. Then we continuously develop, test and demo, knowing and accepting other problems will surface that need to be solved. And, as we know from agile development, it’s all a case of ‘rinse-and-repeat’ – hopefully achieving a state of nirvana – giving your customers software answers to problems they don’t know they have. Problems that lead to increased market share, revenue and profitability.

So is a liberal scoop of Agile or its younger DevOps cousin the answer? Not necessarily — that is until some essential toppings and flavors are added to my somewhat vanilla perspective:
- Executive Flag Waving – try selling a methodology or philosophy based on the premise you’re trying to solve problems you don’t yet understand. Good luck with that. If executives simply want faster, better, cheaper, it’ll be much easier (and safer) to hold back grandiose talk about experimentation, “failing fast” and continuous improvement. Understand, however, that without tangible commitment, half-assed agile or DevOps is like a fast melting ice-cream cone – messy and hard to handle.
- Behavioral Hacking – even with executive champions, persistent and rigid organizational behaviors will need to change. Just hiring lots of agile coaches or mandating DevOps won’t work until there’s willingness for people to be engaged around new ideas and fresh thinking. With that in mind, be prepared to do some heavy lifting – socializing and selling new concepts, but also looking for opportunities to show how the smallest of changes (e.g. team refactoring, incentives) lead to improvement.
- Show me the Money! – So how to you fund agile and DevOps? It’s going to be tricky to cost something exactly when you don’t know the exact outcome of a new project, but businesses may initially insist on this to minimize risk. My advice here is to tread carefully. Stakeholders should be more interested in ROI than coming in under budget, so fixed-cost might not be actually what want anyway. Your job therefore is to convince them by continuously demonstrating ROI – not just from a single project, but also by the accrued savings that agile and DevOps deliver the whole organization when they systematically reduce technical debt and waste.
Developing the wicked traits I’ve described doesn’t mean IT has suddenly become an evil entity. Nope, it’s just a case of tapping into fantastic business opportunities in a constantly changing digital world. Agile and DevOps can help you solve and even create some wicked problems for your customers, but that involves confronting and challenging entrenched thinking as it relates to just about everything.
March 3, 2015
Is Your DevOps Team Ready For The Real World?
Ten years ago I had the biggest cultural shock of my career. It wasn’t witnessing poor collaboration across IT development and operations or whatever tech fiefdom you care to mention. Nope, these were minor issues compared to those facing me on a really interesting project – integrating Industrial control systems with IT management.

Working with a team of product managers, coders and sysadmins, our task was to develop a standards-based application for monitoring a physical research facility. When I say monitoring, I’m not only talking about network and systems management, but also managing equipment operating in the ‘real world’ – like manufacturing robots, pressurized compartments, control networks, smart meters, alarms and controllers.

It was a new experience for the team. Not only were we faced with a new set of technical challenges, we had to quickly become familiar with foreign sounding terms (to us anyway), like programmable logic controllers (PLC’s), supervisory control and data acquisition (SCADA) and human machine interfaces (HMI’s). Add to this a very different set of stakeholders – like industrial and plant engineers, and the initiative started off resembling something closer to a multi-language speed dating session (only without translators) than it did a traditional IT development project.

At the time projects like this were a rarity because the worlds of operational technology and traditional IT was seen as separate. Process control and electrical transmission systems ran on separate control networks with proprietary protocols, whereas IT ran on a business network and primarily handed back-office business systems – the two domains were as different as ‘chalk and cheese’.

But all that’s changing courtesy of the Internet of Things and a software-driven economy.

Today, organizations are realizing that standards, unified architectures and software applications can be used to connect and integrate the devices and sensors in the physical world with IT systems – the benefit – more seamless end-to-end business processing.

Take for example manufacturing. If plant production equipment can be better integrated with consumer demand analytics, then optimum maintenance windows can be identified. Similarly in energy and utilities, integrating the physical world of smart meters with back-end IT customer information applications is a must for efficient billing.

This convergence will drive new approaches to developing and operating applications. The DevOps movement with its focus on teamwork and collaboration to shorten delivery cycles while improving quality might just be the answer, but there are some serious challenges to overcome – not least cultural.

You might think you’ve solved the great IT divide with DevOps, but that’s nothing compared to the cultural chasm you’ll need to bridge across the world of industrial operations and IT. And no amount of IT ego will help, which I learnt the hard way when I was once given a sharp lesson in what ‘real-time’ processing actually is business by a group of chemical engineers – boy was I humbled.

Technology beat up aside I believe there are many ways DevOps can help address the challenges associated with connecting and integrating operational technologies – especially:
- Standards guidance – since operational systems work in isolation and use proprietary systems and communication protocols designed for specific control system networks (e.g. Modbus and BACnet), DevOps teams can help provide advice on new standards and integration platforms.
- Knowledge and Expertise – DevOps teams can provide expertise in the selection of new software platforms that are now available to build and run Internet of Things applications.
- Security and Risk Management – as more and more software finds its way into sensors and operational control networks become connected to the Internet, DevOps and security specialists will have a key role in identifying and mitigating many new security risks.
- Data Management and Analytics– connected sensors and control systems may result in the need to capture and analyse huge quantities of data. DevOps teams can help in areas such as application architecture and design, testing, network capacity and cloud automation.
As operational and information technologies converge, I believe the purview of DevOps will increase considerably. Now teams might include technology constituents from other areas of the business – like engineering and logistics specialists – each bringing new skills to the table that when coordinated with IT expertise in areas like mobility and cloud computing has the potential to drive huge efficiencies.

Application management will change also. Development work will increase, but so too will need to bake quality and resilience into every release. This is especially important because as processes become more integrated (from the factory floor sensor network to a mobile enabled materials management system or from healthcare diagnostic equipment to patient records applications) any minor defect, security issue or performance problem can trigger more catastrophic business failures.

When I look back to my own teams foray into managing ‘real world’ technology, I realize that while we never had DevOps to guide us we still used many principles. Success came when we built a cross-domain business team that maximized a unique set of technical strengths, but also understood each other’s weaknesses.

So DevOps teams take note – as technology convergence accelerates, the goal for many industries won’t be just bridging the IT development and operations divide and releasing more and more software, it’ll also be about developing a new range of end-to-end digitized business processes – from the factory floor to the boardroom – that’s a bigger deal and where success will ultimately be measured.
July 14, 2014