devopsy.kinsta.cloud

Author: Peter Waterhouse

  • You’re Not Doing DevOps if You Can’t Pull the Cord

    You’re Not Doing DevOps if You Can’t Pull the Cord

    There are many concepts in Lean manufacturing that have applicability with software development “factories.” One example is the Andon Cord.

    The Andon Cord concept involves using signals and alerts to indicate that a station on the assembly line has a problem. Alarms are activated manually by workers pulling a physical cord or automatically by the equipment itself.

    Developed as one element of the Jidoka quality method and pioneered by Toyota and the Toyota Production System, the Andon Cord empowers workers to notify co-workers and management of quality and process problems across the assembly line—and even stop production.

    While pulling the cord to stop production sounds crazy, if a defect is found and can’t immediately be fixed, then pulling the cord means stopping the problem from flowing downstream, where it will be costlier to fix and have a severe impact on the customer experience.

    But the point of Andon Cord pulling isn’t only to fix problems. As part of a continuous improvement program, much of the “pulling power” comes from how it helps teams build quality into products and processes. When production stops, improvement begins.

    Many would argue that the application of Lean management concepts such as the Andon Cord is more challenging in software engineering. With software, there are many more opportunities for defects, and testing for problems only occurs late in the development cycle. So late, in fact, that in some cases customers end up doing most of the testing.

    But as the digital and physical world becomes inseparable, excuses and outdated practices are no longer acceptable. Just as we wouldn’t ask a customer to test air bag deployment in a car, we shouldn’t expect them to test whether a new software-enabled telemetry system is subject to security hijacks.

    So having the power stop software development production line to prevent defects, security problems and safety issues during the actual development process makes perfect sense. But how do you start?

    Automate Andon with Continuous Testing

    In many organizations people are often afraid to speak up about problems. Even if bugs are suspected, teams will cover up problems by coding around them, even fudging tests. Owning up to problems doesn’t come naturally, so defects and crippling technical debt accumulates. DevOps practitioners need their own version of the Andon Cord: fully automated testing.

    With DevOps promising software throughput at dizzying rates, quality must accompany the velocity. Fail here and the continuous delivery glass can only be half-empty. This is why the process must incorporate its own automated testing Andon Cord to prevent problems moving downstream. Automation must be comprehensive, providing maximum coverage at every stage in the software pipeline—from requirements definition to full production and including code quality, performance, security, compliance and test data. Then, when a test fails, the system must be capable of stopping everything until the problem isn’t just fixed but can’t happen again.

    Eliminate Waste at the Source

    If you’re pulling the cord but not making improvements you don’t have an Andon Cord. Consider a factory scenario where machine operators always have to sweep up dust that’s clogging production equipment—obviously, they should be eliminating the source of the dust.

    Unfortunately, many software engineering practices involve lots of dust-sweeping.

    When developers need access to dependent systems for testing, they might mock up test code. If QA needs test data in a hurry, it could copy production data but fail to mask sensitive customer details. In IT operations, staff console watch in the hope of gleaning signals from all the noise—the list goes on. Lots of sweeping without ever eliminating the source of the problem.

    Fortunately, tools with Andon-like capabilities can help. Service virtualization techniques can emulate dependent systems to remove testing constraints, while automatically generating synthetic test data helps eliminate the possibility of costly compliance breaches. Modern performance monitoring systems also play an essential role. By providing performance insights before, during and after every software build, they help cross-functional teams swarm problems when the cord has been pulled by an automated test.

    Build the Ultimate Cord – Customer Experience

    If an organization considers itself in any way customer-centric, it will be able to “pull the cord” when it detects anything that detracts from the customer experience as delivered across multiple channels—web, mobile, call-center. The winners go further, using these events as fuel for continuous improvement programs.

    This all requires some serious smarts, not at the least application analytics to monitor the customer experience in real time. Such advanced systems become the ultimate Andon Cord, since they treat every sub-optimal customer experience (e.g. functional issue, usage behavior, performance problem) as both a defect and opportunity for improvement. Through advanced analytics and integration this improvement could take many forms—such as immediately prescribing code improvements to developers, or support systems notifying customers when a poor experience is about to occur—maybe even (shock-horror) compensating them financially for any inconvenience.

    A strong DevOps culture thrives when individuals call out mistakes and inefficiencies. Fail in this respect and organizations risk paying a heavy price, such as broken customer trust and lost business. Seek out automated methods that help you metaphorically pull the cord and keep on pulling it. Only then can teams truly excel in driving real improvements that will help business gain and retain customers.

    — Peter Waterhouse

  • 5 DevOps Techniques to Improve Software Hygiene

    5 DevOps Techniques to Improve Software Hygiene

    Thinking about frequent food contamination outbreaks, perhaps the old adage, “You are what you eat,” should be modified to, “You are what you eat—as assembled across an unknown supply chain.” Even foods advertised as fresh and healthy can be quite the opposite. And it could be because of just a single unhygienic practice hidden within the food production life cycle.

    It’s not unlike the issues facing modern software development. Now with the advent of open-source, cloud computing and component-based designs, the succulent software morsels we deliver can be fouled by a range of contaminates. And while not as a physically debilitating as a case of gastric flu, the impact on business can be extremely severe, such as major security incidents and compliance breaches—the E. coli of digital software production.

    The good news is that IT organizations can take effective steps to mitigate the risks. Much like food processing, software contaminates can be introduced anywhere code and data is handled and stored—from requirements and coding to testing, release and operations. This means all stakeholders must take responsibility and be held accountable for software quality. To that end, here are five good practices DevOps practitioners can use to improve software hygiene:

    Thoroughly wash test data before handling – Creating, maintaining and provisioning accurate test data is one of the most time-consuming and resource-intensive challenges in continuous delivery. It’s also fraught with business health hazards if teams fail to protect personally identifiable information or continually rely on data that’s well past its use-by date. With public cloud services providing convenient platforms for testing, ignoring these issues will only increase the risk of major compliance breaches. It’s important, therefore, to supplement data migration with fully automated methods to mask and subset sensitive data. Additionally, by continually enriching virtualized or emulated services with realistic test data, teams have a repeatable method for improving the quality and efficiency of testing.

    Pay special attention to open source freshness – Open-source software has become a large contributor to the portfolios of many organizations. But just because something is “free” doesn’t negate an organization’s responsibility for due diligence, including understanding total cost of ownership and mitigating security risks. Even though open-source projects have active communities, that hasn’t stopped code pathogens lying dormant for years (for example, Heartbleed went unnoticed for 2.5 years).  With open source, the onus is on development to retain code “freshness”—something that might be difficult if teams are constantly being pulled onto different projects, or if thorough risk control management isn’t a core competency.

    Don’t leave software lying around for too long – Just as many foods spoil quickly, software release bottlenecks can quickly leave a sour taste for business. Like a truffle imbues a marvelous aroma, modern digital business must be flavored with a continuous stream of application updates. And, since windows of business opportunity close quickly and application lifespans are as short as days, it’s critical any deployment delays are removed. Letting software linger in the release queue also introduces other hygiene issues—for example, teams putting off documentation or code-refactoring because they know they can always get back to it later. Inevitably they never do, and when the code finally gets deployed it’s unclean or laden with defects and bugs.

    Set up a “health inspection” SWAT team – In the traditional “software factory,” application performance management, security auditing and compliance checking often comes late in the software development cycle—leading to delays and conflict. DevOps, with its focus on team collaboration and continuous feedback, aims to rectify this by embedding these practices much earlier. But when resources are limited, it’s not practical to assign a security or operations professional to every agile team; it just won’t scale. What’s key, therefore, is to take a more pragmatic approach; developing teams of specialists and repeatable automated practices others can leverage to remove “bad smells” from code. This could involve:

    • Regular education on web application cross-contamination issues and modern API security techniques
    • Moving application performance management into pre-production to improve quality
    • Auditing services to regularly check composite applications assembled from third parties and open-source software
    • Automatically invoking application security testing before and after release and deployment
    • Methods to automate testing as requirements are modeled and changed—immediately linked to correct data and expected results
    • Scoring mechanisms to highlight security and risk problem areas—all bubbling up to a team and divisional level

    Work with your suppliers – In the rush to embrace modern cloud architectures for application scale and resilience, teams often forget that even the most robust platforms can succumb to problems. Savvy teams understand this, working with their cloud partners using modern monitoring and analytical methods to quickly detect and respond to many complex issues well beyond the scope of traditional rules-based methods and slioed tools. By using a design-for-failure approach, DevOps teams understand that no platform is infallible, preferring instead techniques that quickly anticipate and contain inevitable failures to keep the digital business running.

    Even the tastiest of food can be toxic, and it’s the same with software. However functionally rich our software creations appear to be, contaminates lurk everywhere within the software pipeline. Use DevOps techniques to pinpoint and eradicate “digital spoilage” quickly, and avoid making the business and customers ill.

  • Taking Stock of DevOps: Where to from Here?

    Taking Stock of DevOps: Where to from Here?

    Have you noticed how every technology trend goes through a series of stages on the path to mainstream adoption? First comes doubt and denial, then tacit acceptance and finally acknowledgement that it works fine after all. The DevOps movement is no different.

    First came questions about what DevOps actually is and its usefulness in an enterprise context (whatever an enterprise actually means these days), then a trickle of success stories and finally the publication of tangible and quantifiable benefits.

    Thankfully, we’ve moved on from the “Is it a bird or is it a plane?” stage. Of course, that doesn’t stop hype, but at least the publication of benefits—such as those recently illustrated in the global study, “Assembling the DevOps jigsaw“—allow us to cut through all the “DevOps solves global warming”-type of nonsense being written.

    So with the value jury finally in, the next question we should be asking about DevOps is, Where to from here? Interestingly, the old Wikipedia entry for DevOps provides some guidance, stating that DevOps is:

    … a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes.

    That’s pretty good, because it emphasizes the importance of collaboration across all IT functions (not just Dev and Ops). However, this still doesn’t quite cut it for me. Although lots of cultural and automation heavy lifting has been done to bridge Dev and Ops, more work is needed to understand how aligning other IT functions to DevOps actually benefits the function itself and the business. Also—and perhaps just as important—how aligning other business functions, such as HR and finance, can lead to more success.

    DevOps Works Because It’s Good for Business

    Business leaders talk in terms such as faster lead times, customer conversions and time-to-value, which (putting cultural discussion aside) is really the essence of DevOps. And because DevOps adoption directly correlates to making a buck, it has a fair chance of garnering attention and investment from the business.

    That said then, information security and other professionals would do well to hitch their wagon to DevOps. One of the beauties of the movement is that teams collaborate and provide input much earlier in the development life cycle, with processes then automated to ensure faster release time and quality. By also involving security earlier, businesses could feasibly harden security, balancing all those “rain on your parade” audits and checklists with automated methods that ensure compliant configurations lead to faster deployments.

    An interesting but important benefit is how this approach helps align IT security to business outcomes. Rather than proving its worth through a whole raft of technical sounding measures, such as penetration test scores, patched vulnerabilities or security incidents detected, security can report functional value in context of business goals, such as how automated compliance equals faster releases, leading to increased revenue.

    Of course, DevOps doesn’t just involve Dev, Ops and security, it also will involve other functions providing valuable input to help increase speed and quality. Think of it like the old “Stone Soup” story but without the trickery—everyone, be they software designers, product managers and enterprise architects, providing advice and expertise earlier in the software life cycle so essential elements of resilience, supportability and scale are baked in right from the get-go.

    DevOps Shouldn’t Start and End in IT

    The DevOps way suggests that we shouldn’t just restrict ourselves to IT functions—start there, yes, but other facets of the business should also participate. This sounds like common sense, but in reality it’s more critical because the boundaries between business and digital have blurred significantly. Fail to recognize this and we could end up a silo of good intentions: culturally correct but lacking any teeth.

    This explains why even the much-maligned HR function has a DevOps role to play. For too long, many HR departments have been fundamentally back-office, dipping their toes in the muddy waters of organizational culture with periodic staff surveys and performance appraisals. But this doesn’t go far enough. Now, HR leaders must step out onto software factory floor and work hand-in-step with other DevOps practitioners to identify all the organizational and psychological barriers that inhibit business growth and the development of talent.

    Finance, too, must become aligned to DevOps. This involves moving beyond traditional IT cost-center thinking to developing new fluid funding models needed to stimulate software innovation by agile teams. This means business change agents must join forces with DevOps practitioners to look at better ways to capitalize or expense software development costs or optimally monetize new apps, APIs and information assets. Once again, the earlier this happens in the software life cycle, the better.

    From an IT perspective, we’ve come a long way with DevOps in a relatively short amount of time. But this is only the start of the journey. Most businesses haven’t even considered transformation of the whole organization using DevOps principles as a change agent. Those that do most likely will gain first-mover advantage—and that’s a good place to be in an economy increasingly shaped by application software.

  • Using DevOps to Avoid Architecting Application Slums

    Using DevOps to Avoid Architecting Application Slums

    The U.K. high-rise tower blocks and built circa 1950-1980 were heralded as architectural wonders—large concrete towers easily housing the same population as the smaller houses they replaced. With large rooms, excellent views and surrounding open spaces, they initially were popular residences—low cost and futuristic.

    But looking futuristically sometimes has a habit of coming back to haunt you.

    Due cost-cutting, substandard materials, demanding deadlines and rushed construction practices, these modern-age housing miracles became the new slums of the day. By replicating tower-blocks across the country, town planners unwittingly replicated bad designs everywhere. The results: undesirable housing and elevated crime levels, leading to slums and urban decay. Even the surrounding open areas were neglected because no one had ownership over maintenance and supervision.

    In IT, we have our own design and architectural calamities.

    There are many similarities between existing IT apps and decrepit old tower blocks. IT monoliths can be hard to scale, requiring costly maintenance to avoid disrepair. They’re also tough to police, with vandalism and theft being a constant problem. Add to this integration issues (IT’s version of unsupervised open spaces and pathways), and we’re left with systems inadequate to meet the needs of modern digital business.

    Enterprise architecture must change with the times

    DevOps, with its focus on operations and development collaboration across the entire service life cycle, is now seen as the answer to some of these problems—an approach underpinned by agile and lean principles for rapidly delivering high quality digital services. That sounds great, but let’s not forget enterprise architecture (EA), because without it we could still be building IT slums—only more of them and much, much faster.

    Of course, many DevOps stalwarts may argue (with some justification) that heavy EA practices and frameworks haven’t evolved to match the pace of agile. Iterative-style development means where you start isn’t necessarily where you finish, necessitating a transition from set-in-stone architectural designs to a fluid mix on continuously evolving decisions. And, if EA can’t adapt, autonomous and self-managed agile teams increasingly are empowered to make architectural calls.

    But great developers aren’t necessarily brilliant architects; they’re not hard-wired that way. It means they could be charging down a development path without architectural guiderails, or tactically making team-based decisions that compromise program-level objectives. For example, the scalability and performance needed to support a business strategy of tripling the customer base and increasing satisfaction scores could get lost because development focuses myopically on delivering more functionality.

    However, we can’t just lay the blame on developers. Enterprise architects must recognize that any misalignment between their processes (especially the rigid and static ones) and the vision of an increasingly software-driven business will only further isolate the practice.

    Architecting new fluid guidelines and principles

    The role of EA in an agile context is still up for debate, but this shouldn’t hold us back. By using architects and fluid guidelines means we can maintain the pace of delivery while preventing chaos. The trick of course is to limit architectural over-engineering while providing developers with a minimum set of EA to avoid technical and architectural debt.

    Even in minimal mode EA can still help developers identify quickly what should be considered during design. The aim is to guide developers into understanding what architectural facets make up a successful application, and more importantly, how new ways of thinking can support longer and more sustainable software innovation.

    Interestingly, these practices can be presented in a “slum avoidance” context:

    • Avoid substandard or unsupported materials – The tool chains being used will probably comprise both commercial and open source software, but beware of anything unsupported. Just as the cost of maintaining tower blocks reached unsustainable levels, so too, will support burden on an organization if software isn’t commercially backed, especially in IT operations. Also, work with development to understand where constraints lead to testing compromises—the “construction shortcuts” syndrome, resulting in more defects and technical debt.
    • Never use new technologies to rehouse old problems – As tower blocks degraded and vandalism increased, many local authorities tried to contain the situation by housing “problem groups” in the same block. But that just increased problems. Enterprise architects can make the same mistakes—for example, laying down crazy edicts that everything must be containerized, even some legacy apps; not understanding that runtime independence only extends the life of problem applications;  or providing no incentive to drive improvements and clear existing legacy IT slums.
    • Monitor and manage your sub-contractors – Whatever cloud model you adopt, abstracting away the infrastructure or application stack frees development to focus on coding. That’s great, but don’t forget “PaaS-ing” apps to the cloud doesn’t mean relinquishing ownership of application performance and the end user experience. To this end, architects must act as cloud-brokerage advisors—for example, working to ensure any cloud service provides open APIs, so that monitoring can be seamlessly incorporated into a unified solution.

    Of course, architecting great new digital services with DevOps is only part of the answer. The innovations of today may become legacies of the future and many new apps must also integrate with existing systems. It’s essential, therefore, that architecture covers both bases: working in minimal mode to ensure fast construction of quality services, but also applying standards and governance when needs dictate and systems change.

    No matter how good your app designs look on paper, bad architecture just means building more problematic systems and never clearing the existing slums.

  • Why NoOps is a DevOps disaster waiting to happen

    Why NoOps is a DevOps disaster waiting to happen

    Ever since the first hunks of Big Iron hit the streets, we’ve been struggling (and failing) to keep up with the demand for software applications.

    Now it’s getting harder. Not only do we have to satisfy an insatiable appetite for new apps and services, but the nature of the business palate is changing. Business now demands a continuous flow of software innovation over back-office application support – or using the haute cuisine analogy – they want IT to serve a classy degustation menu versus just warming up a ready-made TV dinner.

    All this bodes well for the fast iterative style agile, with fully autonomous development teams empowered to increase the rate of software deployments.

    Sound great, but what does this mean for IT Ops?

    Many would argue that IT Ops is a completely defunct or redundant function. Even credible analysts and thought leaders have touted NoOps (No IT Operations) as a viable option. Some even suggesting that it accelerates DevOps collaboration since by removing the Ops function you remove all friction.

    There appears some validity to these arguments. After all, if developers can codify operational functions into their activities, why do we need the disciple?  Perhaps this logic explains why for some, IT Operations as a function is becoming disenfranchised; no longer having a voice into critical digital transformation initiatives – or if it does, just the sound of “No” – the last deployment bottleneck.

    The death of IT Ops has been greatly exaggerated

    All this NoOps thinking is flawed for a couple of reasons.

    Firstly, however operationally awesome developers think they are, building resilience, maintainability and supportability is not always top-of-mind. Worse still, these elements might be neglected if management is fixated on rewarding developers according to ‘speeds and feeds’. Secondly, even if these elements are addressed, they’re often conducted at the end of development cycles or bolted on after problems are discovered – that’s like baking a cake, forgetting the sugar, and then trying to compensate with a sickly sweet chocolate sauce.

    In reality, great operations engineers are best equipped to help incorporate operational excellence into all practices. After all they have years of experience supporting every new wave of technology – from Mainframes to Microservices. What must change, however, is how their expertise is developed and shared.

    • Become cloud connoisseurs – Like our development colleagues, operations’ emphasis will shift from on-premise to cloud. This means ensuring Ops expertise is fully leveraged to support the business nirvana of delivering high-quality digital customer engagement at scale. So if you’re an IT operations professional still configuring QoS policies on routers or manually provisioning development environments, it’s probably time to skill-up in PaaS, Amazon EC2 and containers. Either that or go and find a gig with a cloud service provider.
    • Embed Ops In Dev – This means being less separatist and silo’d and more inclusive and unified. Rather than focus on technical diagnostics and reacting to failures, the new operations professional will apply systems thinking to holistically look at how business applications and the all-important customer experience can be improved. This is easier said than done with skeptical development teams, so it’s essential that monitoring feedback is automatically shared and incorporated into all practices – like for example, agile sprints
    • Embrace Lean thinking – this involves being less interrupt-driven – fixing one technical problem after another, to working side-by-side with other teams to ensure a constant flow of value is delivered to business with all waste removed. As with our development colleagues, elements of agile (and common sense) thinking will come to the fore – with terms like “never done” and continuous improvement becoming established elements of the new operations and performance management mantra.

    Perhaps the biggest change IT Operations will undergo is how it interfaces with other teams. Here, IT Ops will morph into providing sets of easily accessible and repeatable processes other teams will use to bake quality into everything developed and tested. To this end, operations will transform from a separate “keeping the technical lights on” function, to a high-value to a craft that outfits the software factory and staff with new capabilities they need to ensure speed and quality at scale.

    The new age of Agile Operations “craftsmen”

    For IT Operations this means letting go of many traditional admin tasks – not just running things, but participating in crafting new automated processes. So rather than routinely monitoring applications in production, the Ops in DevOps will ensure this capability is available in pre-production. Rather than mundanely provisioning servers from change requests, new teams will equip the organizations with a complete release environments that support program level goals.

    Operating as DevOps “craftsmen” makes complete sense because it moves expertise out from behind the production curtain. The organizational focus of IT operations positively changes too — from being technically good at describing and fixing problems, to being awesome at prescribing improvements that drive better business outcomes.

    Great organizations exploit their business models with flawless operational execution. With these models now being constantly re-shaped by software applications, IT Operations is more important than ever. Relegating it with NoOps thinking isn’t an option.

     

     

     

     

     

     

  • 4 DevOps strategies to stop your application having sand kicked in its face

    4 DevOps strategies to stop your application having sand kicked in its face

    While recently thumbing through some old comic books, I got a laugh from all the quirky advertisements. Fantastic promotions of awesome gadgets – like x-ray specs and radio wrist watches. Each designed to extract a gullible kid’s hard earned allowance. Of course they were all too good to be true and probably left many kids disappointed — and maybe like me, just a tad cynical.

    I spied one ad that actually got me thinking about DevOps. It’s the famous “kick sand in your face” strip, which to those unfamiliar offered “98 pound weaklings” a sure fire program to build muscles, beat up bullies, and make lots of new friends down at the beach.

    In IT we also have weakling apps we wish could be stronger. Services so brittle and fragile that any proposed change, update or enhancement is seen as risky – perhaps even career threatening.

    So is it possible to beef up these veritable data center weaklings? Can we also find a way to turn the tables and make our apps stronger?

    Well yes we can, but only with some new thinking. But definitely not (if like kids reading comics), we get sucked in by outrageous DevOps claims, along with a raft of gadgets and tools that never quite live up to our expectations.

    Here are four things to consider as you look to toughen up your apps:

    1. Problem fixing is on the decline – successful businesses finally get that mobile and cloud is the only way to engage customers at scale. This means IT operations won’t have nearly as much stuff to keep watch over in the future. While some mourn the loss of visibility, it actually forces us to do a much better job at what’s really more important – ensuring an awesome customer experience. IT Operations as a discipline will no longer be judged on how effective it is at fixing problems, but on its ability to detect service issues before they impact customers. Therefore app toughness means less after-the-fact causal determination and more analytics for pre-emptive strikes on emerging apps problems – especially those about to bite the business.
    1. Issues now lurk in the “shadows” – developers now have lots of control. With the ascendance of agile and new techniques like test-driven-development and infrastructure-as-code, activities like build, configuration management and provisioning can be codified into work streams. This leads many to argue that operations as a function is defunct (or at least soon will be). After all, if operations doesn’t need to be involved, why is the function needed at all? Unfortunately this “shadow Ops” logic neglects to consider an inconvenient truth – basically that performance and supportability are not top-of-mind issues for developers. So while we have faster lead times, the reality could be a steady stream of weaker apps. That’s what you call “failing fast”, but not as DevOps intended it.
    1. Agile Operations is about “Craftsmanship” – since developers can’t always be counted on to instill quality, its incumbent on IT Operations to help teams build these capabilities. This doesn’t mean trying to impart years of ops knowledge, but actually crafting this expertise into automated solutions – solutions others will use to improve the performance and supportability of applications in context of the actual task being conducted. This could include:
    • Making more accurate performance data available during development so as to code and test against realistic conditions
    • Integrating change management processes with release automation so as to activate change approvals and co-ordinate all workflows – essentially ensuring compliance while eliminating typical delays from manual handoffs
    • Providing earlier guidance to developers on code defects by incorporating standardized, consistent and scalable monitoring into every new release stage
    • Working with business analysts and development to gain usability and customer experience insights from newly released mobile apps
    • Collaborating with developers and service managers to make applications easier and less costly to maintain and support – essentially ending the 3:00am head scratching and blame games!

    Looking at these examples it’s obvious the new IT Ops craftsman won’t linger behind the production curtain. On the contrary, they’ll be the ones working across the entire software pipeline; focused squarely on acquiring new skills and building solutions to drive better business outcomes. Those who blissfully remain in denial, preferring instead to watch monitoring consoles and build out physical infrastructure have a lot to lose.

    1. Application resilience is so yesterday – With on-premise applications the traditional logic has been to associate reliability with preventing and/or recovering quickly from problems. But highly distributed, scalable, cloud-native apps are very different, so our approach also has to change. These applications are the epitome of complexity (only on steroids), introduce many more moving parts, and will service unknown demand – meaning failure is inevitable. Therefore design and operational practices must move beyond break-fix, to ensuring inevitable failures have a minimal effect on customer experience and that each application release ‘learns’ from the experience – yes, actually gets stronger.

    While the old 98 pound weakling comic book advertisement is kind of hokey, many businesses are getting sand kicked in their faces by digital disruptors. So before it’s too late, start re-engineering the operational practices needed to toughen up your applications. Just remember that while rapidly developed apps now deliver your business muscle, they’ll atrophy just as quickly if you neglect quality and performance.

  • 5 Tips for Building a Winning DevOps Culture

    5 Tips for Building a Winning DevOps Culture

    Despite our cognitive prowess, humans are just like other animal in the food chain. We follow patterns of behavior based on survival. If food is safe to eat today, we’ll eat it tomorrow. If we can navigate a road without getting hurt, we’ll do it again and again.

    This explains why change is so hard. Evolutionary magic has taught us to avoid stuff that could harm or kill us – in life and even in business.

    But the business jungle is changing faster and far more radically than ever before. The pace and ubiquity of technology has resulted in once sturdy business models being brutally felled. Now we must move out of our comfort zones – or get eaten alive.

    Businesses now require regular and sustained innovation, but this is problematic for many organizations more used to operating in safe mode. For them, fear of change reigns supreme – even when the company is losing market share and talent is leaving in droves.

    The situation is recoverable when leadership commits to building culture of innovation. This is especially important in IT, because it’s here where innovation now starts and ends. To this end, organizations are rushing to embrace Agile and DevOps – the practices of choice for rapidly delivering high quality software.

    But when it comes to culture nothing is easy.  Here are five tips to help ensure success:

    1. Avoid Building Elitist DevOps Teams – the natural tendency with any new initiative is to build out a separate team. This means bringing in outside experts, cherry picking what’s seen as the best internal talent across development and operations, and then operating distinctly from the rest of IT.

    It must be remembered that DevOps isn’t a function, but a model in which development and operations collaborate. Therefore building a separate team is just creating another silo, which because of the “elitist” association could create even more friction.

    DevOps teams can valuable during early stages of a program, but only then as a transitional practice. In these case members of the team are chartered to spread knowledge; building the bridges across the organization and then disbanding once practices start to take hold.

    1. Cull Elitists and Destructive Narcissists – a strong DevOps culture is based on collaboration and empathy; sharing success and learning from failures. If, however, initiatives are led by individuals who strive for personal glory and kudos at the expense of everything – especially staff morale and wellbeing, then there’s a problem.

    Doubly difficult is the fact that people in this category are energetic extroverts; appearing ideally suited to spearheading important cultural change initiatives. But remember this – energy doesn’t always equal intelligence, so seek out the big thinkers over the big talkers. They’re just as likely to be introverts, only netter equipped with the intellectual persistence and the emotional smarts needed to help build a thriving culture.

    1. Reward Wishful Thinking and Craziness – ever wondered where great business ideas from the likes of AirBnB came from? Probably from folks challenging the status quo by creating problems customers didn’t know they had (and then solving them with software)– like – what if we could take that wasteful spare room you have and turn it into a cash generating mini-hotel?

    In a DevOps context, we can use a similar approach. Looking at the ways processes are being conducted and then suggesting counterintuitive alternatives. For example, what if our change management process is completely wrong – can we find ways improve automation for agile development? Why should we wait for production monitoring – what if we build it during coding and testing to increase resilience?

    In a strong DevOps culture suggesting ideas is openly encouraged and rewarded. Without it, great ideas get lost or unnoticed, or worse, get belittled and smashed by those elitists and narcissists.

    1. Have a Healthier Respect for Failure – if you want to kill DevOps before starting, go ahead and talk up past failures – like – “We tried agile five year ago and it was a train wreck –DevOps will be no different!” Glass half-empty thinking like this is detrimental because it put the brakes on talking risks and exploring a profitable digital future.

    Nobody likes talking about failure, but the key to making progress with DevOps is to accept that mistakes happen all the time. What’s key is treating every fail or screw-up as a learning point. That way we stop talking about things that didn’t work before, and start describing them as lessons which help build a brighter future.

    Taking the fear out of failure needs a lot of executive support. Management should begin a parallel process of rewarding sensible risk taking and removing any structures and incentives that stifle innovative thinking.

    1. Walk the People and Process Pipeline – even with great tools, DevOps won’t deliver anything if sub-optimal processes or organizational friction persist. In most cases defects and delays can be put down to people enforcing rigid policies or relying on manual and error prone handoffs.

    One good practice taken from the world of manufacturing is taking a “value walk” across your own software factory. This involves spending time interacting with people as they conduct their jobs (develop, test, release etc.) – not to point fault, but to understand any situational stresses that increase waste and technical debt, and could be remedied from automation.

    Carving out opportunities from the digital chaos and disruption is the perfect place for DevOps. But don’t get carried away – change is hard.  Make sure you have the right leadership, practitioners and counterintuitive thinking needed to start your transformation journey– and keeps everyone onboard.

  • Building a Winning Software “A-Team” with DevOps

    Building a Winning Software “A-Team” with DevOps

    The 1980’s TV show The A-team was brilliant. Four ex-commandos with special skills who help the oppressed. Throw in some over-the-top violence where no-one gets hurt, together with some tech wizardry where the team magically create weapons out of old spare parts. Add a catchy theme tune – and voila – classic viewing.

    In many ways agile software development teams want the autonomy portrayed by the A-team. And why not? Let’s bring together small teams of bad-ass code junkies. Provide them with enough open source software to build, integrate and deploy virtually anything. While we’re at it, give them the power to smash all the evil villains’ conspiring to prevent the rapid release of software for the oppressed business – like out-of-touch operations, process-crazy service management, or security and compliance police.

    So on the face of it our A-Team looks like an IT winner. They save the day and no-one gets injured, right? Well not exactly.

    When Agile Autonomy meets an Immovable Object

    Autonomy is both a blessing and a curse. On the one hand, letting teams self-organize and self-manage without outside interference seems like a productivity no-brainer. However, when team autonomy descends into selfish technology-centric rants over rational program level thinking, all hell can break loose. Suddenly every team has access to a plethora of super open source tools — need a document style database for your new Python coded microservice? No problem, click here to download, quickly brush up by reading some appdeveloper articles, and hey presto you’re off and running. Rinse and repeat for different programming languages, build tools, code versioning, configuration management and your teams are all swimming in a fast river of technical goodness where considerations about resilience, performance and security can be easily lost.

    The only problem is that the river flows down to where it eventually meets guess who – all those out-of-touch operations folks, process crazies, and security guards. Not surprisingly, they view this last minute technical deluge with a more cautionary eye. And since they’re compensated and rewarded on preventing problems, security issues and compliance exposures, they’re not going let any software pass freely into production – well, not without a fight anyway.

    And herein lies the problem with the autonomous A-Team. The “A” suddenly becomes more associated with adversarial engagement than anything remotely agile. So cue delays and the business being ticked off yet again with IT.

    Lessons from the Art World – Rembrandt and Stradivarius

    DevOps with its focus on team collaboration aims to channel the healthy tension existing between Dev and Ops into business goodness. This starts by understanding how and where to engage all that natural team angst.

    There’s a classic old sketch from the late British comedian, Tommy Cooper — who walks on stage with a painting and violin, proclaiming – “I bought a Rembrandt and a Stradivarius – the only problem is Rembrandt made lousy violins and Stradivarius couldn’t paint”.  He then smashes the violin into the painting – hilarious.

    In development we have our own Stradivarius’ who write great code, but have little appreciation on what makes a “complete” system. They can add all functional bells and whistles, but when it comes to compliance, resilience and supportability, forget it – they’re like the maestro violin maker with a paintbrush – pretty crap. Worse still, the only time they get to learn is when they’re forced to slug it out during a gated security review or regulatory compliance session – this usually doesn’t end well.

    These issues may be resolved when development, operations and other stakeholders engage much earlier in the software development lifecycle. Here as requirements are established and code begins to take shape, stakeholders can work with development; guiding them on what’s needed for the software to become more secure, perform optimally and be a breeze to support.  For starters, this has to include IT Operations sharing years of expertise – like how to exploit monitoring tools in pre-production to equip developers with an early warning system on performance issues as they code, or explaining what’s needed to support new distributed architectures at scale.

    Engage Early and Broadly for Success at Scale

    Beyond operations, there are many other IT and non-IT stakeholders who should be drafted into the A-Team.

    When working on a new web to back-end system integration for example, security could advise development on end-to-end transaction tracing and static-code analysis techniques to help detect code or SQL vulnerabilities. In an API project, threat teams should engage early to model and mitigate against many new attack vectors that hackers can exploit, providing advice on strong, API-specific infrastructure and management. Naturally, if your team is doing anything with mobile payments you better hunker down early with the security and PCI compliance team – or if the agile team has devised some neat sensor, software and cloud technologies to capture and analyze customer-centric data ala the Internet of Things, then early involvement with the legal department on privacy gotchas is a must.

    Large scale success from agile teams can be realized when those teams become less feral about their own individual needs and more focused on supporting business objectives at a program level. To this end, and whatever the makeup of individual teams, it’s vital to establish DevOps style collaboration early in the lifecycle; engaging all business and IT stakeholders in order to establish resilience, quality, security and even supportability into the fabric of software as its developed.

    Do this and you’ll deliver great software without ever having to compromise speed. Fail, and your agile efforts may just stutter and fail – of course you’ll still have an A-Team, but the “A” now stands for something far less complementary!

  • DevOps and legacy system lessons from dwarf planet Pluto

    DevOps and legacy system lessons from dwarf planet Pluto

    If you’re a bit of a space geek like me, then maybe you’re one of the 243 thousand plus followers of #NASANewHorizons; diligently monitoring deep space twitter sphere for the latest news and pictures from dwarf planet Pluto.

    The success of the New Horizons project is incredible – especially when you consider it’s all been delivered by legacy technology. Technology that lifted off back in 2006 – when Pluto was a real planet and before we had the iPhone, mobile apps, social media, Big Data, the Internet of Things, oh and DevOps.

    So why did this nine year old piano-sized hunk of tech deliver so impressively, and what lessons are there for those of us in IT managing our own legacy systems and applications? Actually there’s a lot we can learn from the outer reaches of our solar system.

    Even in Deep Space constraints can be managed

    The New Horizons space probe is an extreme example of a constrained physical system. Three billion miles from home, command sequences take 4.5 hours to reach the probe and the consequences of screwing as it barrels along at a lazy 30K mph are, well,  pretty severe. Of course NASA scientists have a fallback – they use a simulator with engineering and telemetry models to test the commands that perform all the vital spacecraft activities.

    In business computing we don’t have the luxury of having spare hardware of tap – especially for large legacy systems. Too often access to systems and data needed for development testing is constrained due to schedules, contention, security restrictions or service provider usage fees. To address this many organizations attempt costly provisioning of lab infrastructure, but these often don’t fully reflect production condition.

    As with New Horizons, constraints within IT can be eliminated with technologies that model and simulate the behavior of dependent systems. This not only lowers infrastructure costs, but also helps teams move testing activities earlier in the lifecycle, where in parallel with development help detect software defects well before they impact production systems.

    Change and release co-ordination is critical

    Just days out from its rendezvous with Pluto, New Horizons suffered a minor glitch and lost contact with the earth. For a $700 million probe at the edge of the solar system, that caused a few jitters. Then after a day of analysis (and maybe some head scratching), mission control tracked the problem to a timing flaw in the spacecraft command sequence. Basically, the main computer went into safe mode after being asked to do two intensive tasks at once.

    In enterprise tech our own legacy applications play an essential role in supporting key business processes. As such, and like New Horizons, changes must be carefully coordinated. What’s different, however, is the scale of the challenge. Multi-tier release deployments by nature are inherently complex with many dependencies. Automation therefore is critical, as are reusable and repeatable deployment processes to streamline application releases involving legacy systems.

    Data can be exposed and utilized in different ways

    It’s interesting to see how NASA have exploited social media for the New Horizons mission. Thanks to blogging, podcasts and Facebook all the news and fantastic images are now easily accessible. From the web-site too, the twitter API has been exploited to provide a constant channel of updates and facts about the legacy technology – did you know for example that the original PlayStation CPU (MIPS R3000) is used in the probe? – I guess if it worked solidly for a 100 million game units it can fire some thrusters 3.5 million miles out in space.

    Businesses operating in an economy governed by software applications can take a lesson from the New Horizons mission. Application Programming Interfaces (APIs) are now central to the exchange of value between companies, so it makes complete sense to use them against legacy systems and line of business applications.  By securely unlocking the application data (and business processes) through API’s and by seamlessly managing message mediation and translation, organizations can have these older systems quickly participate in more modern processing and customer-centric computing.

    Every system is a legacy system

    Just as we rush to embrace every shiny new tech object, we can be equally hasty berating the value of existing systems and applications. To this end, legacy becomes a term of derision; associated with everything bad about enterprise computing. The more pragmatic among us know this isn’t so. After all these systems support a plethora of back-end business processes and continue to be the transactional work-horses upon which many businesses depends.

    There’s a lot we can learn from the New Horizons mission. Even now scientists are saying that the spacecraft has decade’s worth of exploration left in it. So with some quick course adjustments it’s off to explore the next icy world.

    Not bad for old space tech – perhaps in business computing we should be thinking the same way – extracting value before prematurely consigning our own legacies to application scrap-heap.

  • DevOps Empathicus – Where is your organization on the evolutionary scale?

    DevOps Empathicus – Where is your organization on the evolutionary scale?

    Ok, I admit it – I sometimes cry when watching sad movies or as real-life events play out in the news. I put myself in the position of other people, sharing their joy and pain. This got me thinking about change-based initiatives like DevOps and what benefits organizations can realize when IT staff have the support to do what should come naturally. That is, considering what it’s like to be on the ‘other side’ – whether the other side means Dev, Ops and Security or as a customer, and then collaborating to drive improvements.

    Below is a “scale” that can be used to assess the empathetic maturity of your organization. Failing to acknowledge (however painful) your organization’s maturity will compromise any hope of success with DevOps – no matter what tools or would be process improvements you put into place.

    1. Hunter Gatherers – at the lowest point on the evolutionary scale, these organizations have a preponderance of what can be considered IT foragers and scavengers. Singularly focused, their survival depends on technical dexterity in the harsh environments they operate. Using specialized tools and with little management oversight, their sustenance is every technical issue and problem. They trust no-one, thriving instead on the ‘hero status’ they’ve acquired over many years of keeping the technical lights on. To them, DevOps is like every other change initiative – a painful and unnecessary distraction they’ll attempt to discredit – by doing what every good hunter gather does — beating it to death.

    To ferret out hunter gatherers, I’d suggest a simple test. Ask everyone how they dealt with the last Severity 1 outage. If they start by blaming other folks (even on their own team), or claiming that recovery was down to them alone, then they probably should be kept away from a DevOps style initiative. If however, the quiet introspective Ops guy fesses up that he calls for help from his pals in security to tackle problems, or that he’s even sharing monitoring tools with Dev to improve code quality, there’s hope that teams can hang up the spears.

    1. Tribes and Clans – moving up the empathetic scale, these organizations are characterized by teams of technology domain-based specialists – like the NOC clan or the service center tribe. True, these groups share each other’s pain, but this is almost always (in their minds) brought about by the habits of competing tribes. Here, every release delay, lengthy compliance check, outage and problem, is caused by another team – like the hipster, tofu loving developers or those hard-nosed security and compliance specialists.

    With IT teams being traditionally organized by technology function, it’s not surprising that tribalism persists. Many organizations attempt to resolve the issue by for example co-locating teams. But without putting some shared skin into the mix that’s going to be as successful as a TV celebrity sharing some deep thoughts on Twitter. Therefore, start by finding some common ground to help unite teams. A shared area might help for sure — but better still are shared outcome-driven business metrics with linked incentives.

    1. Religious Cults – best practices and methodologies like agile development, ITIL, Cobit and Lean are great for bringing similar like-minded tribes together. Unfortunately, pragmatic application often gives way to dogmatic enforcement; with crusade-like implementations promising much, but never delivering much in the way of business value.

    DevOps itself can still fail miserably if organizations succumb to all the hype and hyperbole; readily accepting rants that whatever preceded it has to be bad. More mature IT teams look beyond industry spin – accepting that many existing practices can have a place in supporting the goal of rapidly delivering high quality software. For example, incorporating existing integrated ITIL incident and problem management processes during pre-production to build and circulate knowledge, or even automating compliance procedures during testing to both harden security and prevent delays.

    1. DevOps Empathicus – it’s no surprise that the most successful organizations display high degrees of empathy. They care about their colleagues and customers. They collectively swarm issues, treating problems as the enemy. For these organizations the ‘smartest guy in the room’ syndrome is a thing of the past – with acceptance that people innovating solutions don’t have to be MBA grads or domain experts.

    But while we can hope that organizations naturally evolve to this nirvana state, it just doesn’t happen that way. It takes subtle intervention from management to get people from a state of just caring about the technology to being deeply connected with the business problems they are trying to solve. Sure, that sounds fanciful, but the first time you see a high-five because the business just exceeded its Net Promoter goal or they’re measuring downtime in terms revenue lost not milliseconds, you’ll appreciate the power and value of empathy — Then perhaps like me you can get out the tissues and have a darned good cry.

    So where does your organization fall in this scale?