devopsy.kinsta.cloud

Tag: operations

  • DevOps and Collaboration: Fraternizing with the Enemy?

    DevOps and Collaboration: Fraternizing with the Enemy?

    DevOps is all about collaboration, we hear.
    “You must collaborate,” say managers initiating DevOps adoption programs.
    HR make new posters declaring the corporate values “We value Collaboration & Teamwork.”
    Can’t fail!
    Then when it appears not to be as effective as we had hoped we tell teams they are “Empowered” to collaborate.
    Blank stares!
    When things still don’t appear to be working we tell teams they are “self-steering,” “self-organizing, collaborative teams”….
    Still, we don’t get what we were hoping for.
    Luckily the tool vendors come to the rescue and tell us why it is failing. We need “collaboration tools and software” – Throw some “Slack” at it.
    Can’t fail!

    This is when we usually get frustrated managers knocking on our door asking about the Phoenix Project DevOps simulation. They are moaning about communication and collaboration! When we talk to the teams they are frustrated and moaning about “empowerment” and managers not “walking the talk.”

    In my mind these lack of collaboration skills is not really surprising. Why?

    Impression: Well, it doesn’t help when the word has also historically been connected to global conflict situations. “He faces charges of collaboration,” “fraternizing with the enemy.” Dev & Ops. Traditional adversaries?

    Interpretation: The interpretation of the word itself is also part of the problem. I googled an English Language Help Desk:

    Collaboration: “To work together with somebody in order to achieve a single shared goal.”
    Cooperation: “Work with other people by achieving one”s own goals as part of a common goal.”

    In an HBR article titled, “There’s a difference between cooperation and collaboration,” it was argued  “… Managers mistake cooperativeness for being collaborative,” adding, “… most managers are cooperative, friendly, willing to share information – but lack the ability and flexibility to align their goals and resources with others in real time.” If managers don’t get it then what chance do the teams have when told – “You must collaborate”!

    Induction: Part of our failure to be able to effectively collaborate could come from our schooling. We are chastised at school for asking others for help, for looking to see how somebody else is doing it—”That is cheating”—things are changing with more “collaborative assignments.” Yet, my children (now in their early 20s) were not taught what effective collaboration is, other than discovering for themselves. As my daughter explained about her school assignment, “You have to do it yourself if you want it done because others will slink off and not do it, then my individual grade may suffer.” They are told they need to collaborate but individual performance grades are scored. My daughter once came home stressed and almost in tears, declaring “Oh no! I have been given another collaborative assignment for my most important subject!” Children leave schools distrustful of collaborating.

    Indoctrination and Imperative: There is also the historical perspective. When we, the current workforce entered our chosen profession we had decades of functional silos that created a “them” and “us” attitude. This stimulates a sort of “territorial imperative” or “tribalism,” not helped by our own “technology” languages and tribal cultures. We have learned over time that other teams “dump things on us” or other teams “stop us from getting stuff done.” In other words, Other teams frustrate work.  (Territorial imperative—”the drive in animals and man to take, hold and defend a particular area, zone, turf,” —e.g. defending their best practice framework! And attacking others that come near!).

    Inconsistency: We then built walls between these teams and if they wanted something from each other they needed to escalate it to their manager. Managers would collaborate or not, especially when they were given conflicting, inconsistent goals and KPIs. Development teams are given goals to release as fast and as often as possible. Operations are given goals to make sure everything is as reliable as possible and available at all times.

    I had an example of this “them and us attitude,” once with a board of directors. I split them into two teams and gave each team a pack of ABC cards (Attitude, Behavior and Culture) and within 10 seconds heard, “Oh good, our team will win this!” automatically creating an atmosphere of competition and winning and losing, I hadn’t even explained the task yet! Teams compete.

    ‘It’ factor: A study by Google (2012) called Project Aristotle, named in tribute to Aristotle’s quote, “The whole is greater than the sum of its parts,” looked into why some teams work and others don’t. One of the key conclusions was that “psychological safety” was the “it” factor, “… a sense of confidence that the team will not embarrass, reject or punish someone for speaking up.” Because we have not learned to collaborate and because we have a “them and us” history often accompanied by blame, there is fear, uncertainty and doubt (FUD) about speaking up, sharing ideas or giving feedback.

    In the dark: However, there is another significant reason for poor collaboration, as I have discovered from running hundreds of Phoenix Project simulations with teams globally. It is the fact that nobody has defined the “behaviors” that demonstrate effective collaboration. We assume everybody knows what we mean when we say, “You must collaborate.” Fact: We don’t!

    Below are the results of a series of recent Phoenix Project simulations. One of the goals was to learn DevOps practices such as CALMS and the Three Ways, while another goal was to learn effective team-working and collaboration.

    I asked the teams at the start of the sessions, “What is effective collaboration? What behaviors will I see today that demonstrate this”?  The teams came up with a list of behaviors (each team came up with a top five or six behaviors; this is the consolidated list from seven teams).

    • Transparent communication – “open and honest” and “frequent.”
    • Calm and constructive dialogue (no blame, accusations and shouting).
    • Feedback “asking for and giving feedback.” Constructive feedback.
    • Shared team goals (and cross team). What are we aiming for? Impact on others and their goals. Maybe conflicts that need managing.
    • Listen – is it clear and understood? Show that you listen.
    • Being open for dialogue, being approachable –not closing the door or hiding behind.
    • Giving and taking responsibility.
    • Sticking to agreements.
    • Central moment to get together (e-2-e).
    • Knowing the roles and responsibilities, sticking to responsibilities.
    • Mistakes are OK.
    • Be proactive to get information needed.
    • Equal treatment: “Respect” ideas from ALL
    • Don’t HOPE, confirm.
    • “Willingness.” Confirm and agree.
    • Ownership of responsibilities.
    • Clear agenda and purpose of meetings.
    • Pleasure – celebrating success.

    “Are these behaviors you want to see more of in your daily work?” I asked.

    Yes! Was the answer.

    “Do we now all commit to sticking to these behaviors today, in this simulation?

    Yes! Was the answer. Easy! See there’s nothing to this collaboration stuff.

    However …

    During the simulation it soon became evident that roles and responsibilities were not clear or not being carried out; it was not agreed which information was needed by the business nor when; goals were unclear, meaning that priorities were arbitrary; people did not understand procedures and did not ask for help; assumptions were made (“I told you …,” “I thought you meant …”); people taking responsibility were not respected (“Let’s stop and try and sort this out …”)— everybody just carried on running around! “Headless chickens,” said one delegate; people were not listening to each other; there was a lot of “assumicide”; mistakes were made and people blamed—there was no exploration as to why and what can be done to stop these  mistakes. Mistakes were often corrected by somebody else but without giving any feedback to correct future mistakes; agreements were unclear across the team; there was no end-to-end flow—work and information was ping-ponging back and forward.

    “All in all, just like reality!” declared the teams.

    See how easy collaboration is. Within one hour of making a list of desirable behaviors, they were totally, utterly and universally ignored. Nobody appeared to own these behaviors or to confront each other on them.

    After the Experiences in Simulation

    At the end of the simulation round we looked at the team scores (number of releases, number of successful releases, business value realized). It wasn’t a pretty sight. The CEO wasn’t happy. The team felt stressed, unhappy, frustrated and powerless.

    We explored what this had to do with “effective collaboration.” Had anybody used the list of behaviors? No.
    Had anybody consciously practiced these behaviors?
    No.
    Did anybody give feedback on these behaviors when they were clearly not done?
    No.
    “Giving feedback feels strange, it feels like we are criticizing and blaming,” said one delegate. Another assumption!

    “It’s hard,” said one delegate. “The manager needs to manage this,” said another.
    “It was YOUR list and YOU all agreed this, why is this a manager’s job to get you to do what you said you wanted to do?”
    There was silence. “It’s not easy, we fall back into old ways of doing things, we need help!”

    Is it a Bird, Is it a Plane, Is it a Bus? No, it’s a coach!

    As the simulation progressed, the team refined their list of desirable behaviors based upon their experiences. They were coached for two rounds and consciously practiced and experimented with the behaviors. The coach role became a critical catalyst.

    In the final round the team owned, and were executing on these behaviors without coaching.
    Why were they suddenly executing?
    They felt less stressed; they were trusting each other; they felt the positive impact on their work; they saw, felt and shared the value gained by these behaviors. They felt immediate positive consequences from these behaviors.
    Why no more coaching?
    The team were taking it in turns to lead their own stand-ups; they were giving feedback and correcting their own behaviors. These were the additions the teams made to the behaviors as they practiced them.

    • Transparent communication – “open and honest” and “frequent” (Relevant, and timely as agreed with the various stakeholders).
    • Feedback “asking for and giving feedback.” Constructive feedback. (Related to agreed behaviors and responsibilities, calling stop, then swarming to solve an impediment, constraint, or remove waste; immediate fact based feedback).
    • Shared team goals (and cross team). What are we aiming for? Impact on others and their goals. Maybe conflicts that need managing. Understanding strategic portfolio and how we contribute to strategy, show you understand and agree common goals and prioritize accordingly. Goals for both value creation (features and benefits) and value leakage (risks, debt).
    • Leadership within team – team members took turns in facilitating the stand-up and retrospective. This role was respected.
    • Listen – is it clear and understood? Show that you listen, sender: Ask for clarification, ask to confirm understanding, listener: summarize what was understood and agreed. Avoid assumptions.
    • Being open for dialogue, being approachable—not closing the door or hiding behind. All were engaged and equal in the stand-up and retrospective. All were asked for feedback, ideas, suggestions.
    • Giving and taking responsibility. Respecting team behaviors; e.g if somebody calls “stop,” or “respect somebody taking responsibility for an improvement stand-up.”
    • Sticking to agreements. Confront people to understand why agreement not met and what can be done about it in the future (e.g conflicting priorities, lack of understanding).
    • Central moment to get together (e-2-e). Scheduled stand-ups, retrospectives, and improvement prioritization.
    • Knowing the roles and responsibilities, sticking to responsibilities, confirming your responsibilities, confronting people on responsibilities (feedback). Showing that you take ownership.
    • Mistakes are OK – no blame, so long as it is used as an opportunity to learn and improve.
    • Be proactive to get information needed (or ensure information is available, e.g. visible management board, avoid unnecessary “disruptions).
    • Equal treatment: “Respect” ideas from ALL. Ask people for input, engage ALL in ideas generation and decision-making.
    • Don’t HOPE, confirm.
    • “Willingness.” Confirm and agree – “Willingness” and “ability,” empower to be able. Share knowledge, cross train, coach, help.
    • Ownership of responsibilities. Confronting people on responsibilities.
    • Clear agenda and purpose of meetings. My role, what is expected from me? What is in it for me? avoid “wasting” time in meetings.
    • Team coaching to give feedback on behaviors.
    • Create trust by demonstrating team behaviors.
    • Check understanding and commitment to “agreements” and confront people (feedback) on agreements.
    • Improving own work as a team.
    • Asking for and offering help.
    • Having insight, overview, visibility.
    • Limit WIP (reserve WIP for learning, improving, technical debt).
    • Pleasure – celebrating success. Product owner shared results including customer feedback. Team complemented each other.

    The teams were then asked to make a list of behaviors and actions they wanted to take away and apply. Their line managers and assigned team coaches were there when the list was made and gave commitment to help guide, coach and empower teams to experiment and practice these new behaviors in their daily work.

    The team would prioritize one or two behaviors to start with and consistently practice in the coming weeks. When the new behavior becomes the accepted norm and a “habit” they would focus on more behaviors.  This fits in with the Third Way of DevOps: Continuous experimentation and learning. “Understanding that repetition and practice is the prerequisite to mastery.” And to tie this back to Aristotle the philosopher (not the Google project): “Excellence is an art won by training and habituation … We are what we repeatedly do. Excellence, then, is not an act but a habit.”

    Meetings, Including Training Sessions!

    And finally, one of our industry’s most popular, and wasteful, examples of collaboration are poorly run meetings, which are the biggest time-sink in organizations and a source of frustration. I include some training classes as forms of meetings.

    In more than 100 training sessions in the last year, less than 10 percent have started on time.
    Nine minutes late is the average. Delegates say this is normal in their organizations. When I ask people what their own personal learning goals are for sitting eight hours in this “training meeting,” more than 25 percent had not defined this and more than 70 percent had not agreed this with the “sponsor” of the training.

    We need to manage our overload of wasted meeting effort. How? One way is the ask for goal and agenda in advance, asking what value you need to bring in and what value you need to get from the meeting; also to limit your time to the agenda items relevant for you or your team.

    Efficient collaborators decide when they do, or don’t, have unique value to add—“maintaining focus on what matters.”

    The same applies to training, asking, “Why am I going on the training?” “What am I expected to learn?” “What problem is this training expected to help solve?”  “What will I be expected to do differently after the training?” “How will the learning be transferred back to the workplace?” “Will I be mentored and coached?” “Will I be given opportunities to experiment?” “Will I get constructive feedback?” “How can we demonstrate that the new knowledge delivers expected benefits?” This is better than the most common answer, “I don’t know I was told to come to this training!”

    A training is also a form of collaboration between you, the trainer and the person sponsoring the training.

    There is an “I” in Team, after all. “I” am responsible for my effective contribution to team performance.

    — Paul Wilkinson

  • Software Delivery: Shaking Loose of a Local Optimum

    Software Delivery: Shaking Loose of a Local Optimum

    This is an article about software delivery, that magical or onerous field between pushing code to version control and using it in production. Why does software delivery matter? Because software matters, and because software developers matter. I don’t want my software delivery to get out of the way; I want it to make me a stronger developer.

    The path from code to production is crucial because it determines the length of our feedback loop and, therefore, our rate of improvement and that all-important lead time.

    Our current tools think of delivery as generic—they take control, and we fill in the blanks. They let us build, test, deploy. But these words mean something different for every application, on every team and within every business. And there’s a lot more that goes on, or can go on, between the writing and the running of code. There’s often linting, approving and coordinating. There’s data to think about and security policies, business team structure and auditing. There are other steps we don’t bother with, because there’s no slot for them: updating libraries, fixing the code to meet standards, taking a different set of steps for a documentation-only change.

    What else have we not thought of, because it doesn’t fit inside the current set of tools? I want to think about delivery from the top. In doing so, I can gain a greater understanding of my work.

    We’ve seen major shifts in Operations technology: containers, log aggregation, observability, programmable infrastructure and Kubernetes. In my opinion, the killer feature of Kubernetes is its API—it is controllable by automation. It accepts instructions and sends out events.

    The potential here is even bigger than we have realized, for operations and for delivery.

    Modern options for operations have achieved new levels of abstraction and potential. Yet progress in delivery has only been incremental.

    Our tools are very configurable, with plugins and scriptable bits, and we can always fall back to running shell scripts of unlimited length. But are we stuck in a local optimum? Are we thinking big enough?

    What We Have

    Today’s popular software delivery tools are fill-in-the-blank. This is great for getting started, when you don’t know what you’re doing. What about when your business grows, when your delivery is no longer cookie-cutter? Let’s look at two examples.

    Travis CI is a favorite of mine because it’s so easy to get going. It will happily build any public GitHub repository, on its infrastructure, for free. If my repository fits a few simple profiles, such as a Rails app that deploys to Heroku, poof! Some very simple YAML in your repository:

    I like that YAML. It is declarative. It contains data. It fills in the blanks that Travis provides. But the real measure of a tool comes when you get off the happy path. You can do just about anything in Travis; just call out to a shell script. Here’s a small piece of one of my .travis.yml files:

    It has shell commands. Shell syntax for an if-condition. And that infinity of flexibility, a call to a shell script. That travis-build.bash script happens to contain four functions, eight conditionals and (if I’m reading it correctly) up to seven program calls.

    This is programming, OK? As soon as your YAML calls out to a shell, it is not declarative. It is imperative. Or a mix. 😕

    This is programming, but my YAML and Bash do not qualify as software. Bash is a scripting language. It is designed to glue programs together, not to construct programs; it lacks composable modules with encapsulation. I can live without types, but in Bash you don’t even declare function parameters! And it’s full of pitfalls. Use Bash for what it’s good at: calling other programs. As a fallback for flexibility that we lack at a higher level, Bash does not shine.

    YAML is not a programming language, although it does support variables and has its share of pitfalls. Until the YAML contains commands that get executed; then it becomes a program, with even fewer options for error-handling or clear naming than in Bash. Please use YAML for data, not for code.

    There is a gulf between programming and software. We have standards for software: it has design—domain-driven design that we’ve worked through with experts. It has automated tests. It has modularity. It uses libraries and abstractions. I want to use a language designed for all this.

    Skip ahead to Jenkins. This is more widely used than Travis for more extensive delivery flows. It can run on-prem. There are hundreds of plugins available. So many blanks to fill! So many options! And now, there’s pipeline as code. For one thing, you can write a piece of Groovy to define which repositories and branches get a pipeline. That makes Jenkins more responsive to new repositories and branches. It’s a step in the direction of responding to new projects.

    The people building Jenkins recognize that delivery needs to be controlled by the people who control the code. Instead of constructing each repository’s build in a GUI (hello, TeamCity), we put a Jenkinsfile in each repository. That file is in Groovy. Groovy is a programming language with all the modularity and libraries of the JVM behind it. Except, you don’t get all that in your Jenkinsfile. You’re limited to what is available when Groovy runs within Jenkins. If you want consistency from one repository to another, you can install custom-shared libraries in Jenkins and call them in each Jenkinsfile. Rolling out change is challenging; any of those disparate Jenkinsfiles can break after the shared library changes. It’s pretty fragile and still limited. It’s progress.

    This is a step up as programming, but I still wouldn’t call it software. When I write an application, I work in more than a programming language; I work in a language system. A language system includes dependency management, frameworks, linters, build systems, editors, community and the whole ecosystem of libraries available in the language. Great language systems exist around the JVM, Node and .NET, for instance. Groovy that you run can participate in the JVM language system, but Groovy running within Jenkins does not have this access. The Jenkinsfile DSL is still fill-in-the-blank.

    What We Need

    Conceiving delivery as software invites us, development team members, to sit down with the local experts on our deployment environment and security and other concerns. It invites us to have a conversation: What does delivery mean for this application, on this team, in this organization? Where do we want consistency and variability? Then we can create and encode a model—a model of the domain of software delivery, within the context of our business and our environment. We can brainstorm events such as a push or a new repository and what we want to always happen. We can divide the parts into team-specific, organization-specific and community-wide (open source).

    Instead of, “How do run the build? How do we run tests?” we can ask, “What needs to happen after a code push?” And we can think carefully about how we work, how our software works and how our business works with it. We are the domain experts in our own delivery.

    This is the top reason I want delivery as software from the top down. By thinking about delivery as software, we think rigorously about our work as software developers. And we get better at it.

    I want more than delivery-with-some-code. I want delivery as software. Controlled top-down by my team, with a rich ecosystem of libraries, with frameworks and services for the parts that are common in the community. I want delivery that’s event-driven, that’s testable and that communicates with me when it’s time for a decision.

    We’ve come a long way since the days before automated deployment. Our existing continuous integration/continuous deployment solutions have served us as far as they can. Incrementally better (fill-in-the-blank) won’t take us to the next level of software delivery, and it won’t help us reach the next level as software developers. It’s time to break out of this local optimum.

    Let’s Go

    To this end, we composed the Software Defined Delivery Manifesto. Say what you want about manifestos: They have a certain flair, a certain impact, a certain dramatic longing for a better world. That is what we are expressing here. Yes, it’s a better world only for development teams, and only for development teams in complex organizations. But we matter! Software has a huge impact on the world, and to make that software better, we need to get better at delivering it. Let’s push past what we’re used to. Accept that our job is not writing code, but running useful software in production. Let’s get better at delivery by modeling it in software.

    If you’re in, join the movement. Sign the manifesto.

    — Jessica Kerr

  • Ops: The Other DevOps

    Ops: The Other DevOps

    Much of the conversation around DevOps, including right here on devopsy.kinsta.cloud, is about helping developers get easier access to production systems, increasing agility by shortening release cycles. Most of the time, this focus on developers leads to a concentration on so-called “Day One” tasks, the Dev side of the equation, to the exclusion of “Day Two” operations—the Ops side of things.

    Cows in a field with farmer
    Photo via Damien Kuhn – https://unsplash.com/@producteurslocaux

    Recent moves in the DevOps market underscore the fact that the weight is still very much on the Dev side, and not so much on Ops. Microsoft buying GitHub is huge news, don’t get me wrong, not least because many people are still stuck in a very Nineties conception of Microsoft, and this acquisition should help clarify that this is not your parents’ Microsoft (full disclosure: my employer, Moogsoft, has a strategic partnership with Microsoft). News on the Ops side of the house, meanwhile, rarely achieves the same prominence.

    The reason for this imbalance is to be found in DevOps’ origin myth. The story is all about developer frustration with risk-averse operations procedures which prevented them from deploying as rapidly and frequently as they would have liked to. As with most good stories, there is a kernel of truth here, but the truth is more complicated. Ops people are risk-averse for good reason: When they mess up, they don’t have a safety net, and they get a lot of very public blame.

    That said, it is certainly true that a reasonable aversion to risk can be taken too far. Couple that with a very human tendency to define oneself by one’s job, and you wind up with people putting hands to keyboards for even basic, routine tasks, becoming bottlenecks for huge processes.

    What Does DevOps Mean To Ops?

    The biggest benefit of DevOps from an Ops perspective is that it has forced that hard conversation about moving from artisanal, handcrafted IT infrastructure, to industrial and scalable models that don’t require a human in the loop for routine operation. This cultural shift began with the famous pets-versus-cattle analogy, but the transition is far from being complete.

    A big sign that there is still work to be done is the relative lack of sophistication around tooling  in IT Operations. There is a lot of talk about software-defined infrastructure and desired-state automation, but relatively little practical uptake, especially when it comes to operating such frameworks day-to-day. These new models will require a wrenching transition from traditional Ops models, with pervasive automation working to augment human sysadmins’ capabilities.

    This is not the old style of automation, with cron jobs and shell scripts, which any BOfH worth their etherkiller has been doing for decades. The new serverless frameworks do away with the last vestiges of persistent coupling of infrastructure components to individual business services. What vMotion promised in 2003 is now coming to pass, with workloads moving, growing and shrinking to support changes in user demand, and infrastructure reconfiguring itself top to bottom in response.

    Key Questions To Ask About Operational Automation

    In this context, operational automation can no longer work on the old model of “set it and forget it.” Every process set in motion needs real-time contextual awareness of what is going on around it and how that is constantly changing. Without that visibility, backup scripts may go on faithfully backing up what used to be a high-priority server, but is now a backwater of the application infrastructure. Meanwhile, the cloud services where the bulk of the application’s logic runs are not meaningfully integrated into the disaster recovery (DR) plan.

    All of these facets need to be reviewed regularly, in terms of need, function and ownership.

    Who Needs This Automation?

    Anything that runs should be linked to some kind of business requirement, however indirectly. “This ensures that our infrastructure can scale and flex seamlessly in response to user demand”: Good. “This keeps our operating systems patched”: A valid goal, but do we need to do it in production if the average lifetime of provisioned operating systems is shorter than the patch cycle? Just patch the templates and you’re done. “We’ve always done it this way, and I’m not sure what might happen if we stop”: Time to set some chaos monkeys loose, and see what happens.

    What is the Function of This Automation?

    We are in the middle of huge technological transitions, and the waves of innovation keep coming closer and closer together. As an industry, we had just about come to terms with virtualization by the time cloud computing came along, but that was far from settled when containerization got underway, and it looks like that may be undermined by serverless architectures before it even comes to its full fruition. Watch out for hidden assumptions in terms of the way you go about a task, and check back in regularly. Ops types should take a leaf from Dev’s book, and regularly consider refactoring their automation to remove cruft and take advantage of new developments.

    Who is the Owner of This Automation?

    Someone has to own every piece of automation, so that they can maintain it over time according to the previous two points, and so that everyone else knows who to call if (or realistically, when) it breaks. Again, developers have been talking about technical debt for a long time: the impact of older, unmaintained code. Meanwhile, sysadmins used to pride themselves on the uptime of individual servers. That attachment needs to move from the individual server to the overall infrastructure and the automation and management that develop over time to ensure its availability and performance. They’re cattle, not pets, remember?

    Ops is the Other DevOps

    As Ops becomes more and more about managing the automation, the distinction between Dev and Ops starts to blur. With Dev roles “picking up the pager” and taking responsibility for production operation of their software, Ops roles develop more and more code to fulfill the promises of those applications. With traditional Ops tasks evolving or even disappearing, new roles will appear. The job of automation architect will become much more relevant than any job that requires someone to ssh into an individual server or router. In this transition, old-school Ops people can learn a lot from what our Dev colleagues have been up to over the last few years.

    One day, the term DevOps will no longer describe developers and non-developers, but two different types of developer, one focused on business logic and one on architectural underpinnings.

    — Dominic Wellington

  • 4 Bad Habits DevOps Administrators Fall Into

    4 Bad Habits DevOps Administrators Fall Into

    The concept of DevOps is central to any organization’s ability to deliver applications and services quickly and efficiently. As it gains traction in many high-profile organizations, DevOps administrators must work to integrate DevOps practices, tools and techniques into a wide range of teams, from development and support to operations and management. Completing this implementation seamlessly leads to better productivity and a smoother overall workflow.

    Bad Habits Derail DevOps

    But along the way, administrators tend to fall into bad habits that affect their ability to deliver these services effectively and efficiently. Here are a few of the most common habits that need to be broken.

    Working in Silos

    Some DevOps teams work in silos, in which development isn’t just one team but many. Think about a group project where everyone divides and conquers on tasks and at the end comes together to integrate different parts, which in turn have different formats, styles, etc.

    This is a bad habit that needs to be broken as it produces an additional level of complexity, with everyone working to solve their own issue at hand in their silo. Operations then struggle to unify the activity into a consistent process. This, coupled with occasional tensions between development and IT operations over different approaches to infrastructure and competition for resources, creates opportunities for failure. As the growing emphasis on DevOps places an added burden on infrastructure teams already struggling to cope with the digital transformation of their organizations, unification will be key to success.

    It is also true that DevOps can find success in small organizations where teams of people are focused on smaller, one-off IT projects, rather than the enterprise as a whole. These smaller teams have the luxury of sticking to themselves and the allowance to work on sequential projects, while their projects don’t scale up to the enterprise stage because they don’t have to. But smaller organizations such as these also likely won’t see much of a difference if they were to adopt a DevOps approach.

    That difference will be most noticeable in substantially larger organizations that see software release cycles rapidly changing on a daily basis. It is imperative for these organizations to avoid silos and to instead have pronounced team roles, better communication and the ability and willingness to break free of old habits that only cause chaos and confusion in the rapidly evolving delivery cycle.

    Planning Too Much, Too Soon

    Success requires continual improvement and refinement, rather than an initial push for certain changes without an overarching plan. The focus of administrators looking to implement DevOps should be on doing all things continuously by starting small.

    Early focus should be on selecting one or two new DevOps teams that start a new initiative. Then, with these new teams in place, you can build on successes and work through challenges slowly, and begin to scale this through the organization. In the beginning of the process, it’s also important to identify one person such as director of operations or chief planner who can be responsible for seamlessly integrating DevOps into your project. This lead owner can then work project by project, setting goals and showing the rest of the team the results in real time. Insight into the progress will allow the team to see their work paying off, and in turn, make it possible for them to realize the benefits of scaling up slowly.

    You need to know what the business problem is and the anticipated benefits, therefore measuring every factor that contributes to DevOps success. DevOps promises an array of automation. This can’t be achieved in a single attempt or overnight. Enterprises have to focus efforts on continuous integration, following it with continuous delivery and eventually continuous deployment.

    Believing Implementation is Easy

    After admins are finally swayed by the promise of DevOps, a belief often held is that the implementation of the practice is smooth and easy. And while DevOps can often lead to a simpler life for specific jobs in IT, the reality is that in many instances, the ease of deployment comes at a cost. It can shift an imbalance of responsibility for production control and quality management to the operations side, and burden the team rules and processes if IT departments fail to understand how to adapt.

    As with any process in any organization, the amount of effort required to support software delivery is a constant. Often, administrators fall into the habit of believing DevOps will handle itself. But if you are managing a large enterprise, you will have teams accountable for quality, performance and change management. The difference with DevOps is the way in which automation is leveraged to place control in the hands of application development teams and redefines the relationship between operations and development. This is a good thing, but the administrators need to be there every step of the way in the process.

    Failing to Take Full Advantage of Continuous Integration and Delivery

    Continuous integration and continuous delivery (CI/CD) are key to successful DevOps teams. However, a bad habit many DevOps teams can fall into is failing to take full advantage of all the CI/CD benefits. The process must to be fast and efficient, as anything that slows down any part of the process increases the cycle time and slows down delivery. To break the habit, DevOps teams need to focus on eliminating bottlenecks in their pipeline to then facilitate agile development processes.

    Your CI/CD pipeline allows for the chance to administer efficient automated testing into your process, but shouldn’t stop at continuous delivery and continuous integration. Another step in doing all things continuous is continuous improvement. DevOps teams and administrators are constantly evolving to fit the needs of the pipeline, and it is imperative that teams spend some time every day discussing and hashing out the small things that can be improved upon. Over time, these changes—no matter how small—will have a profound effect on the enterprise and overall quality of morale between administrators and their teams.

    — Matt Geddes

  • DevOps Transformation: Leave No Ops Behind

    DevOps Transformation: Leave No Ops Behind

    In adopting DevOps, it’s easy to forget what you know

    Have you ever found yourself struggling to explain something that you know how to do without a moment’s thought? I’m not talking about the amygdala, the part of our brain that initiates the automatic “fight or flight” response. No, I’m referring more to the muscle memory-type behaviors that we’ve collected through life and, likely, pay little attention to while executing.

    Things We Take for Granted

    Muscle memory is awesome. Through the magic of muscle memory I can type fast (occasionally with okay-ish accuracy). It enables me to drive in peak-hour traffic without having a nervous breakdown, and it’s the only way I can ski down a tree-lined path without ending up in hospital.

    It’s through training, practice and repetition that we can carry out an increasing list of mundane tasks without being distracted from the bigger picture. In contrast, consider the expanding influence of DevOps across organizations, and how we’re pushing developer principles and practices onto operational facilities that have never been exposed to such. We’re asking people to break a habit of a lifetime and to think like a developer practically overnight. This can create some significant speed bumps in your operation transformation toward achieving continuous deployment pipelines.

    Type Mismatch Error

    On one side we’ve got infrastructure operations, where we’ve seen two decades of little change in the way services are provisioned and managed. Heck, for the most part, spreadsheets are still the source of truth—spreadsheets without versioning!

    There’s no “commit log” trail, no integrated testing and definitely no consistency across the vendor technologies that must be managed. Imagine what root cause analysis after an outage looks like? A team huddle for a round of “Who touched it last?”

    To software engineers, it must seem like the Wild West but, it’s important to acknowledge that this problem is largely the fault of the technology vendors. Until the principles and practices of DevOps shone a very bright light on operational inefficiencies, infrastructure operations weren’t challenged to increase the speed of business. Hence, the technology vendors hadn’t been pressured to evolve.

    A New Hope (Yes, Like ‘Star Wars’)

    Most infrastructure vendors have responded to the DevOps calling by re-architecting products in support of programmability. Many of them have even done an alright job of it, adhering to RESTful standards and publishing API schemas. However, there’s one key element that’s missing, and will be the cause of further challenges if not addressed.

    To an experienced developer, operating with declarative interfaces, idempotency and continuous integration is practically muscle memory. But to infrastructure operations, it is completely foreign. Progressing with the kind of operational transformation required to achieve continuous deployment is possible but only though closing this operating model gap.

    “Give me a Swiss army knife, and MacGyver I do not become,” said I at DevOps Enterprise Summit 2017.

    As the lines between development and operations increasingly blur, be super-aware of the development principles and practices that are just a part of life for the software engineers. To close the operating model gaps, run workshops for your pals in operations. Teach them all the fancy words. Show them how to manage their config files and architectural notes in a code repository, so they can familiarize themselves with the tooling.

    Most importantly, leave no ops behind!

    — Nathan Pearce

  • Event Management: Let the Noise Wail Without Going Deaf

    Event Management: Let the Noise Wail Without Going Deaf

    In part 1 of the blog series “Putting the Ops in DevOps,” by James Moore, we discussed the awareness and anticipation of operational management and the best practices that can improve the app at every stage of its life cycle. In this blog, I would like to cover one of the event management discipline maturity and share some best practices.

    If you are an operations professional on a team that’s beginning to embrace DevOps methodologies, you probably are facing some significant challenges.

    You know the business is applying pressure to get to continuous integrations/delivery, where release cycles can be measured by value delivered in days or even hours rather than weeks or months. Your Dev teams have embraced agile development, and are producing (what should be) production-ready code in short iterations. Your combined orgs are working toward implementing an integrated delivery toolchain that can build, test and deploy a new release version at the touch of a button. Your Dev teams are leveraging cloud technologies and architectural patterns that enhance their agility, auto-scaling microservice architectures of unprecedented scale and modularity. They are producing, as a matter of best practice, highly instrumented code that provides a dense trail of data relating to the state of the applications deployed on runtimes that constantly report their own state. You may be responsible for physical and virtual compute, network and storage infrastructure that itself is monitored with tools that spew a continuous stream of state information.

    In other words, you find yourself faced with continuous change (both automatic and human driven) and a barrage of potential noise. If you are a seasoned operations professional, bitter experience may have lead you to see change as the enemy—as change tends to zero, then so does operational risk. Now, you may to have to deal with continuous change as an inevitable force for advancing your company’s business goals. How will you separate the signal of service status from the noise of service state? Doing this successfully will be the difference between knowing quickly that there is a problem with an application or service before your users notice and being told about it by your users. It may also be the difference between getting a good night sleep and unnecessarily responding to a page indicating a benign application state.

    Enter event and incident management. The discipline of event management has matured, and has continued to mature, over decades as a mechanism for determining the status of elements of a managed environment from the state. Best practice in event management consists of:

    • Filtering out events that are not likely to be service affecting. A trace message from an app log is not worth your attention, unless you need to view it as a part of a diagnostic procedure. A synthetic user transaction timeout may require immediate investigation.
    • Correlating events that are likely to be related, so you get one notification per true incident rather than, say, 20.
    • Enriching events with context, so if a single service instance fails in a redundant array of five instances, route the event to the operations console but don’t wake anyone up unless the service is affected.
    • Implementing X-in-Y policies. In a large and complex system, a single microservice-to-microservice HTTP timeout may not be a huge problem. But you may want to investigate if you start getting 20 per second.
    • Implementing runbooks for common failures. Collaborate with development on the definition of those runbooks. Tie those runbooks to incidents as they occur so that the first responder has a process they can reinstate a service or prevent an outage in the first place.
    • If possible, automating those runbooks. If the problem is a failed process, then restart it. A disk is full? Free up some space.
    • Leveraging analytics and machine learning to gain insights from the reams of data your tool collects. Can you learn from event history to suggest correlations or areas for improvement to ops efficiency?

    With rapid delivery of new features and capabilities, the Dev team should already be mindful of the operability of the software they produce. Shift awareness of the event management tools and logic to the left. Have development use the same management tools that Ops use in production, but in pre-production. And then learn from what they see in pre-prod to improve. This should be done in combination with systems failure testing. Perform the test and see how the operations tools responds. Ask yourself, “Would I have been able to fix this if it had happened in production? Would I even have known what to do?”

    As is obvious by now, successfully operating an application or service in a modern business requires unprecedented co-operation between development and operations teams. Event management as a discipline and the tools that support that discipline are as critical now as they have ever been, if not more so.

    Listen to my recent podcast where I spoke about the common operational challenges many DevOps teams are facing today and how the traditional IT Operations best practices could be leveraged for use in a DevOps methodology.

    Also, download this white paper, which includes best practices for DevOps transformation and improving event or incident management.

     

    About the Author / Dr. Kristian J. Stewart

    Kristian Stewart is Architect – Hybrid Cloud Event Management and Analytics at IBM. He currently leads architecture for IBM’s Netcool Event Management offering, and is part of the team providing as-a-service capabilities to IBM’s clients with Cloud Event Management. He has worked in Systems and Service Management for 18 years. He lives in England with his wife and two daughters, two cats, and five Raspberry Pis.

  • SMEs Can Benefit from DevOps, Too

    SMEs Can Benefit from DevOps, Too

    The DevOps approach has several clear-cut benefits with the value centered in several key business areas: improved market agility, increased quality and revised culture of cooperation and effectiveness in staff operations. By promoting and implementing a DevOps culture in your SME’s business, Dev and Ops teams align much closer together and can better share in-between knowledge. This lets you create products more fluidly and increase deployment rates. The time for evaluating the business idea, development, testing and moving the finished product into the operations department is reduced dramatically. Quality margins improve, and this feeds back into the ability of your teams to continually fine-tune the products you offer.

    Businesses see a point in adopting DevOps only if it is directly contributing to the company’s increased effectiveness, productivity and, of course, profitability. The only way that it is possible for small and medium enterprises (SMEs) is when their IT and DevOps teams are working in sync and streamlining the provision of tech solutions that solve the business needs. And only a successful and timely DevOps adoption inside your organization will help your company beat competitors and stay afloat as the development across the industry speeds up.

    In the last years, we’ve seen a tremendous application delivery shift in businesses. We’ve moved away from the old ways of having large-scale system implementations based on one single project and its continued maintenance. SMEs now perceive application delivery as a continuous evolution. DevOps plays a leading role, as this software engineering trend ensures that such a major shift is possible. It brings all of the different parts of the enterprise together: business, development and operations teams collaborate on a different, more in-depth level with a goal of automating processes. This helps eliminate the usual constraints and takes the time-to-market issues off the table.

    In the latest ECS-Digital research on adopting DevOps in businesses, 57 percent of SMEs have partially implemented DevOps practices in their daily routine. The key benefit of DevOps for them is improved inter-departmental collaboration. But for the 43 percent that haven’t yet adopted DevOps, the collaboration issues are among the last priorities on the list.

    Step-by-Step Guide to a High-Level DevOps Life Cycle

    According to IBM Cloud Director, the end goal of a relevant DevOps practice is to cut the development time and improve enterprise’s efficiency. A growing business has to start from the “business needs” of the company and go down the road to make it come to life. The goal of adopting DevOps in your company is to push your business needs into production at a quicker pace more reliably. The goal in any DevOps practice is to cut the time and increase the efficiency of processes. You need to move from the “business need” to the “it’s live!” stage.

    Business Needs are a DevOps Priority

    But does your business have to get involved? Of course, your SME leadership focuses on the “business needs” of your company. But when we look at organizations that have implemented proper DevOps practices, the internal structure of the company shifts to focus on delivering great business results across the whole life cycle of your product/services offering.

    DevOps also ensures that employees have the ability to correct their work direction during the whole lifetime of a project. Things and goals tend to change, competitors may roll out some features that may make some of your project’s parts trivial, and all of this makes the need to adjust your project accordingly. This is easily done when your business need is close to both the Development and Operations teams that are able to steer and drive this alignment of goals.

    Have a look at the infographic below that clearly shows how a DevOps delivery cycle looks like:

    The Clash Inside Organizations

    Measuring the impact of DevOps on the things that matter to business is vital. Most SMEs are experimenting with DevOps, and there are some within the company who want to prevent DevOps adoption. The culture within enterprises is shifting too drastically and dramatically, although the effects are mostly positive.

    Such a shift is impacting all operations, with software development impacted the most. SMEs have it easier to change and adapt compared to larger businesses, and this lets them move the workplace culture a step ahead of their larger counterparts.

    If we look at business, operations and developer teams from a historical point of view, it was never a goal to establish inter-connection between them. The teams started cooperating only when it was absolutely necessary to get the other team’s input. SMEs are still faced with such a separation, as most responsibilities are set in stone for each team and the individual employee. All business units are used to working in their closed-in niche, and there is a lack of will to leave the comfort zone and act outside the predefined roles. Such actions in the past could have been viewed negatively.

    While most businesses that want to implement DevOps have employees who want the business to prosper, it can be difficult to force people to act outside their duties. However, to reach the end goals of any business—namely, ensuring fast and quality product delivery—and solve issues on the go, people must interconnect their duties and reach outside their business unit. With proper DevOps application, such collaboration can be smooth and result in the improvement of cross-functions.

    DevOps Represents Change in Internal Business Structures

    Many SMEs are faced with internal structural issues that limit the company’s prospects: Engineers are not willing to be on-call, the Ops team has a negative attitude toward those who write code and executives are slow to embrace automation. Organizations must overcome the culture war to be able to approach the agility and productivity gained by a DevOps model. The faster they can get there, the faster these organizations can take the competitive edge away from traditional enterprises.

    DevOps culture looks very different from organization to organization. However, regardless of the business implementing a DevOps approach, the potential benefits are the same. From faster deployments to increased efficiency and cultural changes toward experimentation and away from rigid planning, a DevOps culture has the ability to transform an organization completely. Wholeheartedly adopting a DevOps culture has real implications on how well a business is positioned to take on the market. With many industries facing uncertainty and a host of upstarts disrupting the status quo, shifting to a DevOps culture can be an important weapon in repositioning your business.

    Empathy is the Core of DevOps Adoption

    One of the foundational ideas behind DevOps is mutual empathy. Having empathy from operations toward developers and vice versa, as well as empathy in other business-related and non-technical company units, remains vital. It’s not about patting your colleague on a shoulder. We are talking about the moral mandate of a developer who just creates their part of the software and doesn’t care about how it can be deployed in production. The goal is to push employees to think outside of their small task, and to take into account the outcome of their work: Is the developed app secure? How hard is it to deploy it? Is it easy to keep it running? etc. If those questions are not accounted for, the developer’s colleagues on the Ops side will pay the price if something goes wrong past the release.

    By ensuring that the SME has empathy for each other at its core, businesses will deliver better products/services. And doing just that is what DevOps aims to bring to companies. Until this idea is ingrained, DevOps will face a cultural war of sorts.

    Your Most Valuable DevOps Resource

    By considering to adopt the DevOps approach to running your SME, you can better rethink the ways your organization operates—how it deploys its teams across the enterprise and how often it addresses structural problems that appear over time. Now you can lay out a new foundation for your SME based on the culture of cooperation and experimentation. With proper guidance in the application of DevOps culture, you can address the waste of time, human resources and money in your business.

    Becoming a DevOps organization is no easy task and taking up on such a challenge requires faith that changing the existing processes will bring its benefits. Reworking and optimizing organizations is the new norm, regardless of their size, industry, and goals. The faith in DevOps culture is driven by the firm and quantifiable statistics, but the intangible benefits to corporate culture are just as compelling. Many businesses are looking at implementing DevOps, and even more, SMEs are in the active process of its application. When your company is delivering quality ahead of time, there is a strong chance your competitors will be left behind as you are getting the bigger piece of the pie.

    Key Takeaways

    • To stay current, businesses need to adapt to an ever-changing environment, respond to competitors and stay on top of the innovation cycle, increasing customer value via responsiveness to change.
    • Increased service quality depends on both the availability of the service and being able to restore the service to an error-free state. Thanks to fast feedback loops and high release velocity, service breakdowns can be removed much more quickly than in the past.
    • DevOps are built on the sound principle of waste reduction such wait times and process overhead due to continuous improvement. This helps to improve agility and has a positive impact on costs.
    • Fostering continuous learning and improving culture through efficient principles is what DevOps is all about. In a fast-paced, unpredictable environment, non-stop improvement is paramount for sustaining your competitive advantage. It is also important for attracting exceptional talent and keeping employee satisfaction at a high level.

    About the Author / Stephan Pushkarev

    Stepan Pushkarev is the Head of DevOps Practice at Squadex.com and CTO at Hydrosphere.io. Co-founded and managed engineering teams for eCommerce, IoT and Ad-tech companies. He is responsible for the full products stack: math models, infrastructure & operations, enterprise applications as well as recruiting, establishing engineering culture and delivery processes. Stepan combines strong technical and managerial skills, and entrepreneurial spirit. Connect with him on LinkedIn.

  • DevOps: Time for Developers to Dig In

    DevOps: Time for Developers to Dig In

     Speaking to developers and project managers at the Gartner Catalyst Show, Jenkins World and Atlassian Summit revealed some insights about how most IT specialist perceive DevOps. Yes, I had a busy end of summer schedule attending events and talking to a wide range of developers and project managers about DevOps.

    two dogs digging together in one hole in the ground

    What I learned is that most people have not yet figured out what is meant by DevOps. What most people told me is that they are doing DevOps explaining they have a CI/CD pipeline. When questioned further about how they perceived DevOps, they believed DevOps was about creating a CI workflow that developers and sometime-testing teams could execute to automate builds, deploys and testing. One of the most ironic of these conversations was project managers making the claim that their CI server performed builds and deploys.

    Well folks, DevOps is not about a CI/CD pipeline. And contrary to popular beliefs, your CI server does not perform builds and deploys any more than it performs automated testing. Your CI/CD pipeline orchestrates build and deploy processes in the same way that it executes an external tool for test automation. The ugly truth is that your CI/CD pipeline is executing one-off build and deploy scripts, the same scripts used in your waterfall practice, but just triggered by your CI/CD process. These scripts cannot easily adapt across the continuous delivery pipeline. Testing and production environments are configured differently. The build and deploy scripts must be tweaked, maintained and debugged in the same way they were during waterfall practices. Once ready, they are then called by your CI/CD workflows. DevOps is different.

    DevOps demands that your developers and operation teams share and collaborate from the earliest phases of a project. In essence, this requires that more of the operational duties be shifted left and directly onto the shoulders of developers. This is precisely were the responsibility and control should be. Operation teams need knowledge and transparency so they can communicate infrastructure requirements early and often to the developers. Eventually what we will see is a decline in the number of operational resources. A “release engineer” who managed production deployments said, “If we define our DevOps process correctly, my job will not exist in the future. Instead, automation will replace the manual Ops scripts, allowing Dev teams to do 70 percent of my work.” And that, my friend, is real DevOps.

    Developers, Dig In

    So it is time for developers to really dig in to DevOps. What I mean is developers must begin walking away from automating most of the processes by a one-off script and begin incorporating tools that perform DevOps tasks from continuous deployment, security scanning and performance monitoring. And different platforms need different tools. Project teams will need the power to select the tools that work for them. The days of “legacy” purchases of a single tool for the enterprise is long gone. High-dollar operational tools are high-risk and incur technical debt when developers choose to use a different solution. DevOps means giving development teams more control and purchasing authority for automation tools that can be managed at a project level, not an enterprise level. These tools must support a process where the developers work on the declarations followed by testing and production consuming the process with a good amount of transparency. Scripts cannot achieve this, even when triggered by CI.

    Our future is becoming more and more complex. Most companies I spoke to still managed over 70 percent of monolithic applications. The writing on the wall is in the 30 percent. With microservices and containers, operational tasks are going to become more complex, not less. The sooner developers can become the core owners of operational tasks, the easier it will be to adopt a microservices architecture. Managing microservices will not be the job for the operations team. It will need to be done by the developers who have the intimate knowledge of their application.

    To mature from CI/CD to DevOps is going to require a close look at the engine under your CI/CD hood. When developers and project managers begin to really dig in and see how the engine is running, it will be clear that achieving DevOps is a bit down the road. And there is urgency in this process. Leaner and better ways of writing and releasing code is upon us. Most of us are just not ready to toss out some of the old habits from waterfall practices to get there.

    Some departing thoughts:

    • Acknowledge that CI/CD is a good place to be, it is just not DevOps.
    • Look carefully at your underlying processes that are called by CI/CD. How well do they support your production environment?
    • Can your CI/CD workflow at Dev support a production deploy? If not take the time to understand why.
    • Are your developers working late to get a code update to production?
    • Do you have a clear continuous feedback loop that shows where a code update is running, and on what exact endpoints?
    • Do your developers know what infrastructure updates are going on in production today?
    • How do your developers communicate infrastructure updates to production teams? Can they do it alone?

    The time is now for self-examination and a clear honest appraisal of just how automated your core operational tasks are. If they are built upon one-off scripts, you should assume you have some work ahead to really begin moving to a DevOps practice that can be declared by developers, consumed by all runtime environments and is transparent to all.

    And remember, microservices are chasing you.

    About the Author / Tracy Ragan

    Tracy Ragan is COO and Co-Founder, DeployHub. She has extensive experience in the development and implementation of business applications. It was during her consulting experiences that Ms. Ragan recognized the lack of build and release management procedures for the distributed platform that had long been considered standard on the mainframe and UNIX.  In the four years leading to the creation of OpenMake Software she worked with development teams in implementing a team-centric standardized build to release process. Connect with her on LinkedIn.

  • Cloud Capabilities Turn Developers into Operational Experts

    Cloud Capabilities Turn Developers into Operational Experts

    The upside-down world of digital transformation can leave you feeling like you’ve fallen down an internet rabbit hole. If you’re a DevOps denizen, the swirling pace and scale of virtualization might give you that “lost in the Technicolor forest” feeling when it’s time to release an app into the wild. “Curiouser and curiouser!” you say as you try to sort through hybrid combinations of private and public clouds, CDNs, and data centers.

    As a developer, you worked hard on the design, build and test—now you want to make sure it takes the best, fastest path to microservices and users. It’s folly to trust the answer from any one source—synthetic measurements may purr like a Cheshire Cat over hardware health, meanwhile the White Rabbit of RUM (real user measurements) is spazzing out over latency and traffic jams. The lesson Alice finally learns in Lewis’ timeless tale? She must learn to control the journey herself.

    Developing for the Complex Cloud

    Fortunately, there’s a way for developers to mix that “just right” solution that will reveal the right path through the internet forest. This used to be the purview of operations—setting up servers and delivery conduits. In the pre-cloud era, developers just wrote code. Somebody in Ops pushed it live, and that part of the process was opaque to the developer. With the proliferation and maturation of the cloud, the Ops team is increasingly focused on dynamic infrastructure and services. Developers routinely spin up their own servers. As DevOps teams focus more on microservices and containers and less on “stacking and racking” servers, performance, control and delivery become the focus areas that matter.

    As the division between Dev and Ops experts continues to dissolve, each becomes increasingly aware of the other’s roles, responsibilities and skill sets. The goals of agile development, non-stop testing and optimized delivery become the shared destination; paths merge until everyone is creating momentum around the prized feedback loop. As infrastructure and services become more abstracted, the focus is less on writing original code and building stacks and more on selecting resources. The selection and use of these resources has to be strategically guided by real-time intelligence to meet multiple performance and business objectives. The more abstract and hybridized the infrastructure becomes—and it is moving rapidly in that direction for every business—the harder it is to navigate. If you can’t keep all your resources straight, you can’t optimize them. Centralizing and automating tracking and traffic management is mission-critical.

    The Delivery is in the Code

    Which brings us back to the developer’s mandate to control and optimize delivery from the application. Cloud-based delivery tools combine real-time user measurement with scriptable application traffic shaping and built-in cost controls. Gone are the days of blaming poor performance on the network. End users expect seamless, instantaneous experiences, no matter what. This is a tall order for modern apps that combine video, microservices and dynamic content. To ensure high quality of experience (QoE), you have to automate and optimize with software-defined delivery. Developers can now contribute to optimization by coding in delivery intelligence.

    This isn’t merely another task for the developer; it’s an opportunity. Developers are now empowered to deliver the functionality of their product or service exactly as designed. Developers are now actively part of the process that brings together global and local traffic management data (RUM and infrastructure health), application performance management and cost considerations to ensure the cheapest, fastest path to that microservice. The balance and mix of these factors can be customized and fine-tuned by the developer (a.k.a., the newly minted operations expert).

    The Bleeding Edge Becomes the Bedrock

    You know the cloud is no fairy tale when 74 percent of tech CFOs claim that this year the most measurable impact on their business will come directly from cloud computing. Spending on cloud computing is set to grow at 19 percent CAGR through 2020, six times the rate of overall IT spending. Developers who can harness the magic of hybrid cloud/CDN for video, web content and application delivery will be rock stars. Picture it: fully automated delivery based on business rules, actionable data from real user and synthetic testing, and self-healing network optimizations. Developers can push their capabilities and control right to the bleeding edge, making them an even more agile contributor to the continuous improvement cycle.

    Soon enough, as technology progresses and infrastructure is increasingly software-defined, these skills will become a requirement. Over the next few years, this data-driven approach to app and content delivery will be considered a standard capability for digital businesses and a core responsibility for developers. The forward momentum of organizations that have committed to both cloud-first and DevOps-first strategies is powerful and will quickly boost early adopting competitors out of reach. The number of developer-operators capable of managing application delivery in capacity-aware, consumer-responsive, application-tuned ways will soon build to critical mass. Don’t get left behind, wandering lost in the woods.

    About the Author / Andrew Marshall

    Andrew Marshall is the Director of Product Marketing for Cedexis, and has driven product marketing efforts at New Relic, Amazon Web Services, and Microsoft. Connect with him on LinkedIn.

  • Kanban for Operations and Traffic Court

    Kanban for Operations and Traffic Court

    Howard stepped off the elevator and into the lobby. His eyes were puffy and felt like someone had scrubbed them with sand paper. Looking around the office, he made a beeline for the coffee machine. It was one of those fancy jobs where you use a touch screen to select all the different flavors, textures and aromas. Howard thought they all came out tasting like boiled socks. As the last few drops of black sludge dripped into the mug, Lisa peeked her head around the side of the machine.

    “Getting ready for stand up?” Lisa asked, eyeing Howard’s coffee.

    “Getting ready for life.” Howard grumbled as he headed towards the stairs. “You coming?”

    “To stand up, yes. To life, not until I have some of this delicious coffee-flavored oatmeal. See you upstairs.” Lisa said, putting a cup into the machine and starting to make her first of 30 mandatory selections.

    Howard strolled into the Operations area, starting to feel a little more human with some coffee in him. Looking around, he spotted Paul hunched over his keyboard, typing furiously. His desk was in its standard form of disarray. Empty Diet Coke cans covered the desk, with a few crumpled in the middle like fallen soldiers on a battlefield, the other cans standing around the crumpled ones in mock solemnity. The keyboard was one of those mechanical DAS keyboards with the blue keys, so it made clicking noises as Paul’s fingers danced across them.

    “Good morning Paul.” Howard said as he lumbered to a stop beside Paul. Paul raised one hand in a warding off gesture.

    “Can’t have any more interruptions or I’ll never get this done. See you in five minutes at the stand up.” Paul said without looking up. With that, his hand flew back to the keyboard to join the other, like a juggler taking in another bowling pin and seamlessly integrating it with the ones he is already juggling.

    Howard nodded and shuffled off to large Kanban board on the wall of the Operations bay. It was a large screen projecting the digital board onto the wall via a small Raspberry Pi. Howard was quite proud of this board. It had taken some time to convince the team that they could use Kanban in such an interrupt-driven world as Product Operations. The team went through several iterations before coming to the current board on the wall and even now they were willing to make changes.

    All new tickets for the team funneled into the backlog column on the left side of the board. These were automatically marked as untriaged. The team would review these tickets in the morning stand up and then prioritize them as a team. This excluded emergency tickets which were closely inspected. Emergency tickets would enter directly into the Expedited Lane of the board, and the team would swarm them until completed. Those tickets were almost exclusively the domain of outages and customer impacting events. New tickets filtered in through either email or the BOT in the Operations Slack room.

    The team began gathering around the board to review and prioritize the tickets in the backlog. Howard noticed that there were a few tickets in the active column that he didn’t recognize. He made a mental note to ask the team about this after they “walked the board.” The team took turns reading the new tickets in the backlog, and they discussed the work effort, blockers, dependencies and perceived priority of each. The tickets were then rearranged to be in order of priority with highest on top. Next they worked from left to right, with each person explaining the tickets they were responsible for.

    “I noticed that there are a couple tickets in here that were not there yesterday and were not covered in the triage. How did that happen?” Howard asked before everyone drifted back to their desks.

    Chris Fallon, known simply as Snark to everyone else, spoke up. “Those are the ‘important people’ tickets, boss. They are the ones we get from the constant ‘drive-bys’. You know, when an exec or a sales person steps into your cube and tells you that this is going to be the next big thing that makes millions.”

    “Wait a minute.” Howard said, turning to look at Snark. “I thought we all agreed that we would only take tickets in via the backlog, unless it’s an emergency.”

    Snark laughed, “Yeah, well, they seem to think their poor planning and ineptitude is somehow synonymous with being my emergency.”

    Howard took a deep breath and said, “OK, we can’t do this or it undermines the process. When this happens, it’s like the person standing over your desk is saying their time is more valuable than yours. That is simply not true. Let’s take the guidelines we all agreed upon for expedited tickets and print them out. We will put them up on the wall and the next time someone comes up with a ‘favor to ask’ point them to the rules. If they still don’t let up on you, point them to me. We have to be fair, though.”

    Howard thought for a long moment and said, “OK, here is what we will do. We will hold daily traffic court for anyone that puts in a ticket. This is completely optional, of course, but if the person feels like it is an emergency, and yet doesn’t meet the qualifications, they can come to the stand up the next morning and plead their case. If it seems like the need is real, the team can agree to put that item above all the other items to be next pulled into the Kanban process. This doesn’t mean we swarm it like an outage, but they do get to the top of the pile. Thoughts?”

    Everyone silent for a moment, and then the agreed. Everyone but Snark, who continued to look up at the ceiling, apparently deep in thought. At last he lowered his eyes to the group and said, “This will work, IF you get the right people on board first AND you support all of us in sticking to the rules.”

    Howard nodded and said, “I already have a mental list of the people we will need on board. I’ll get that taken care of today. Can you write this up and get it up next to the Kanban board?”

    “I’m on it.” Snark said, and turned with the rest of the team to leave.

    The concept of the traffic court ended up paying dividends in ways Howard hadn’t counted on. It turned out that, much like regular traffic tickets, people didn’t want to invest their time to come in and argue the case with the court. Instead, they decided to take the easier route of just paying the fine. This was their opportunity cost; time over money was the same as time over prioritization. There were cases where people would show up and win or lose their case. In the end, Howard and the Operations team began to understand who was just wanting to jump line and who had a genuine need. It was no surprise that each type tended to be repeat offenders. Over time, the ones exhibiting negative recidivism, began to learn that their behaviors were time-consuming and not being validated. Their behavior didn’t change but how they actively interacted with the team did and thus the outcome was still the one Howard’s team wanted.

    — Pauly Comtois