devopsy.kinsta.cloud

Tag: value stream mapping

It Is Time for DevOps to Focus on Ops

The advent of both Agile development and DevOps were great steps in improving an IT process that had become ossified, with policies that alternated between resistance to change and trying to do things better/faster. Between them, Agile and DevOps broke the cycle of resistance … but only for development.

The confusion around CI/CD is a great example. CD is both continuous delivery and continuous deployment. Do you know why? Because up to that point (and even recently), DevOps included Ops in the name but did little with or for operations. I have no doubt that it was originally intended to include everything from build testing to full-on deployment, and we are finally getting there.

You have no doubt noticed that both operations and security are finally getting DevOps’ attention at critical points. I have been chatting with coworkers about value stream mapping and, if you map the value stream of a software delivery process, you will find that all of the weak points have shifted right. Thankfully, we are also paying attention to them these days.

From GitOps to infrastructure-as-code (IaC); from cloud-first to policy-as-code (PaC), we are finally approaching the operations side with some level of seriousness. And it is generating results for those involved.

My two areas of attention these days are DevOps and application security (AppSec). As I’ve written elsewhere, both are benefitting massively from things like policy-as-code. The ability to apply corporate deployment policies, security policies and compliance policies from a centralized location is massive. It is early days, but community contributions to PaC libraries are already massive, making it easier to deploy without recreating that big old list of generally useful policies.

Given a project, a set of tools and a timeline, you could fully automate the process from the first line of code to running in production, with stops for approval and the specialized steps that still require human interaction (no matter how many public policies are out there, someone with knowledge of your network still has to set up IPs or ranges, for example. Or someone still needs to write the code). Someday, it may be possible to say, “We need a phone app that targets both major platforms and is hosted on Linux that does X, Y, Z,” and it just happens … but we are not there yet, and, IMO, the domain knowledge will be a roadblock for a while. But you can take advantage of what we have to work smarter, not harder. And the organization will be better off for your efforts.

So keep at it, start to evaluate your processes and consider placing any of the great new DevOps technologies into your path. Let them do some of the grunt work so you can focus on adding value. And for all those coworkers that still don’t get it, thank you. You are keeping them in a job, whether they admit it or not.

June 14, 2023
Improving Security with Mapping, Automation and DevOps
Strong cybersecurity is paramount for every organization. But all too often, security teams are disconnected from ongoing development processes and viewed either as an impediment to bringing products to market or as an afterthought. This leads to increased costs and possible gaps in the attack surfaces of new platforms.

While many organizations are turning to security automation to help them manage and mitigate their expanding threat landscape, quick fixes rarely work. Planning is required to help organizations determine their pain points, add security early in the software development process, and determine where automation can deliver the best ROI.

What Should Be Automated?

Security automation can programmatically test, institute guardrails, provide feedback and remediate cyberthreats by executing tasks as part of the continuous improvement pipeline. This, in turn, can help understaffed security teams by streamlining complicated and time-consuming testing processes, enabling teams to maximize their time, be more productive and focus on risk priorities. Once the manual process is removed, the automated workflows can then carry out a variety of more minute actions using intelligence and logical decision-making.

Security-focused areas that may benefit from automation technology include but are not limited to:
- Static and dynamic code analysis used to discover security vulnerabilities
- Test harnesses for software configured to run a program unit under varying conditions to discover and monitor its behavior and outputs
- Upstream code analysis that examines code such as complete code blocks or patches
- Vulnerability testing, management and remediation to discover and fix weaknesses that may be susceptible to attacks
- Penetration testing used to simulate cyberattacks on an organization’s computer systems and evaluate security posture
- Automated enforcement based on data classification and tagging strategy
- Deployment and management of infrastructure components
Focus on ROI

Despite the benefits, many organizations face substantial hurdles in developing, implementing, testing and optimizing the comprehensive, integrative, long-term approach needed to use security automation to cope with the evolving threat landscape. It’s all too common for organizations to begin projects that start with automation as the outcome rather than rigorously mapping the processes and determining where value from automation can be gained.

Consider an organization that decides to automate the process of deploying a firewall to an AWS cloud. A team spends several weeks developing and testing an automation playbook that can spin up virtual firewalls, license them, connect them to a central manager, deploy configurations and apply them and send logs to the appropriate places. Testing proves the automation works.

But when asked how many firewalls they expect to spin up this year, the answer is two–maybe. How many next year? None. The team, therefore, spent more time automating this function than it would have taken to just deploy the systems and the organization won’t likely realize any direct return on this work for at least two years.

Value Stream Mapping

The first step in achieving real value is to understand that automation is a second-order operation that relies on a thorough assessment and plan of all the steps required to achieve real technical and business outcomes. That means conducting value stream mapping to analyze, design and manage workflows that best suit your environment.

Mapping will help you analyze and define security processes. Without analyzing your processes, you’ll have no idea of where the bottlenecks are or how they can be removed. Better understanding the processes will also allow you to refine them as needed. Without doing all this, you’ll have no way to prove the value of the automation you’re implementing, and no compelling business justification for it.

The Role of DevOps

Few organizations are able to effectively realize the benefits of automation right out of the gate. Value stream mapping enables organizations to determine their pain points, pinpoint exactly where they want to use automation and identify processes that cannot or should not be automated. Clear-eyed planning and assessment will help teams define a process and determine the value of return on automating certain processes.

In addition, for security to be effective, we have to shift security left in the software development life cycle (SDLC). Security practitioners should be engaged in the development pipeline, including collaborating on automation initiatives and building harness and test cases as the work progresses. This will save time, improve quality and ultimately add value, ensuring the success of security automation projects.
August 4, 2021
A Daily Gemba Walk for a Software Delivery Organization?
Lean thinking has had a substantial influence on helping enterprises to improve the flow of value across their software delivery organization. Yet, for many companies in what industry visionary Brian Solis calls the “Novel Economy,” where market conditions have yet to be identified or understood, the degree of responsiveness and agility required to adapt and remain competitive is still not sufficient.

By measuring how value flows through a product’s value stream, however, everyone from practitioners to business leaders can initiate “daily Gemba walks” to obtain real-time continuous insight into the software delivery value streams that serve internal and external customers. You can see what’s slowing you down today and where to invest resources to continuously improve where it matters to support both the customer and employee experience.

Unique Challenges in Software Delivery Value Streams

A software delivery value stream is the sequence of activities an organization undertakes to respond to a customer need that results in the delivery of value. Yet, they are not so easy to identify and measure because much of the work is knowledge-sharing. While Lean practices such as value stream mapping have helped us to identify the current state of processes, learn to see where bottlenecks are and learn to improve with experiments to reach a desired future state, there are significant challenges.

Logistics such as scheduling time, people and the right kind of meeting room, as well as extracting objective data for improvement strategies, make it difficult to keep a regular cadence of assessing the value stream. And, within a relatively short amount of time, the information we captured goes stale, because priorities and strategic initiatives are dynamic and ever-changing.

Furthermore, since the software delivery work from ideation to operation is mostly cognitive and often complex, it is difficult to accurately map the flow of value. The process flow is not linear—it’s more like a network. The complexities of the interrelated components and variability of how teams work are difficult to capture in an understandable manner. As a result, we lack complete and objective visibility into the end-to-end flow of work to make time-critical decisions to improve performance.

Going to the Gemba

There’s a concept in Lean thinking of learning by going to the place where the actual work happens—“the Gemba”—to observe the work first hand. However, in software delivery at scale, even if we take a Gemba walk through the floors of our buildings where members of the value stream are working, we won’t gain much understanding of what’s going on. We might see kanban boards, huddles and charts, but what we mostly see are people working on their laptops at their multi-screened-in desks. We only really see a small slice of the set of activities that create value.

So how do we see all the activities across the whole value stream at a continuous cadence to help us make both reactive and proactive decisions to improve flow in the most pressing areas? One approach is Flow Metrics, as described in Mik Kersten’s book, “Project to Product,” via the Flow Framework.

A lean and prescriptive approach that guides organizations on what and how to measure in software delivery at scale, the framework uses Flow Metrics to provide business and IT leaders with shared, real-time insights into the relationship between technology delivery and business outcomes. It helps you to establish your baseline to see where you are today and pinpoint where you need to focus your energy.

These value stream metrics track the flow of business value by correlating the investment in accelerating flow for each product’s value stream with the desired business outcome. We can use these metrics to scale what we used to do in our value stream mapping sessions to a daily Gemba walk based on objective data without interrupting the teams involved.

We can see first hand the velocity at which our teams are producing value. We see the efficiency and distribution of their work. We see the work-in-progress (WIP) and the lead time for value creation. We see the correlation between the product value stream contributors’ hard work and the resulting business results (such as a new feature for revenue; fixed defects for product quality/customer; paying down tech debt to improve delivery speed, security and risk work to protect against breaches, and so on).

Business-level Metrics That Count

The Flow Metrics are:
- Flow Velocity gauges whether value delivery is accelerating. It is the number of flow items of each type completed over a particular period (also known as throughput).
- Flow Time can identify when the time to value is getting longer. It measures the time it takes for flow items to go from work start to work complete, including both active and wait times.
- Flow Efficiency can identify when waste is increasing or decreasing in your processes. It is the ratio of active time versus wait time out of the total Flow time.
- Flow Load monitors over- and underutilization of value streams, which can lead to reduced productivity. It measures the number of flow items currently in progress (active or waiting) within a value stream.
Alongside the above set of Flow Metrics, the measurement of Flow Distribution helps prioritize specific types of work during specific time frames to meet the desired business outcome. Flow Distribution measures the ratio of the four flow items (feature, defects, debt and risk) completed over a window of time.

Flow Metrics ideally should be displayed in a single dashboard, providing IT and business leaders with a complete view of value delivery and an understanding of how the metrics are affecting each other. Here’s an example:

Flow Metrics are opening a real-time window to see the value stream activity that we previously could not see as frequently and as objectively. Consider taking a daily Gemba walk for software delivery to ensure you’re meeting the most pressing needs of your customers, internal and external, to navigate through the opaqueness of the Novel Economy.
July 24, 2020
Value Stream Optimization: Putting Business in the Driver’s Seat
How value stream can be a major driver of DevOps success in today’s software-only world

Even as DevOps reaches a pinnacle in acceptance (as of 2018, only three percent of firms had not adopted DevOps and had no plans to do so), many organizations are still struggling to get high-quality software out the door. DevOps has been promoted by many experts and consultants as a solution to the cumbersome inefficiencies that were common with waterfall methods. However, as is common for many operational activities, applying a new approach or technology doesn’t automatically resolve entrenched problems.

Nowhere is this more obvious than in value stream management (VSM). This approach, which focuses on removing the waste from processes to increase their efficiency, sounds like a panacea for resolving software delivery challenges. In practice, it often falls short of its goals.

Why is this? There is nothing wrong with the concept of the value stream. The idea of breaking down a complex process into its components and optimizing each one of them, thereby optimizing the chain, is a long-recognized pillar of lean production. In my view, the problem isn’t with VSM or even DevOps.

The problem is that in moving from waterfall to DevOps and/or agile, many organizations figuratively “throw the baby out with the bathwater.” They are uncertain how to adjust for the extreme acceleration that occurs when teams move from rudimentary software processes to modern ones, so they either jettison important elements of software engineering or adapt them incompletely.

Why Continuity Matters

By way of example, consider measurements and metrics. The traceability and proactive remediation enabled by measurements and metrics are two of the most important aspects of software development. However, it’s not just the development and test metrics that matter, but also the business metrics—metrics of quality, user acceptance, security and ease of use. Since the dawn of software development, those two groups of data have been inextricably linked, but some modern software teams appear to have forgotten that lesson.

No amount of process improvement metrics can ensure a software release is secure or that users will like it. Yet, if a major software upgrade is flubbed, it can impact operations so negatively that sales momentum stalls. (If the company sells the software it develops, the ripple effect is even stronger.) Even more disconcerting, after a major data breach, a firm can find itself out of business.

Don’t get me wrong. I am not suggesting that flow and other DevOps metrics don’t matter, because they do—very much. I am suggesting that many teams and their development managers have become too divorced from the business metrics, which can have staggering negative consequences.

From There to Here

In the days of waterfall, development artifacts, defects and other data were warehoused in an offsite location, and after each release (which might take days, weeks or even months), teams would engage in several days of data extraction. Then, a separate team would attempt to map the development artifacts to the defects and enable traceability. After hours or days of organization and cross-mapping in Excel spreadsheets, reports would be generated for management and the C-suite. Once stakeholders and management completed their evaluation, the team would be cleared to make adjustments before the next release cycle.

Now, our development paradigm has changed so dramatically that many teams aren’t giving metrics their due. That’s an issue with defect metrics that could impact the user experience or software reliability. It’s a serious, potentially game-ending problem when the metrics are related to security.

Completing the Value Stream

To create and administer the complete value stream, software teams must be supported in three areas beyond the traditional DevOps framework where iterative development activities are the primary driver of traceability measurements. With these elements in place, management can map their software teams’ efforts to the business metrics that are the final measurement of success:
- Organizations must have a mechanism for integrating development and testing tools so that defect measurements can flow back to the development team in real-time—matching the speed of DevOps.
- Critical software activities that still take place outside of development and testing processes, such as generation and approval of change orders, must also be monitored for inertia, with approvers being held accountable for holding up paperwork that delays releases.
- Security defects must be given parity with, and preferably priority over, all other code defects. All defects that impact the user side, such as functionality and launch speed, are important business metrics. But security must be paramount.
If you are a development team member reading this, or even a development manager, you may think business metrics and the related activities that happen outside your purview, don’t apply to you. I urge you to think again. Software has become a critical pillar of many companies’ operational success, and that connection is only growing stronger by the day. Everyone involved in software development, from the developer to the VP of Operations and on to the CEO, must view the entire process through the lens of business.

— Joe Schulz
June 3, 2019
Scripts in the Development Pipeline: Best Practices for Avoiding the ‘Scriptocalypse’

The use of scripts as a key tool in the average developer’s bag of tricks has altered the development pipeline in significant ways. It has even led some organizations to outsource their entire team to manage the glut of scripts that have been developed over the course of time.

To help companies move beyond the “scriptocalypse,” we recently participated in a webinar with our friends at Tricentis. Sunil Mavadia, global head of customer journey at Electric Cloud, and Thomas Stocker, director of product management at Tricentis, walked through best practices to follow to avoid the scriptocalypse. Here are some takeaways from their discussion.

Improve Process Robustness and Scalability

Of course, everyone wants to improve their process robustness and scalability, but how? One of the most effective ways of doing this is to employ a tool that helps orchestrate your releases and provides reusable objects. This reduces risk because you no longer have to write scripts for every single application, which creates a cascading effect—not only do you write fewer scripts, but also you don’t have to maintain these scripts across the board. As a result, you are assuming much less risk across many levels of your technology and pipeline. Additionally, system-level visibility across all of your processes will increase—which has many benefits for your entire development life cycle.

Don’t Just Automate, Orchestrate

Many developers make the mistake of thinking that because they have scripts, they’re automated. But to be truly automated, every action needs to be automated and driven by your pipeline. Your approvals, your handoffs … everything. Oftentimes throughout the release process, team members find themselves having to stop in the middle of their tasks to ask for approval. This only creates a process that interrupts the entire pipeline. But what if you could automate all of that?

A great quote from the “2018 Accelerate State of DevOps Report” that speaks to this says, “High performers automate significantly more of their configuration management, testing, deployments and change processes than other teams.” If you look at it purely from a team perspective, the high performers are those that have fully automated all their processes. While this can certainly sound like a daunting task, it’s best to look at it with an agile mindset—this is an iterative process. You have to start somewhere, while keeping the goal of 100 percent automation in view.

Not only should you be automating, you also should orchestrate your pipeline. This is critical. By automating and creating reusable objects, you can get to the point of using your pipeline as a service. Imagine reusing this particular pipeline for every application within your process, so you could have hundreds, even thousands, all using this pipeline and all these objects underneath that are all reusable objects. It’s a pretty picture, isn’t it?

Create a Value Stream Map

When you create a value stream map for a pipeline, you want to be very inclusive about who you invite in the room to create the value stream. Include representatives from release management, development, deployment engineers, team leads, executives and infrastructure engineers. Basically, anyone who has “skin in the game” for your release should be present.

Once you feel you have a solid representation of who is impacted by your releases, sit down and draw out the entire process. This includes figuring out what each script does. While that may be a big task depending on how your pipeline is currently structured, there’s actually a lot of opportunity there. For example, you might find there is one application for which you use one script to do that deployment. Typically, that script is duplicated for another application and so on. Imagine doing that 100 times. You’ll have 100 scripts or variations of that one script for your 100 applications. That’s an obvious and beneficial place to deploy a bit of automation.

This is also a great time to understand what tools you are using for each step. When you’re jumping from toolset to toolset, it can be difficult to get a holistic view of how this is impacting your overall performance without a complete picture of the value stream. When you have that picture, you can start to ask yourself questions such as, “What are all these tools that we interact with that we can integrate into the pipeline? How are we handling approvals?” By including detailed sequencing timelines and identifying bottlenecks, you can realize some impactful efficiencies.

Finally, creating a value stream map gives you an opportunity to organize your scripts. Laying out the entire value stream can give you a picture of the variety and quantity of scripts you’re running—whether they are CI scripts, deployment scripts or test automation scripts—and bucket them in a logical way.

Identify Redundancy Dependencies and Bottlenecks

Of course, identifying (and hopefully rectifying) dependencies and bottlenecks is a good best practice for any aspect of your development pipeline. But it becomes even more important in the context of using scripts. If you have one application dependent on another application, it’s almost impossible to script. For example, imagine you’re trying to do some testing with two dependent apps. If you don’t have the right versions across both, you can’t test one because the other one is not up to speed or it hasn’t been updated in an environment. Something even as simple as that can be really disruptive and make scripting very difficult. By understanding where your dependencies and redundancies exist, you can take the first step toward solving the issues they present.

Scripts are one of the most effective ways of streamlining your pipeline, but they need to be used in a way that will allow your organization’s applications to be scalable and sustainable. By following some of the best practices outlined above, your organization can use scripts like pros—and avoid the dreaded “scriptocalypse.”

For more insights watch the full webinar here.

— Anders Wallgren

October 11, 2018
DevOps Chat: Value Stream Mapping with CollabNet’s Eric Robertson

This DevOps Chat features Eric Robertson, VP of product and engineering at CollabNet. We discuss value stream mapping, which comes from Lean methodology and has been around for a while. But do you actually know what value stream mapping is? How can it help your DevOps transformation? Eric does a good job of explaining.

As usual, the streaming audio of our conversation is immediately below, followed by a transcript of our chat. Enjoy!

Streaming Audio

Transcript of our Chat

Alan Shimel: Hey, everyone. This is Alan Shimel of devopsy.kinsta.cloud here for another DevOps Chat and our guest on this chat is Eric Robertson, VP of product management and engineering at CollabNet. Eric, welcome to DevOps Chat.

Eric Robertson: Thank you. Thank you, Alan. Glad to be here.

Shimel: Yeah. We’re glad to have you here, Eric. I don’t think we’ve had anyone from CollabNet, actually, on DevOps Chat before, though I think we’ve done some video interviews over at DevOps Enterprise Summit—maybe it was San Francisco in October and November—but we’re—so we’re glad to have you here. We’re eager to find out about what’s going on with CollabNet and about some of the stuff you’ve been doing. Eric, if you don’t mind, though, there might be some people out in our audience who are not familiar with CollabNet, so, if you can, just—doesn’t have to be long, but maybe a quick elevator pitch on who CollabNet is, what you guys do.

Robertson: Sure. Sure. So CollabNet’s been around since, I think, 1999, and it was one of the foundational on the open source, with our Subversion version control software. We since then moved into the ALM, application lifecycle management, platform. It’s our product offering called “TeamForge,” where we started linking in that planning, development, and testing life cycle, with integration to tools. And now, with me, we actually extended that life cycle all the way down into the operations—thus, the concept DevOps. And now we are launching—have launched, actually, a series of products and partnerships around DevOps for our customers.

Shimel: Fantastic. Fantastic. So, Eric, one of the things that, when we spoke off mic earlier, we spoke about value stream mapping, and I know that’s something kinda, if not near and dear to your heart, something that you’ve been involved in with CollabNet. Why don’t you give our audience—let’s start off with this. There may be people out here who, just as they didn’t know CollabNet, maybe they’re not familiar with the term “value stream mapping.”

Robertson: Yeah. It’s very interesting. Value stream mapping is actually an older concept. It’s part of the Lean management framework. For example, many may have heard about Toyota. It was started there, as far as best practices around mapping material and information flow and how could we apply to, for example, the value chain, and pretty predominant in the manufacturing domain. And it’s kinda of interesting because I have a background in software engineering and I have degrees in computer science and engineering, but my first job was actually in architecture—not in software architecture, but actually designing building. CAD design.

And it was interesting that, during that time, they started talking about design patterns and best practices, as far as architecting and putting together systems and designing. And I said, “Wow, this is very similar concept to what I learned back in college, when I was doing software engineering, software design.” And, very soon, that started translating into object-oriented and design patterns and things of that nature. So it’s interesting how patterns and techniques that were very popular and used in other domains, like manufacturing and architecture, now being applied to software. In other words, software is maturing, right? The software development process is maturing. It’s starting to utilize these type of practices.

And the value stream map is just an example of that, right? It’s Lean management method for really analyzing that current state and designing a future state, based upon a series of events that take a product, a service, from its beginning through to its customer. Right? And, again, like I mentioned, Toyota—this is known as that material and information flow mapping, and it can be applied to any value chain, for example, right? And what is a value chain? It’s just a set of activities that take, for example—that delivers a valuable product or service to the market, right? So you got, in the manufacturing world, this was machines that did it; in the software world, it’s now tools that we use to deliver a product from a planning all the way to its operations stage.

Shimel: Yep. Got it. So, Eric, this is something, obviously, that you have written and you’re talking a lot about it. Can you tell our audience, if they want—maybe if they wanted to find out a little more, where can they go? Before we dive in more than that, maybe where can they get more information on this?

Robertson: Yes. Actually, they could come out to our website, collab.net, but, also, I recently published an article around how value stream mapping delivers a better DevOps tool chain, and we can post those links—I believe it was also on devopsy.kinsta.cloud and we have one that was delivered through TEDxBeacon, so there’s a series of outlets, and we’ll post those links there on where they can get more information about this.

Shimel: Excellent. You know what? We’ll, for sure, do that. And then, Eric, you had also mentioned that you guys were doing some sort of roadshow or a little city tour, where you’ll be talking about this, as well as other areas of interest around DevOps and what CollabNet’s doing. What’s going on with that?

Robertson: Yeah, so we’re actually having a series—we’re also doing it through webinars, but we’re also doing it through seminars, and we’re gonna be having these seminars throughout the different series. And, basically, what we’re doing there, what’s really critical there, is that we’re going to be not only talking about the value stream as far as more of the how this applies into the DevOps—in DevOps world, but also we’ll have customers there talking about how they are leveraging value stream mapping and how they’re mapping that to their tool chains and how they’re able to, from bare, start applying measurement, which, to them, is what they’re calling “the last mile” of DevOps, right? And it’s really—they’re gonna talk about how that provides a window to allow them to monitor, measure, improve their workflow processes, all around a series of KPIs, both velocity KPIs and also quality KPIs.

Shimel: Mm-hmm. Excellent. So, again, I think, Eric, what we wanna do is post a—maybe if you have a schedule of cities that we can post with the notes from today’s show.

Robertson: Yes.

Shimel: I think that’s the way to do that. So—

Robertson: Yeah.

Shimel: —let’s return, though, for a moment, to value stream mapping and kinda dig in here a bit. One of the things—and, again, as we spoke about off mic, I’ve had some experience with value stream mapping—one of the things, though, in speaking to potential customers of value stream mapping, is it’s easy to say, “Oh, yeah, that sounds good.” Right? “Oh, that’s something we wanna do.” Okay, well, let’s schedule some time and I’m gonna need the right people at the table because, if you don’t have the right people there, it’s very hard, number one, to truly map out the value stream and, number two—

Robertson: Exactly.

Shimel: Right? And harder than that is let’s get some consensus on what we’re gonna do to make it happen.

Robertson: Exactly.

Shimel: You know, unfortunately, it can’t be some developer who raises his hand and says, “I really wanna make a digital transformation here and I need to convince my CIO. Can you come in and give me a value stream mapping kind of session or workshop or can we discuss it?” It’s not a one-man show.

Robertson: Exactly. Exactly. And it is very interesting you said that because, because value stream looks at the lifecycle from planning to operations, there’re different stakeholders, right? And what’s happening in, especially in enterprise, is these stakeholders now—they have implemented the tools or they have guys that work for them or folks that work for them that implemented these tools, and they’re starting to ask questions or they already started asking questions around value. What I mean by that—you have the executive and he’ll say, “You know, my folks are telling me we did four deployments today.” Great. What does that really mean for the business?

Shimel: Yeah.

Robertson: Did we get little features out to the business, where they get utilized? Are they utilizing that? Did I actually place the right emphasis on those features? Right? The last release that I deployed on time, at least from a developer perspective, said I deployed this, I deployed it on time, but I’m having this increase of service desk tickets. What’s going on here? Why am I being called? Right?

Shimel: Yeah.

Robertson: And I _____ –

Shimel: Oh, yeah. Exactly.

Robertson: Right? [Chuckles] And it means that the IT office guys, right? They have their own set of—type of metrics that they’re looking at, right, and they wanna understand, right, “Is there anything impacting this release?” and without having to chase down and talk to every other person or look into every tool, to try to gather this information, right? And then you’ve got the engineers, like the QA engineers, and now compliance, security’s coming into play as well, right?

Shimel: Absolutely.

Robertson: So now folks wanna know, “Before I deploy, did all this pass through the quality gates? Was this sufficient?” Right? “What about security and compliance checks? Can I automate that feedback? Can I get that feedback to the appropriate stakeholders?” Right? So they’re already involved in asking these questions, but they really don’t have any direction there on how to get that information. And this is where—this is the impetus to say, “Well, you know what? Now’s the time to start looking at these tools, where you’re gathering this information, what’s generating this information flow, and how we bring this together.” Right? So it kinda starts to endow that value stream mapping conversation pretty well.

Shimel: Yep. Excellent. Excellent point. Because, like DevOps itself, value stream mapping is a team sport, right? And—

Robertson: Yep.

Shimel: And it’s cultural and you need that group kind of involvement to make it happen. So, Eric, I’m sure you probably have a few war stories around this, though. Can you share with us sort of—you know what I always say? You learn more from your failures than you do from your successes.

Robertson: Oh, my God.

Shimel: Can you share with us maybe some value stream mapping engagements that didn’t work out and, of course, only if you’ve learned something from it, and what was the valuable lesson you learned?

Robertson: Exactly. I mean, that’s—it’s very interesting that you said that because we had one customer—and I can even use it from my experience. I mean, I worked at a company and I used to dread Wednesday mornings because Wednesday mornings was—wasn’t really a scrum meeting, but it was more of a meeting as far as, “What is the latest status of this release that went out the previous day?” because we did weekly deployments, weekly deployments. And everybody would dread that because it was really no place to go to get a full understanding that, “Well, if this release actually was deployed, what was in that release? If it wasn’t deployed, what was the bottleneck in there?” Right? “Was it that it failed a certain testing that it didn’t get through or did we discover that, in this environment, we did some monitoring and it didn’t pass this stage?” We had no idea. I mean, did the builds fail? Why? There was no insight to anywhere in that chain, right, from the planning, ’cause it’s all kinds of activities that are happening in the coding, the building, the testing, the deploying a release.

And so, if you don’t really have an idea, as far as that information flow in a place where you can go and view that information and quickly understand “What are my exceptions, right, that I need to quickly identify and quickly address?” everybody’s chasing their tails there. And I’ve been on so many projects where that transpired and happened. And so this was really impetus of why we wanted to start having—start putting—leveraging this best practice, leveraging these practices, and then the tooling becomes more secondary type of aspect, rather than primary goal for driving processes.

Shimel: Yeah, I agree with you. The tooling then winds up taking sort of a backseat to it. So let’s—well, we’re almost out of time, unfortunately, Eric. As I mentioned to you, the 15 minutes goes quick. We do the best we can.

Robertson: No worries.

Shimel: Let’s put—

Robertson: I’m enjoying it.

Shimel: —value stream mapping to the side a little bit. What else is—

Robertson: Sure.

Shimel: What else is going on with CollabNet?

Robertson: So CollabNet is also partnered with a company called “Clarive”—it’s release automation software—and what we’re doing is is that we’re linking the release automation capabilities into our value stream management platform. Now what’s nice about this is that this allows us, very quickly, to start getting metrics across different type of applications and projects. And what I mean by that is, if you go into a larger enterprise, for example, they – of course, they’re experimenting with container type technology and Web-scale IT technologies, but they’ve also got some legacy applications, right? There may be Java, right? There’s—or on WebSphere or a traditional type of applications or architecture. They may even have mainframes or they may even have package apps, right?

So one of the things that we’re doing is that we’re getting—we’re creating this abstract _____ layer around all those tools, so customers can have, enterprises can have a common user experience around release automation, whether or not—so, again, we’re abstracting the tools, so you’re focused on the process, and it doesn’t matter what tooling is utilized underneath. If it’s Webscale IT technologies, container-based technology, it does not matter. You can basically have a common release process across your different type of applications—package, legacy, and traditional—so we’re very excited about that.

Shimel: Excellent. Sounds very cool. Eric, we’d love to talk more to you about this, but we kind of are over time. What I’d like to do, though, is maybe invite you back and we can talk. You know, we’ll leave value stream mapping for a moment and we’ll talk about other stuff next time, but I think we’d like to hear what’s going on in the world of DevOps, obviously, and with the players in it, and CollabNet has certainly been on the scene here and lots to tell. Until then, though, it looks like we’ll call it a wrap on this version of DevOps Chat and we’ll speak to you soon, and I—Eric, we’ve gotta get those links over. There’ll be in the notes for people who are listening in. But until then—

Robertson: Great _____.

Shimel: Eric Robertson, VP of product management and engineering at CollabNet, thanks for being our guest on this episode of DevOps Chats.

Robertson: Thank you, Alan. Glad to be part of this.

Shimel: My pleasure. This is Alan Shimel for devopsy.kinsta.cloud and DevOps Chat. Thanks for listening in today and we hope to see you soon on another DevOps Chat. Buh-bye.

— Alan Shimel

May 16, 2017
Lean for DevOps: Mapping the tributaries and deltas of demand
Here at IBM’s Lean practice we use a diagram called ‘Mississippi’ to analyze demand. It is not so-called because it originates in Mississippi – in fact I’ve been given to understand that this tool was developed, and named, in our UK practice*. Instead, it’s called the Mississippi because, like that great river, demand comes from many different sources (or tributaries) and can leave a process through a number of different points (the delta). And, apparently, at times the Mississippi delta reflects tidal patterns and flows upstream. In the same way, demand can route back into, or upstream within, a process – for example, in the shape of rework.

We want to analyze the volume and frequency of demand for a number of reasons. Understanding demand helps us to:
- Identify ‘runners’ (everyday requests), ‘repeaters’ (less frequent, but not unusual requests) and ‘rarities’ (infrequent requests). This might be in terms of user groups, applications, or type of request (for example – small changes vs major project). When we design our improved future state, we will design it for runners and repeaters. Sometimes the impact of exceptions can be so painful, team’s lose sight of how infrequent they are, and design a process to prevent or manage those rarities – which will be inefficient for all those runners and repeaters.
- Identify failure demand – Where are additional requests re-entering the process, as rework? Where is there process activity which doesn’t ‘go anywhere’?
- Illustrate channels in and out of the process, so we can identify potential control points, or illuminate a lack of filter, and see where demand might be routed in via unexpected channels.
- Tell us what we don’t know: which elements of demand are we unaware of? We need to understand all demand so that we can build and resource (load level) a process which can meet that demand. Sometimes finding out what we don’t know is a lesson in itself.
These benefits are as relevant to Lean for DevOps as in any other area.

It’s hard to say what you’ll learn from conducting this analysis, until you’ve done it, as every Mississippi is different. But to demonstrate some of the valuable lessons I’ve learned from drawing up Mississippis, here are some real life examples:

Lightbulb moment: ‘Churn is a fact of life – we need a better way to handle it’

Lightbulb moment: ‘We need to start saying no to enhancement requests’

Lightbulb moment: ‘We have way too many changes taking the emergency route: a faster ‘normal’ route will reduce that demand and the associated risk.’

In their different ways, each of these helped make the case for a shortening of the SDLC, closer coupling of development to business demand, and a more nimble deployment process.

*If you’ve seen this in use outside of an IBM engagement, I’d love to hear about it! Find me on Twitter to set the record straight.
July 14, 2015
Lean Value Stream Mapping for DevOps
Value Stream Mapping is a crucial step in assessing an organization’s DevOps capability. We create a value stream map of the software development lifecycle early in any DevOps engagement, because it helps us to-
- Provide context for our architects and technical experts who are, in parallel, examining tools and technology.
- Establish a performance baseline, against which the improvements delivered by adopting DevOps will be measured. (See my other blog post on Box Scores for more on this).
- Begin the process of winning hearts and minds, and engender cultural change. By working side-by-side in a workshop, Dev and Ops resources, as well as those involved in planning, test scheduling, and other activities, start to gain an understanding of how the work they do is linked to steps upstream and downstream. This lays the groundwork for closer coupling of those activities in the future.
What’s interesting is that the template we use for our DevOps Value Stream Map is almost identical to that which we use for other service processes. It’s been used by IBM Lean Coaches to assess processes as diverse as Marketing Campaign Development, Order-to-Cash, Mortgage Applications, and Production Support to name but a few. This template was evolved from more traditional Value Stream Maps used to capture manufacturing processes, and a slightly different template also exists for innovation or product development processes.

At the start of every Lean project – whether it’s in DevOps, Insurance Claims, New Product Development, or Financial Forecasting, I review the Value Stream Map template. I ask myself and my team if the questions make sense for this process and if the boxes we are seeking to fill are relevant. Some of the language might be slightly different, and the outputs will certainly differ – from a high frequency process where a step takes minutes and seconds, to a creative process where a step could take weeks. But almost without exception, the template remains the same.

We ask the same questions across all these types of processes – and now we ask those questions of the Software Development Lifecycle. Those questions are:
- What activities are being performed?
- Who delivers those activities? How much of their time is dedicated to those activities?
- What is the yield, i.e. ‘right first time’? What % of rework is generated? What causes rework?
- What tools are required to complete the step? And what other inputs are required?
- How many units (e.g. work packages) are ‘work in progress’ within this step?
- How many units are waiting in the ‘inbox’? How will they be prioritized or scheduled? How long will they, typically, wait for?
- How long does the step take? How much of that time is effort, or touch time, versus duration, or in-process lead-time?
When all’s said and done, all Value Stream Maps have something fundamentally common: they are all about flow – and DevOps is no different.

About the Author/ Katie Sotheran

Katie is an IBM Lean Master based out of Los Angeles, California. Katie has many years applying Lean principles to processes as diverse as beverage distribution in Spain, lubricant product development in the Netherlands, Insurance Sales in the UK … and IT operations, all over the world.
May 19, 2015
DevOps and Organizational Debt

I recently published a blog on the Perficient Digital Transformation blog entitled “Is Cloud & DevOps Factored Into Your Digital Transformation?” In that blog I discuss the concept of organizational debt and its impact on your DevOps adoption. Here’s what I had to say in that blog:

If technical debt addresses too little investment in upgrading, poor vendor selection, bad platform selection, etc. then, many businesses have also built up organizational debt. Organizational debt could be over regulation, too many low-value policies, to much middle management attempting to hold onto their area of control, and too much governance. If these are not revisited often to ensure that they are continually adding value for the cost of reducing throughput, then that organizational debt will continue to increase backlog.

Interestingly, there’s a lot written about technical debt, but very little has been written about organizational debt, which can have a significantly deeper impact on your ability to continuously deliver than your technical debt.

In working with clients on their DevOps initiatives, we map the current value stream for a given solution area. What becomes apparent from these value stream maps is that controls and policies have been derived within organizational silos with no attention given to how that will impact IT’s overall ability to be agile and deliver more quickly for the business. Moreover, the owners of these controls and policies seem reticent to remove these bottlenecks even after being shown how it impacts business agility.

Here’s an anecdote from one client when asked why they aren’t using snapshots to speed preparation for QA testing. The suggestion was made in response to the client stating it can take upwards of a week to prepare the QA environment for a single round of testing. Their current approach serializes all testing and believes they need to perform a complete regression test due to the resource contention issues. Hence, they cannot test small changes easily and they cannot test often due to these limitations.

“The main issue is that we don’t control the application or database environment, so we need to prepare a QA release and then have those groups perform their duties in preparation of the QA environment for testing.”

Upon hearing this my mind summoned up past experiences working conferences at major venues, such as the Jacob Javitz Center in NY, where you can’t plug in an extension cord or run an Ethernet cable yourself, but you need to use the Javitz Center’s electrician because of union rules.

Is this what IT has become a set of unofficial unions that hold the business in a death grip? This is clearly a case of organizational debt that needs to be addressed.

Of note, I’m not admonishing the various groups for instituting their controls and policies. I’m sure they have emerged due out of need due to past grievances, failures or as part of IT management practices. Regardless, they are now acting as a bottleneck in implementing proven practices for delivering fewer features more often, which have been proven to increase trust and facilitate better prioritization of work.

The result of this organizational debt is lead times have grown to four months or longer to release functionality that is directly related to key business initiatives by the business for increasing revenue.

In his book, “Kanban,” author David Anderson discusses his recipe for addressing bottlenecks and constraints with existing teams. One of his steps is ‘attack sources of variability to improve predictability’. However, this is his last step because, as he notes, “some types of variability require behavioral changes in order to reduce them. Asking people to change behavior is difficult!” And, so it goes, many businesses never address this key constraint because the hurdles of organizational debt have become so burdensome to overcome. It requires a high degree of trust and political capital along with executive sponsorship. As shown by the anecdote earlier, even if the constraint is directly impeding other business priorities it is not easy to overcome.

I believe DevOps practices can help organizations significantly reduce their technical debt. I have seen and read about organizations that have been able to rapidly reduce their lead times while still refactoring and removing backlog. However, the goal of DevOps is to improve the entire flow, not just development, and a good majority of the release aspects of applications will require organizational debt to be addressed to enable improvement across the entire continuous delivery pipeline.

February 6, 2015