Tag: technical debt

Adding Accountability with ‘You Build It, You Run It’
Let’s see if this scenario resonates with anyone: Your department, organization, company, etc. is beginning to embrace DevOps practices, which is, of course, an amazing step in the right direction! One of those key practices is the “You Build It, You Run It” (YBI/YRI) mantra that helps facilitate a higher level of quality and accountability. This common practice also solves the mentality of throwing your solution/code over the fence for some other team to fix.

I’m a huge proponent of YBI/YRI. However, in the case of existing brownfield solutions, there are several things you need to consider when transforming into the YBI/YRI model, including:
- What if you have an existing solution where the collective “Y(ou)” in ‘You Build It’ does not include any or many of the original team that built it? This would be anyone who was part of the end-to-end creation of the solution and was empowered to make decisions impacting the design, development and delivery of a solution. In this case, it becomes, “Someone else built this mess, now you run it,” [SEBTM/NURI] which not only makes for a horrible acronym but is also a very realistic scenario that requires special consideration we’ll discuss below.
- What if you are still heavily siloed and have a squad spread across multiple organizations, each with different goals and objectives? If you are still running outside of a true squad model, it can be difficult to ensure everyone’s priorities and goals are aligned to produce quality solutions with the right functions customers want.
- What if you have varying definitions of “You” within your organization? Often in the YBI/YRI model, the interpretation makes some roles or key decision makers exempt from the glory of pager duty or answering to an angry customer. This can be especially true when you’re still trying to get to a point of having fully cross-functional, empowered squads.
I think it’s fair to say that most DevOps transformations don’t necessarily happen on greenfield solutions, which means, especially at the enterprise level, the reality can end up being a beast of a monolithic solution(s) built by others who may or may not still be around.

How do you fix it? My suggestion is that every person who was empowered to make a decision that contributed to the “B” (Build), becomes a “Y”(You). Of course, this doesn’t work if “Y” is no longer with the group or company, in which case I think it becomes a different issue of ensuring the new “Y” is 100 percent empowered with the ability to make decisions that enables that team to successfully run it.

One more complication that adds to the mix of a YBI/YRI strategy is putting that strategy in place without a solid foundation of agile practices. Ideally when you are adopting DevOps practices, you already have a solid agile foundation with empowered cross-functional squads; however, in practice we often see the agile foundation still being built and/or refined at the same time as incorporating DevOps practices. As a result, there still exists a lot of hand-offs while two transformations are trying to run in parallel. If you have hand-offs combined with team’s with separate business objectives, those blurry lines of accountability will become an issue sooner than later.

During transformations, if you find yourself in that in-between state of not having fully empowered accountable teams and are working with solutions your team may not have actually built, below are a few suggestions to get through those initial challenges:
1. Ensure you allow for enough time to address technical debt during your sprint planning. Continuing to add features and functions on a poorly designed and/or broken solution will only cause further issues and likely increase the level of frustration on the team.
2. Ensure that EVERY person who contributes to key decisions on a solution is included in the pager rotation in some way. Until you are in a state of fully empowered cross-functional squads, it’s important to drive the importance of accountability. As an example, if you have an architect in one organization who is working with a development team in another organization, that architect needs to be included on the pager as well to be aware of any problems resulting from a key architecture decision.
3. Get in the habit of operational retrospectives. Begin (or continue) the habit of operational retrospectives that bring together the end-to-end squad into a cross-functional review of the operational incidents, root cause assessment and stories/tasks that need to get added to the backlog and prioritized.
Although the transition into YBI/YRI for brownfield solutions can be muddy, empathizing with the mountains of technical debt and fully empowering solution teams can go a long way in truly helping your team’s succeed through the beginning phases of your DevOps transformation.

About the Author / Shelbee Smith-Eigenbrode

Shelbee Smith-Eigenbrode is a senior software engineer/IT architect at IBM and has been in IT for 19 years. She has worked across varying roles spanning the software development life cycle. Her experience across functional roles as well as performing that work deeply inside traditional silos has contributed to her passion for embracing the DevOps culture and transforming the way teams are delivering technologies by adopting DevOps practices. She is a member of the IBM Academy of Technology committed to driving innovation and technical advancement.
July 7, 2017
Best DevOps Practices for 2017

Because the idea of DevOps is still so new, we’re continually having to hone what we view as best practices. At the start of the new year, many people make their own personal goals clear for the year (regardless of whether they intend to keep them), but we tend to head back to our work desks without reassessing how our projects are going. We just jump back in unquestioningly and continue down the path we were on before the holiday.

Mindless plodding away at a goal that had been set too far in the past is perhaps the biggest business mistake we can make. Re-evaluating our approach is often the difference between stagnation and real innovation. A willingness to change tacks and the agility to do so speaks volumes about an organization—that flexibility is rewarded by a market that demands responsiveness from its providers, and it can be a big differentiator between a company and its competition. With that in mind, there are six big goals that we can adopt as DevOps practitioners that will have enormous results on our ability to answer the demands of our users.

Make paying off debt a priority

Most companies are so saddled with technical debt that they hire more people just to cope with it. This is like putting more hoses on a forest fire. When fighting a forest fire, a portion of the team creates a firebreak to deprive the fire of fuel. Likewise, organizations need to dedicate a portion of their workforce to do nothing but eliminate the technical debt that is fueling their operational fires.

Set ambitious goals, but don’t sweat the details

Have goals and objectives, but allow the minutiae to be developed over the course of short intervals. We need to set business-level objectives for our DevOps teams, but we have to be careful to not overprescribe the details. There are many DevOps tools and methods and more arrive every day. Plan to work in two- to four-week sprints with measurable outcomes, instead of six-month project plans. Use the feedback and lessons learned from those short sprints to inform the next sprint’s work.

Don’t leap, but get moving

DevOps is meant to be about quickly moving product from left to right. A common mistake is for companies to wait for “the perfect moment” and then jump headlong into DevOps, creating a lot of disruption at once. This is a recipe for disaster. Identify workstreams that are ripe for improvement from DevOps and start by moving one new benefit from concept to production. Remember, single-piece flow is your friend.

Identify the ‘credit hawks’ and neutralize them

I once heard the saying, “Just because you run out in front of a parade doesn’t mean they are actually following you.” One of the critical pieces that makes DevOps work is a culture of sharing. That means sharing in knowledge, work, credit and blame. A “credit hawk” is someone who likes to jump in front of work already being done and claim he or she is orchestrating it. This seems to be commonplace in situations where DevOps teams exist under mid-to-upper managers with “old guard” mentality. If they take the credit, that leaves the DevOps team with the blame. Neutralize the credit hawks at all cost; they will undermine your culture.

Plan for failure, success through learning

Planning for failure does not equate to “expect failure,” but because we want to move faster so we can try more things, the law of averages tell us that we will also fail more, too. Planning to fail means we know how to conduct a blameless post-mortem, understand lessons learned and incorporate that feedback cycle to increase the likelihood of success.

Promise to break the rules

Your DevOps team needs to be trusted to find the best path, even if that means ignoring a 70-year-old process. This statement might invoke a knee-jerk reaction, but this does not imply “anything goes.” Your objective is to find more productive ways to get product to market and to create safe systems of work. Part of your lease to your DevOps teams also should be to explore new processes that eliminate red tape, bureaucracy and waste. What has worked for the last 70 years may be what is holding you back today.

2016 rewarded those who were able to move fast, and 2017 promises to require us to move even faster. By resolving to make these changes in your company or organization, you are making a commitment to shed the shackles of yesteryear. When freed of these fetters, your application teams will be able to focus and deliver on the one thing we are all trying to do: deliver value directly into the hands of our customers. Is 2017 the year your company will triple its release cadence? Is this the year you deliver every new product into all markets simultaneously? Or perhaps this is the year when your applications suffer zero outages. The goals and the stories are yours to tell, but one thing is for sure: Resolutions without action are just wishes.

About the Author / Adam Bowen

Adam Bowen is an information technology leader with more than 15 years of experience in technical, sales and senior leadership roles. As the Worldwide Innovation Lead at Delphix, Adam works with thought leaders of the Global Fortune 100 to eliminate key business constraints through innovative new processes, offerings, and technologies. Connect with him on Twitter and LinkedIn.

January 6, 2017

DevOps Survey Says! State of DevOps, Reliance on Scripting and IT Automation

What our surveys told us about the state of DevOps, Reliance on Scripting and IT Automation

DevOps Survey Results: Scripting & IT Automation

A lot of IT Operations teams including application owners, configuration managers, middleware specialists and database administrators are managing infrastructure configuration change with manual processes, spreadsheets and scripts. They are spending a lot of time writing and maintaining a lot of scripts. Even with all those scripts, spreadsheets and processes and the resources they consume, teams are still not getting the visibility, control and error-proofing that they require.

That is the message from five recent surveys conducted by Orcaconfig among IT industry pros. Perhaps these results should not be that surprising. But several years into the DevOps and IT automation movements, there is still plenty of room for improvement.

About one-third of respondents said they spent more than half of their time writing and maintaining scripts.

In complex application environments with multiple relationships and dependencies between applications, databases, middleware and operating systems, “changing this can easily (and unwittingly) change that.” Gartner refers to this as a domino effect. So, scripting to deploy changes becomes more and more tricky and thus, requires more and more time to do, test, redo and retest.

what_percentage_of_time_is_spent_writing_or_maintaining_scripts_-_survey_from_orcaconfig-com_2

Because of the risks involved with release or configuration-related scripting errors in production environments, some organizations hand these typical operations responsibilities to developers. The question is worth asking: Is release and configuration-released scripting the best use of Dev or Ops’s time?

how_do_you_maintain_scripts_when_the_author_is_no_longer_available_-_survey_from_orcaconfig-com_3

With all these scripts holding key configurations together, are organizations at risk of “losing the recipe” if key employees leave? Apparently so. In our surveys, 30 percent of respondents admitted, “It’s a problem, we don’t really have a solution in place,” when asked about maintaining the spaghetti-works of scripts if some of those key employees suddenly leave the organization. Nearly half mentioned that they rely on transfer of information (TOI) before a key employee leaves.

Employees exit companies for a number of reasons and some are in a bigger hurry than others to move on. Does TOI provide all the safeguards needed for the enterprise?

Scripting might consume a great deal of IT resources and introduce a considerable risk with the loss of key talent. But does it at least accomplish the functionality that teams need? The next few charts provide insight to how well scripting achieves the visibility and error-proofed automation that teams need.

Ecosystem Visibility

Configuration Drift Detection

For performance, security and audit reasons, organizations have an ongoing need to maintain configuration compliance in their application ecosystems. So detecting configuration drift is essential. Automatically detecting configuration drift is the goal. Despite complaints about the method, approximately half of respondents relied on scripting to accomplish drift detection.

how_does_your_organization_detect_configuration_drift_in_complex_application_database_middleware_environements_-_survey_from_orcaconfig-com_2

how_do_you_know_which_databases_are_being_used_by_which_applications_across_your_ecosystem_-_survey_from_orcaconfig_2

Mapping Databases to Applications:

Sometimes IT operations are asked simple sounding questions such as, “What are all the applications that use this database?” and vice versa, “Which databases support this application?” Once again, nearly one-third of respondents admitted, “We really don’t know which databases are being used by which applications. Wish we did.” Another 36 percent relied on scripting or other manual processes to get that answer.

Is it reasonable (in 2017!) to still rely on scripting to get basic information about interrelationships between applications, databases and middleware? Is it acceptable to operators, managers and auditors simply do without this information? Will Visio and similar static files always provide up-to-date information that IT Operations needs?

Automation & Error-Proofing

Environment-Aware Releases
When moving an application through pre-production stages and into production environments, Release Managers want to ensure that the new release will just work … as planned … drama free. This requires among other things that

the configurations are up to date and production-ready, and
that those business-critical applications are promoted simultaneously with their production-level configurations.

how_does_your_organization_achieve_environment_aware_releases_

Thirty-eight percent of respondents said they relied on scripting and other manual processes to try to accomplish environment-aware releases. To ensure ‘everyone is on the same page’ many organizations use multiple planning meetings, sharing and updating spreadsheets and SharePoint files with (hopefully) the latest configuration information. Even among this group, respondents commented that scripting just does not work well for them.

Despite the security, compliance and performance implications of configuration and release problems, 15 percent of survey respondents admitted they “really don’t achieve environment-aware releases.”

Scripting isn’t Free; IT Ops Needs More

Scripting is quite naturally the go-to method for many application owners and IT Ops teams. The cost of scripting or even finding and fixing a scripting-based error hardly ever appears clearly in budgetary line items. In the moment, it can feel like it is free. But over time, maintaining scripts and homegrown tools can build a hidden but costly mountain of technical debt. It might well be worth these costs and risks if scripting effectively resolved ongoing operations needs such as ecosystem visibility and drama-free deployments. However, time and again our respondents said they lacked basic capabilities such as configuration drift detection, mapping applications to databases and middleware or ensuring environment-aware software releases.

— Scott Turner

December 2, 2016

Pay Down the Skills Debt Before the Technical Debt

When we talk about DevOps, we talk about addressing technical debt. We talk about how modern delivery processes and solutions might help you cure that technical debt of monolithic systems that are built on obsolete technologies, using antiquated development methodologies. But that jumps past the first critical step toward transformation, as technical debt is not the biggest inhibitor to DevOps or faster delivery. By far the biggest inhibitor to companies moving forward is what I call the skills debt.

I constantly hear this when talking to executives: “I’d like to move forward with a change, but … my team is …” and that statement always represents a skills debt. The primary limiting factor is not the systems in play, it is that the people aren’t ready or positioned to make the necessary changes. We need to make sure we have a strategy for eliminating the skills debt before addressing technical debt. This is the single-biggest risk to DevOps inside the organization.

Why is the skills debt so endemic? It’s not that people don’t care about their jobs, or that they are not smart or capable enough. It is a self-fulfilling prophecy for most companies because people accrue habits, usually over the course of a decade or more, in the way things are done.

Changing Mindsets to Fill the Skills Gap

Ossified attitudes. Just like we talk about addressing the technical debt of hardwired, inflexibly constructed monolithic apps, organizational structures naturally will harden and develop into ossified attitudes as people favor procedures and skills that are already in place. An organization behaves according to Newtonian physics when it comes to change: An object in motion wants to stay in motion, an object at rest wants to stay at rest.

It’s not as though companies want to have a skills debt, but over time they come to value “the way things are done around here” as part of their identity, and the way corporate structures handle career advancement can even play into this ossification. Think about the technical leadership of the company, the people who rise to the top might have managed successful projects in the past, and while they may not crack open an IDE and code anymore, they will still tend to advise teams to stick to what worked in the past.

New people and ideas can’t stick. Let’s say you have this established skills debt running through your halls, and you bring in a new hotshot to shake things up. Once this person tries to introduce change, they will immediately find themselves lacking traction at every point. The skills debt is like a Teflon coating around the organization’s IT practices, protecting the core of the business from its own ability to change. A new person with new ideas will just skate off the top and bounce right off.

If you work for a cutting-edge startup, or someplace like Google or Apple that naturally has innovative engineers lining up to get in, you probably don’t have a problem with recruiting and allowing new hires to make an impact. But what if you work at a big bank, or a large retailer or a consumer goods manufacturer? Changing attitudes and new skills will largely have to be cultivated from within.

Consulting doesn’t address your skills debt. So if you can’t change fast enough, or hire the right people, how about getting some outside help, some “Pros from Dover” to do that transformation for you?

Consultants will approach an engagement from two angles. Many times, the consultants are very academic. They’ll come in and show you a fancy PowerPoint, maybe teach a class on Agile Methodology and tell you what’s wrong with your processes, without a practical starting point. That solves neither the technical debt, nor the skills debt.

Or in the outsourcing model, the consultant can offer to take all your problems away, and have their own resources do all the work in the project for you, even implement a whole new system. That may solve your short-term technical debt, but it will simply allow the skills debt to continue to accrue. You now have a permanent dependency on this external team, unless you allow the project to end. In any case the people responsible will someday move on to other projects and away from your company.

Digging Your Own Grave

So, give up and stand pat? Remaining static is a huge mistake in today’s economy. I don’t have to explain that standing still means certain death for your company. On average, a firm in the S&P 500 gets replaced every two weeks. You’ve got to be able to constantly outpace competition—and even your own past results—on a consistent basis.

What you need to do is take the people that are stuck and unstick them. Yes, sometimes specific people will need to be replaced, but by and large the skills debt is something your whole team has to own responsibility for as a first priority before we talk about the technical debt of DevOps. You have to start breaking skills debt down from the inside out, because doing it from the outside in has been proven to fail.

Start changing the DNA. You have to start somewhere. Nobody will be motivated by hearing that the way they are is “no bueno.” If you go and mandate a sweeping cultural transformation across the whole company, everyone’s heads will explode.

The linchpin to making transformation work? A single project. A real piece of work done a different way, by a representative group of the people who need to change. You know how gene therapy is done? One project that allows teams to use the right tools for the job and modern techniques will insert a bit of code into that organization’s DNA. That replicates into beneficial changes elsewhere in the organization as the success and better experience of that team is propagated.

Skills debt is only paid down through direct transference during actual work. This doesn’t happen without effort, but few worthwhile things are achieved without a little struggle. You can’t just tell people about it, or teach them about it—you have to get people involved in projects that resolve the skills debt of DevOps, before you solve the technical debt.

One technology executive at a big media company who made this change said his teams used to ship code to release every seven months, and now they are shipping once every two weeks, with an eye to releasing once a week. That’s a huge impact that would not happen without paying down the skills debt. There’s not any one magical piece of software the company installed to make that change; in fact, it is still working on the technical debt side of the equation today. It made the change happen within, starting with one project done the right way.

Getting people ready for DevOps-style transformation takes a focus on activity and real action on a real project. Non-passive learning cycles, and hands-on work with the new technologies and new methodologies can cure your skills debt, and then you can start paying down your technical debt.

May 2, 2016
Only DevOps and Taxes

As software eats the world, it’s hard to have a conversation about anything that isn’t directly related to or driven by modern mobile, web and IoT applications. DevOps describes the modern approach to building these applications. But like all new movements, there is a cost. And while some of the costs associated with modern development are not new, it’s becoming more and more clear that DevOps organizations have to pay their tech tax or prepare for trouble.

(more…)

January 12, 2016
DevOps Leadership Series: Boston Common

DevOps is coming, DevOps is coming! You could hear the rallying cry around the city of Boston on September 15th and 16th. This is when the DevOps Leadership Series crew headed up north to attend DevOps Days Boston.

We first caught up with Beno Chapman (@BenoChapman), release engineering leader at Constant Contact. Beno shared, “Don’t be afraid to prototype something”. Sage advice for many DevOps practitioners. He also provided perspective why we should all consider “productizing services” that we offer to stakeholders across the different functional areas of our business.

https://www.youtube.com/watch?v=nWQitydh904

Next, we hung out with Anthony Spring (@DevOpsAnthony), minister of DevOps culture and tech lead at Yieldbot. Anthony delivered a compelling lightning talk in Boston around quantifying technical debt. He shared a brief recap of that topic and then discussed a new theme in DevOps conversations that points to challenges around hiring the right people and vetting the right tools.

https://www.youtube.com/watch?v=sQufvcFiRmo

Jeremiah Sullivan (@Jeremiah_NYC), Technical Solutions Engineer at Puppet Labs, joined us next. He talks about the growing interest in containerization and solutions like Docker and Rocket. Jeremiah then shared some insight into how DevOps teams are expanding beyond automation at the server level to now include managing infrastructure like switches, load balancers, and hypervisors.

https://www.youtube.com/watch?v=KEcIeBIrQNI

We’ll continue coverage from DevOps Days Boston in our next post with videos from James Meickle (Site Reliability Engineer at Harvard) and Jennifer Davis (co-author of Effective DevOps – O’Reilly 2015).

Want more? You can find the entire library of 21 videos from this series on YouTube.

Special thanks to my colleague, Mark Miller for capturing these videos for us at the event.

September 24, 2015
The Cost to DevOps: 27 Mufflers

Imagine

Imagine that you are designing the 2016 Range Rover line of sport utility vehicles. Like all gas powered vehicles, each one needs an exhaust muffler. Range Rover likely has narrowed in on a preferred provider of mufflers. But imagine what would happen if the designers and factory line workers could pick from any one of 27 past versions of that muffler from their preferred provider for the new model year. Yup, choose any one — even if it is outdated, lower performance, does not meet current emission standards, or has a known defect. Order it, place it on the vehicle, then ship it.

We all realize this scenario would never happen. No one would select a known defective part for the product they are manufacturing. And who would knowingly choose the 10 year old version of a muffler without the latest advancements to support air quality? No organization would give a each of their designers and line workers free reign on procuring any one of 27 different mufflers for a single model year. Imagine the operational cost of testing the different mufflers. Imagine the maintenance cost of having different versions of the same part shipping. Worse yet, imagine the liability of shipping parts with known defects.

Land Rover is not taking this approach, but software development teams are.

27 Versions

My latest research report, the 2015 State of the Software Supply Chain, reveals that large software development organizations sourced an average of 27 different versions of open source components that they used in development last year. To be more specific, out of the top 100 open source components these companies downloaded from the internet in 2014, they averaged 27 versions of each component.

While auto manufacturers like Land Rover have preferred suppliers of vetted parts, the procurement of open source components is often more of a free-for-all. If an organization has 300 developers, they effectively have a procurement department of 300 for sourcing externally developed software components. If you have 4,000 developers, you have 4,000 performing procurement. Even as we all know only the latest version of a components is the most feature rich and of the highest quality and integrity.

Not all organizations allow free-for-all sourcing practices for components. Fifty-seven percent (57%) have open source policies in place as a method to guide developers toward higher quality parts with the fewest known defects or security vulnerabilities. Organizations like Google mandate the use of no more than two versions of a given open source component across their development teams.

Good Cost, Bad Cost

What does this mean for a software development organization — especially one focused on improving it’s DevOps practices? First the good news: developers are using open source components to speed time to release and improve innovation. Both good things.

The bad news: Organizations are building mountains of technical debt through poor software supply chain practices by sourcing an average of 27 versions of the same component, impacting quality, complexity, maintainability, supportability, etc. They are doing this across the top 100 components being downloaded in a single year which equates to managing over 2700 different versions (when an optimized practice would only require the 100 best).

Organizations are sourcing component “parts” for use in their applications that have up to 26 known better versions available (more features, fewer bugs, improved performance, etc.).

Organizations are downloading and using parts that are sometimes 9 years old, or even older (simple math: open source projects release 3 – 4 new versions of their components each year). They are electively using outdated parts.

Organizations are also downloading some versions with known, documented security vulnerabilities. They are electively using known vulnerable parts.

Net Innovation Through Improved Sourcing

While we often hear people tout a DevOps mantra of “even faster”, there is a consequence to pursuing speed at any cost: technical and security debt.

Open source components are enablers of speed. But free-for-all sourcing and use practices, add to technical and security debt that reduces “net innovation” and adds to the cost of operations. As technical debt grows, net innovation and business value drops.

Image source: http://joapen.com/blog/2014/06/17/how-to-monetize-application-technical-debt/

If reducing the use 2,700 versions of 100 components was not motivating enough, consider this: the research also revealed that large development teams were consuming software components from over 7,600 external open source projects (suppliers). Including all versions of those components, they average organization consumed nearly 19,000 unique components in 2014.

Improved sourcing practices are just part of the overall story for improving net innovation and increasing business value. Improved sourcing practices can also enable more efficient operations, reduce mean times to remediation, and eliminate unplanned work for many DevOps practices. Improved sourcing practices have been employed across many industries resulting in sustained competitive advantages. Improved sourcing practices are part of a broader software supply chain management strategy.

Continuous Learning

There are so many lessons we can learn from other industries who have automated and optimized their supply chains — like Land Rover and other auto manufacturers. I’ll be sharing more of those lessons and perspectives from the 2015 State of the Software Supply Chain here this summer as well as expanding the discussion around “net innovation”. Stay tuned.

July 20, 2015
DevOps and Bi-Modal IT
Can a business have two competing approaches to delivery and operations of their information infrastructure and be successful? If you follow Gartner recommendations, it would seem that bi-modal IT is the way to go. Bi-modal IT has been explained many different ways, but, ultimately, comes down a failure in leadership; allowing a subset of IT to continue to operate in an non-optimal way because it’s too expensive and difficult to transform. This results in formation of a secondary subset of IT leveraging emerging trends that have demonstrated results to deliver modern information infrastructure faster with reliability.

If we examine from the 50,000 foot view, bi-modal IT actually makes a lot of sense. Disruptions are happening everywhere and businesses need to respond quickly. For many of these businesses IT can barely keep the core systems afloat let alone bear the burden for handling creative exploitation of massive data growth, enhanced customer experience and transformation to new compute platforms. A nimble team with the right skills can facilitate forward movement in these areas with little disruption to the core business. Moreover, they can establish a framework and approach that enables the core business to adopt when ready.

The issue is when we move down to 5,000 feet, this strategic viewpoint becomes a series of tactical actions that have the potential to spell long term disaster for the business. For example, the data that feeds many of the new digital applications is actually housed in systems being managed by the “traditional IT” group, which means that the “fast” team now is reliant upon the “traditional” team to enable access. Needless to say this is fraught with many hurdles; everything from heel-dragging due to animosity to lack of skills to lack of slack time needed to make the changes to support. While the stick might help with heel-dragging there’s little it can do to fix the lack of skills or slack time issue.

Technical and operational debts are finally coming home to roost for many of these businesses. Lack of attention to automation, documentation, training and staffing are now creating a Sophie’s Choice for many IT and business executives.
- Where do we invest?
- Do we spend time and money making what we have work better before addressing the needs to compete in a digital economy?
- If we do this, how far behind will we fall?
- If we follow a bi-modal IT strategy, do we risk losing critical resources needed to support our existing systems?
For years we’ve been telling the business that downsizing and budget cuts for IT is a mistake. The reduction in headcount and budget continued to squeeze existing resources to take on more responsibilities. As we now know through DevOps research the larger the quantity of work-in-progress being managed the more likely the increase in defects during the build process and the higher the likelihood of outages in production.

If there is a possibility to escape this conundrum, it most certainly is not bi-modal IT. Leveraging a nimble team to meet the needs of digital economy is fine, but they must work cooperatively with the traditional IT teams. Prioritization of backlog and increased transparency with the business is critical for moving forward. The business has to accept they can’t do everything they want that’s new and keep all that’s running today. Specialization must be reduced through standardization to reduce complexity and bottlenecks. Increased automation efforts will reduce time spent on manual operations and afford time to support other key efforts. Visualization and communications of work-in-progress across teams will minimize conflict and foster faster prioritization of critical efforts. Finally, moving to fewer changes with increased number of releases will enable the business greater opportunity to prioritize efforts.

In summary, one could say that the fix for IT isn’t going bi-modal, but adopting continuous delivery and lean IT approaches.
May 8, 2015
DevOps and Organizational Debt

I recently published a blog on the Perficient Digital Transformation blog entitled “Is Cloud & DevOps Factored Into Your Digital Transformation?” In that blog I discuss the concept of organizational debt and its impact on your DevOps adoption. Here’s what I had to say in that blog:

If technical debt addresses too little investment in upgrading, poor vendor selection, bad platform selection, etc. then, many businesses have also built up organizational debt. Organizational debt could be over regulation, too many low-value policies, to much middle management attempting to hold onto their area of control, and too much governance. If these are not revisited often to ensure that they are continually adding value for the cost of reducing throughput, then that organizational debt will continue to increase backlog.

Interestingly, there’s a lot written about technical debt, but very little has been written about organizational debt, which can have a significantly deeper impact on your ability to continuously deliver than your technical debt.

In working with clients on their DevOps initiatives, we map the current value stream for a given solution area. What becomes apparent from these value stream maps is that controls and policies have been derived within organizational silos with no attention given to how that will impact IT’s overall ability to be agile and deliver more quickly for the business. Moreover, the owners of these controls and policies seem reticent to remove these bottlenecks even after being shown how it impacts business agility.

Here’s an anecdote from one client when asked why they aren’t using snapshots to speed preparation for QA testing. The suggestion was made in response to the client stating it can take upwards of a week to prepare the QA environment for a single round of testing. Their current approach serializes all testing and believes they need to perform a complete regression test due to the resource contention issues. Hence, they cannot test small changes easily and they cannot test often due to these limitations.

“The main issue is that we don’t control the application or database environment, so we need to prepare a QA release and then have those groups perform their duties in preparation of the QA environment for testing.”

Upon hearing this my mind summoned up past experiences working conferences at major venues, such as the Jacob Javitz Center in NY, where you can’t plug in an extension cord or run an Ethernet cable yourself, but you need to use the Javitz Center’s electrician because of union rules.

Is this what IT has become a set of unofficial unions that hold the business in a death grip? This is clearly a case of organizational debt that needs to be addressed.

Of note, I’m not admonishing the various groups for instituting their controls and policies. I’m sure they have emerged due out of need due to past grievances, failures or as part of IT management practices. Regardless, they are now acting as a bottleneck in implementing proven practices for delivering fewer features more often, which have been proven to increase trust and facilitate better prioritization of work.

The result of this organizational debt is lead times have grown to four months or longer to release functionality that is directly related to key business initiatives by the business for increasing revenue.

In his book, “Kanban,” author David Anderson discusses his recipe for addressing bottlenecks and constraints with existing teams. One of his steps is ‘attack sources of variability to improve predictability’. However, this is his last step because, as he notes, “some types of variability require behavioral changes in order to reduce them. Asking people to change behavior is difficult!” And, so it goes, many businesses never address this key constraint because the hurdles of organizational debt have become so burdensome to overcome. It requires a high degree of trust and political capital along with executive sponsorship. As shown by the anecdote earlier, even if the constraint is directly impeding other business priorities it is not easy to overcome.

I believe DevOps practices can help organizations significantly reduce their technical debt. I have seen and read about organizations that have been able to rapidly reduce their lead times while still refactoring and removing backlog. However, the goal of DevOps is to improve the entire flow, not just development, and a good majority of the release aspects of applications will require organizational debt to be addressed to enable improvement across the entire continuous delivery pipeline.

February 6, 2015
Q&A: Gene Kim on the pain of accruing technical debt

Last week, we ran the story Paying Down Technical Debt. This week, we’re discussing that very topic with Gene Kim (@realgenekim). As a follower of DevOps trends, you probably are already aware of his most recent book project, The DevOps Cookbook, and most certainly his previous works, The Visible Ops Handbook and The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.

Gene also founded the security firm Tripwire and was the CTO there for 13 years. Gene has made a career of studying high performance companies and we’re please he took the time to share with us his thoughts on technical debt.

devopsy.kinsta.cloud: Tell us what has been keeping Gene Kim busy lately?

Gene: There is a lot going on. Right now, I’m very concentrated on the DevOps Enterprise Summit taking place in San Francisco October 21 through October 23. The summit will focus on Lean IT, DevOps, and Continuous Delivery. We’re pretty excited about it. We have 50 speakers from large enterprises such as GE, Disney, and Macy’s, and other incredible speakers.

devopsy.kinsta.cloud: Sounds like a great event, Gene. I’m curious to get your thoughts on technical debt. Some people argue that it doesn’t exist. Others contend that it does. Does technical debt exist, and if so how do you define it?

Gene: There is a great definition of technical debt that was given by Ward Cunningham: “Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite. … The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a standstill under the debt load of an unconsolidated implementation, object-oriented or otherwise.”

My simple take on that quote is that technical debt is what you feel the next time you want to make a change. It accrues from not only all the shortcuts you make in a project when it’s rushed, but even every time the developers don’t write an automated test – every time that you don’t do data code analysis. When you skip these things, the debt builds up every day. It’s basically all the variants, from the right way to do things versus the way we actually do things. The more of those that accumulate create this ever-increasing amount of technical debt.

devopsy.kinsta.cloud: What does technical debt commonly look like in an enterprise?

Gene: Randy Shoup gave a great talk on this, The Virtuous Cycle of Velocity: What I Learned About Going Fast at eBay and Google, at FlowCon San Francisco 2013, when he was CTO at KIXEYE. In that talk, Shoup provided the countermeasure to technical debt. He stated that any sort of thing that affected performance and quality, as defined by reliability of scaling, was treated as a priority zero defect.

Essentially, what he’s saying is whenever there’s a reliability or a scaling issue, it’s treated as a priority zero defect. In other words, it is as bad as a site down issue. What it means is that it’s like Toyota and when they have a serious issue, everyone stops work in order to swarm the problem until it is resolved. If they can contribute to helping the problem, they do. This is the opposite of what happens in most organizations, where it’s “never my problem” and when technical debt is allowed to accrue. Where you put problems on to the defect backlog, you put it into the security remediation queue, but it will never get done.

It goes there to die. It’s the landfill of good ideas that never get implemented. What Randy Shoup tells is that investing pays off. I would say that’s exactly the culture that enables high performance. High performance is where you’re getting 30 or fewer code deploys, very short lead times, high success rates when they deploy, and short lead time to prepare. That’s as true for operations as it is for information security.

So, technical debt is bad for ops; it’s bad for security. It’s also bad for development, because it slows down future flow.

Shoup nails the difference between the virtuous versus the vicious cycle. The virtuous cycle is described as one having a solid testing cycle. You have good testing so you have a solid foundation, which lets you be more confident. This lets you go faster and faster. The vicious cycle is you don’t have testing. You have a very shaky foundation. You’re afraid to make changes. And you go slower and slower.

devopsy.kinsta.cloud: In your experience of watching high performance organizations, what is the challenge of getting from the slide that discussed the vicious cycle and the technical debt world to one with the virtuous cycle? How do you implement what’s detailed on slide 13, I think it is?

Gene: It really is policy. We have to have a policy that elevates paying down technical debt and enables teams to focus on the nonfunctional requirements – a policy that one’s definition of done is whenever the developer says it’s done, then it’s done. It’s not done until there’s an automated test, an automatic deployment process, so that anyone can push the button and it goes through the test cycle and gets deployed into production.

Deliverables also have to operate in production as designed. That’s a good definition of done. That prevents us from going from sprint to sprint and accumulating technical debt behind us. I would say the countermeasure to technical debt is the right definition of done. It’s like in the agile philosophy quote: The definition of done is at the end of each sprint interval we have working and potentially shippable code.

I would add that it is integrated into chunks. There’s a checking depository. There’s automated testing that anybody can run before the code can be deployed. I think these are good countermeasures.

devopsy.kinsta.cloud: That’s excellent. Now, to get from the vicious to the virtuous cycle, I imagine it takes significant investment upfront before the payoff?

Gene: Unfortunately, this is one of those things where you go slower before you can go faster. But the benefits are so compelling. We know that high performers are doing 30 times more frequent code deployments, and they can do them 8,000 times faster. That’s 8,000 times faster. So the payoff is huge.

Think about it: If you’re not doing a deploy a day, that means that you are 100 to 1,000 times slower than high performers. High performers can execute a deployment in minutes or hours, whereas lower performers need weeks, months, or quarters. Of course, we have to take investment to go from weeks, months, and quarters to on-demand, executing within minutes. But the value of doing that is too high to ignore.

October 7, 2014