Tag: technical debt

  • Paying Down Technical Debt

    Paying Down Technical Debt

    When watching IT shops over the years it seems some, despite trying to cram two weeks worth of work into six days a week, never seem to move very far forward. And yet other teams seem to make tremendous strides in productivity and outcomes despite not working as many hours as the poorer performing organizations.

    Why is this? Like many aspects of life and business, it’s about working smarter, not harder. When it comes to enterprise IT that often means avoiding technical debt. The generation of demand for work to fix issues that should have, could have, been avoided had the work been completed properly the first time. “In an engineering context, I believe we are talking about the negative side of the tradeoff ledger,” says Kevin Behr, the chief science officer for the CXO and Board Advisory Practice at Praxis Flow. “Engineering inherently involves trade offs between what we would like to do and what we can actually do, so we often cut corners in the name of expediency,” Behr says.

    Aater Suleman, co-founder and CEO of Flux7 put it another way: “It’s when you have urgency that takes precedence over the right thing to do. In that rush, you’re not making decisions the right way, you’re just getting things done really quickly in the heat of the moment. And you have to go back and fix those later,” Suleman says.

    The high cost of technical debt
    How much technical debt does your organization have? How much latent work is hiding in rework and hurdles that need to be leaped because of shortcuts taken and ongoing bad decisions made in design, infrastructure, and operations. How much debt has been accumulated today, this week, month, or year? And, more importantly: how do you pay down that debt and not accumulate new debts in the future?

    According to Donnie Berkholz, an analyst at Redmonk, DevOps relates to technical debt in that DevOps is a more collaborative approach to software development and enables the people building and running the software to have a fuller understanding of all the places technical debt can live. “Sometimes it’s not directly in the code base but rather in the release tooling, or the production systems,” he says.

    An example Berkholz cites as a n example of the potential cost of technical debt is Knight Capital’s $460 million loss due to lack of documentation and automation of their server configuration.

    Paying debt down
    As a way to help reduce technical debt, Berkholz advises enterprises consider running “blameless postmortems” after outages or issues. “It’s about explicit acknowledgment of the facts as well as setting aside time to fix underlying problems without searching for blame,” Berkholz says. “Technical debt won’t fix itself; you need to set aside time and effort to pay it off, without worrying where it came from beyond lowering future debt as much as possible,” he says.

    And for avoiding technical debt during ongoing operations, Suleman advises enterprises continuously test their operations and applications. According to Suleman, most companies that have taken on significant amounts of technical debt actually don’t have adequate testing in place. That, by itself, is a challenge itself, but it turns out many organizations actually have another issue – some enterprises hold an unsubstantiated level of faith in their testing. “I’ve seen many companies where they think they have automated testing, and hence, they rely on it. But the testing is not exhaustive or complete enough to actually rely on,” says Suleman.

    For instance, Suleman explains, it’s rare that enterprises conduct performance tests on the code they commit. “Which is why performance testing provides a classic example regarding technical debt, because the enterprise can be doing everything right, and adhere to all of the general DevOps methodologies, and yet the application doesn’t run properly because they haven’t tested correctly,” he says.

    When balancing the IT ledger, think long term
    While it may sound straightforward enough to start testing applications and system changes continuously, and spending a little more time thinking through design issues upfront to avoid trouble later – continuous improvement efforts, such as trimming past technical debts incurred, don’t take root in organizations overnight. It takes time to build these processes in. Think marathon more than sprint. “Continuous Improvement is not as much for the impatient as it is for those willing to invest in steady, long-term activity,” says Behr.

  • Top 3 must-haves in Dev/Ops project handoffs, Part one

    Top 3 must-haves in Dev/Ops project handoffs, Part one

    There isn’t much that makes IT operations happier than receiving that brand new application neatly boxed, wrapped, tied with a bow, and outfitted with exhaustive instructions for every what-if, what-then, and what now scenario.

    Before you give the gift that keeps on giving in all the wrong ways, take these top three must-haves in developer to IT operations project handoffs to heart.

    Number One Must-Have: Define Normal and Abnormal Operational Conditions

    What does the development team consider normal operational parameters? At what point does the software break? How? If, giving an extreme example, at 20K users per server IT operations should expect ~50-70% CPU load but at 2K users per server they should not see 100% CPU load, they need to know this.

    “Development can outline similar behavior/symptom scenarios for infrastructure (CPU, MEM, DISK, NETWORK) and application-specific (USERS, SESSIONS, USE LATENCY, TIME BETWEEN ACTIONS) metrics,” says Ivo Vachkov, whose title is literally DevOps Engineer, Xi-Group Ltd (Bulgaria; www.xi-group.com). This is the kind of information that helps operations to recognize when a software component is in trouble.

    This may sound like obvious information for developers to include in an application handoff. “But many developers (if not most) will not be able to precisely define what is the normal operational condition for their software,” says Vachkov, lecturer in networking and programming security, New Bulgarian University (http://nbu.bg/entrance.php).

    IT operations needs to know about factors that affect normal operation including whether the software is CPU-bound, i.e., whether its performance is directly related to the processing power of the computer it runs on. Another such factor is whether the application is IO-bound, whether its performance is directly related to the throughput of the IO (MEM, DISK, NET) subsystem.

    Development should prep operations about other software behaviors and risks. If the application will degrade gracefully under increased load, they need to know that. If it will come to a screeching halt under load, they need to know that even more. If the software will crash when it slams up against an administrative/technology timeout, operations can deal with it, if they know ahead of time.

    “As operations people, we usually don’t expect too much. I would usually be happy to know that at 50k users per server the main process will crash. And this is OK, because I can work around it. I have the tools to prevent a breach of that limit. I would be happy to know that every night there is a batch processing that will hit the DB hard, instead of being awakened by an alarm about my DB being inaccessible,” says Vachkov.

    Number Two Must-Have: Share the Known/Collected Technical Debt

    In many cases the technical debt is not obvious, i.e., the documentation states that the component needs library X to work properly, but due to some particular reason, the developers used a specific version of the library that is not available in the default package repository of the OS. Someone needs to build it in manually and take additional care to keep the configuration in sync to make the code work. “Technical debt may have many faces, from bad/rigid pieces of code to bad algorithms to unspecified dependencies,” says Vachkov.

    Technical debt is also commonly omitted by the development team. “Pride, politics, other considerations are in play here. Many developers will just not keep track of technical debt. But this is crucial for proper operations. I want to know the shortcomings of the piece of code I’ll be dealing with,” says Vachkov.

    IT operations wants to know, for example, whether that key algorithm is O(n) instead of O(log n) because this will enable them to correlate metrics and explain what’s going on. These notations are part of Big-O Notation, which is used in mathematical and algorithmic analysis and generally represents a function that describes the upper limit of performance of a particular piece of code in relation to the number of input elements it needs to work with, according to Vachkov.

    Cliffhanger!

    This concludes part one of “Top 3 Must-Haves in Development/IT Operations Project Handoffs”. Tune in for part two when we take up with technical debt and Big-O Notation, learn more about why the Big-O is so important to operations and to corralling unwieldy software, and discover the third and final must-have in development/IT operations project handoffs. Don’t touch that mouse or touchscreen! Stay tuned to www.devopsy.kinsta.cloud.