devopsy.kinsta.cloud

Category: Enterprise DevOps

  • Visibility Drives Data-Driven Decision-Making

    Visibility Drives Data-Driven Decision-Making

    One of the biggest challenges of DevOps today is lack of visibility—especially shared visibility—into application delivery process and outcomes. A common data fabric allows all stakeholders to share visibility into DevOps processes, so they can more easily collaborate and communicate, use objective data to make decisions, and deliver better code faster.

    The DevOps Visibility Challenge

    A typical DevOps process relies on a loosely coupled tool chain. It includes separate tools for discrete functions including project management, source code control, provisioning and configuration, test execution and workflow automation. A tool chain approach adds much value, but becomes complex quickly. Also, it makes it difficult for stakeholders to get accurate insight into application delivery.

    This challenge is especially endemic to large and distributed enterprises. It’s even more difficult when there are thousands of developers and dozens of data centers across multiple geographies. Having duplicate tools, specialized teams, varying processes, distributed locations and multiple business units makes it even harder.

    As a result of this poor visibility into application delivery, stakeholders from business and IT:

    • Cannot manage the velocity of feature delivery. They cannot detect and fix slowdowns or bottlenecks in the process or allocate and prioritize resources. They also cannot plan for downstream activities or triage slowdowns and stoppages in the software development life cycle.
    • Cannot make objective decisions on quality. They risk releasing error-ridden software, slowing down response times, introducing security vulnerabilities, oversaturating infrastructure, damaging reputation and adding unnecessary costs.
    • Cannot properly collaborate on detection, diagnosis and remediation of production problems. They also cannot share an understanding of the impact of code changes on business key performance indicators. Customer satisfaction, customer signups, cart fulfillment rates and revenue all could be affected.

    Data Mining Enables Visibility

    To overcome these obstacles, successful DevOps teams are using objective metrics mined from machine data. That helps them gain insight into the end-to-end delivery cycle, from idea to release.

    Collecting, correlating and analyzing machine data from multiple sources gives dev and ops (and others) continuous insight. They can see the velocity, quality and business impact of application delivery. It provides individual contributors the data they need to be responsible for their own work, accountable to each other and aligned with business objectives. It enables managers to make data-driven decisions about workload prioritization, resourcing, scheduling, performance and value delivery.

    How to Get to Data-Driven DevOps

    Some of the key actions you can take to start on a journey of data-driven DevOps include:

    Measure your speed of application delivery: The machine data generated by your DevOps tool chain provides a wealth of information about all phases of the development cycle. It shows what is in development, close to release, in backlog and slowing you down. It shows which stages in the process are failing and what teams are most efficient. Consider mining and sharing velocity-related measures such as commit rates, sprint times and “idea to cash” time, to name a few.

    Measure the quality of your application code: Even if you are already using testing tools to understand release quality, there is more to quality than test coverage. Again, choosing the right KPIs is critical, as is sharing those measurements throughout your team(s). Some other key metrics may include the number and importance of defects, defect variance from release to release (or from team to team) and build/integration/deployment failures.

    Measure the business impact of application code: All too often organizations measure activity, but fail to align application delivery with business goals. As one customer recently told me, “Shipping crappy releases faster won’t help.” Business stakeholders especially are looking for business KPIs such as:

    • user signups or cancellations
    • customer satisfaction
    • cart fulfillment or abandonment
    • time on site
    • transaction failure rate
    • sales volume
    • application revenue
    • app downloads
    • offer open rates
    • customer satisfaction; and
    • customer acquisition costs

    Measure the human impact of DevOps: DevOps is mainly about people, so you must find ways to measure culture change, too. You can mine various applications for objective metrics such as number of sick days, “work from home” days and variance in team and individual productivity. However, machine data will not provide all the relevant KPIs; you will need subjective measures too, such as from “stay interviews,” feedback forms or surveys.

    Data-Driven DevOps with a Common Data Fabric

    A data-driven DevOps approach more effectively drives continuous improvement and higher levels of agility. It also helps achieve greater collaboration, improved security and compliance, and better alignment with business KPIs. These all occur while enabling rapid iteration and innovation.

    A common data fabric provides the objective measurement and shared visibility critical for a data-driven DevOps approach. With comprehensive and continuous visibility into key performance measures, provided through a shared data fabric, DevOps teams can:

    • Isolate “waste,” detect and correct slowdowns, and deliver applications faster
    • Correlate test and QA outcomes to find more problems sooner and improve code quality
    • React faster to detect and address problems that do get through to production; and
    • Use real-time insight to measure business impact and iterate faster on good change

     

  • Infrastructure-Testing Dominoes

    Infrastructure-Testing Dominoes

    Understanding the artifacts of a given technology component is critical to an enterprise-class DevOps service offering. Second-most critical is understanding the dependencies between given components. Testing helps illuminate those dependencies.

    For example, let’s assume I have a server running the previous version of the Windows operating system (OS) that I now want to upgrade to the latest version. The previous deployment package of this version of the OS would be an artifact. It should also include all of the test automation I run to validate that version of the OS is working after I install it. The artifact also might include configuration files I deploy when I install that version of the OS on this type of server in this class of environment. Having these bits of automation bundled together allows me to do a successful install of my OS (assuming no hardware errors) on perhaps many servers.

    As I create a new artifact for the latest Windows OS, I keep in mind all the things I had in my last version. I might include a newer config file and perhaps a set of tests (albeit new ones) to make sure it is working when I am done. Bundling all these bits gives me a new Windows OS artifact or package that I can use over and over on this kind of server in this class of environment. The idea is that I am creating a single kind of artifact or package to cover this kind of technology—in this case the latest version of the Windows OS operating system. All my artifacts should be version-controlled. So when I’m done I have an artifact for the previous Windows OS and one for the latest version. Each artifact version has embedded or included testing (pointers to, or discreet automation) to verify the validity of the install, and perhaps a little more to prove the OS works.

    Within my DevOps tooling, I likely have some sort of artifact management system. It may not be as formal as a full-on configuration management database (CMDB) system, but whichever tool I use will have some sort of database to manage my artifacts or packages by version. How long I retain these artifacts will be defined in my retention policies, but likely I will retain every version in use at my company today. This is step one, and it forms the foundation for my next effort in cataloging the dependencies that exist between artifacts.

    Enter the Spider’s Web

    Under the scenario I have described, when I deploy either the previous OS or the latest one, I would run a series of tests specific to that version to ensure the OS is working properly. As an infrastructure engineer, I tend to think my job is done at this point. But it isn’t. To truly validate that my OS is working, I really need to know what is running on the server I just deployed to. If my server was bare or new, then I am really done. But if my server has middleware running on it, then the job is not finished until I re-deploy the right version of the middleware on top of my new OS and verify it is still working properly.

    To pull this off, I will need a catalog of the technologies that depend on each other. The OS example is kind of simple, because it lives at the bottom of the stack where a server is concerned. But on top of the OS might reside a middleware platform (perhaps messaging or a database). On top of the middleware might reside an application or perhaps several applications, depending on the horsepower of the server in question. Upgrading the OS component implies I am redeploying the middleware layer and then the application layer on top of that.

    Enter full-stack engineering. It takes an application-centric view of the world, but this implies a single application-centric view. It looks at one app from top to bottom and treats the entire stack as a single version-controlled artifact. But if you have legacy environments that run multiple applications on the same physical box, perhaps on the same virtual instance, you have the makings of a spider web of dependencies. There are no full-stack solutions (to my knowledge) that have figured out how to incorporate multiple applications at the top of the same stack, varying middleware in the middle and “n” number of virtual instances that ride on one OS on one physical box. Instead, each piece of the technology tends to be treated separately. Thus the need for some sort of catalog that tracks the interdependencies between them all.

    Starting the Domino Testing

    So once I have discreet testing that is version-controlled (or associated with) each type of technology component, I will have to re-execute those tests if anything “under” me undergoes a change. For example, if the version of my WebSphere is upgraded, beyond just the core WebSphere testing that occurs, I really should execute the tests associated with each application to ensure “that app” is still working after the upgrade. You could apply this premise to a database layer (say, upgrading the current version of Oracle, as an example). To really know if the database upgrade was successful, you should run the testing of each dependent app and ensure “that app” still works after the upgrade. The results are likely to be surprising.

    To execute this kind of domino testing, even if you have the web of dependencies defined and the artifacts or packages managed, you can take one of two approaches: redeploy everything, although this method may not actually kick off tests if it detects the versions did not change from the last deployment (smart overwrite), or set up discreet testing jobs within your release orchestration software to execute the entire web of dependencies and track the results. In either case, you are going to need human oversight on whether it is better to keep the new version or roll back to the old one. Only a human can prioritize the impact of what might seem small or catastrophic (based on a human perspective) of a given app or series of apps.

    Impact to the Bottom Line

    True quality can be assessed only through testing. Part of breaking down silo-thinking is taking a holistic view of the technology your organization supports. Just because the OS support engineer’s testing passes does not mean the upgrade was a success. For an upgrade to be a success, the tests of everyone else who relies on that technology underpinning must pass to prove it worked. The entire team needs validation when the underlying infrastructure undergoes change. The dominoes of tests must pass from end to end. Only then can true quality or true success be measured.

    Of course, most engineers are reluctant to have “app software” guys influence whether they did their own jobs right. After all, most software quality problems are in the code, not due to infrastructure errors. However, if the applications upon which your business relies cannot work, no matter what the reason, after an infrastructure upgrade, it is a problem for the entire business, not just the software guys. To tear down the silos, we need to look at impacting events for change across boundaries and drive toward overall results, not just departmental ones. When this occurs, the cost of quality is significantly reduced.

    To continue the conversation, feel free to contact me.

  • 3 Rules for Successful SAFe Adoption

    3 Rules for Successful SAFe Adoption

    Startups and enterprises alike are redefining—some say disrupting—markets, industries and customer experiences by delivering innovative ideas at a rapid pace and responding to feedback. Often they fail, but they fail quickly and adjust. You know the ones: Spotify, Etsy, Netflix, Airbnb. How do they do it? What can we learn from them that will work in our own organizations? What is different? In a nutshell, all of these successful organizations innovate completely, not only in terms of what they do, but also how they do it. And the common thread is the application of lean and agile practices to improve their culture and deliver innovation to customers faster than the rest of us.

    Today’s business imperative is to be the disruptor and more and more organizations are turning to enterprise scaled agile as a set of practices to help lead transformations to achieve that goal. The Scaled Agile Framework (SAFe) is the industry-leading enterprise scaled agile framework with a proven track record in thousands of organizations for several reasons, not the least of which is the incredible breadth of definition and collateral so that you just don’t have to make this stuff up yourself. Nevertheless, there is not an one-size-fits-all framework and SAFe is no exception. But there are three key concepts prescribed by SAFe that, if you do nothing else, will help you be successful in your evolution toward becoming that disruptor.

    Go SAFe or Go Home

    1. Apply Lean Principles at an Enterprise level
    You’ve likely heard the term “MVP,” or minimum viable product. This is about investing “just enough” to deliver a capability—a solution, product or application—and get feedback. It is about avoiding the pitfalls of over-engineering. This is perfect embodiment of lean thinking. By focusing on investing only as much as necessary to get feedback quickly (and doing this continuously) you not only reduce or eliminate wasted resources but also speed time to feedback. Lean is also about the notion that “capacity rules”—in other words, only invest as you have capacity to follow through. Otherwise, what’s the point?

    2. Apply Economic Thinking
    Traditional methodologies typically focus on priority to rank at all levels, from business initiatives and projects down to team-level work that delivers changes into products and applications. In the agile world, story points (or complexity) also might be used. So what’s the problem with this? First, it is only a single dimension. What about business or customer value? What about cost of delivery or reducing risks or enabling market opportunities? Is timing a factor? All of these aspects should be considered when ranking because, at the end of the day, organizations are looking for the “biggest bang for the buck.” By applying economic thinking, you are maximizing the business’s ability to deliver the most value at the lowest cost in a timely manner.

    3. Focus on Delivery of Value
    I’ve often heard that revenue generation is the primary goal when organizations look to speed delivery cycle times. In fact, the goal should be delighting the customer; revenue is a bonus, a side effect of happy customers who use your stuff and pay for it! A focus on improving feature delivery is too narrow. What you really want to do is improve time to value. That means articulating what value means, associating the notion of value with features that are delivered, assessing that value and then tracking so you have insight into your organization’s ability to deliver value and get better at it.

    SAFe embraces all of these concepts through artifacts and activities anchored in lean and agile practices. Try it, you might be surprised at how impactful this can be in improving your ability to innovate and be the disruptor instead of being disrupted.

    Read how a typical enterprise dipped its toe in SAFe water to drive delivery of a critical project, and be sure to visit the SAFe landing page to learn more.

    About the Author/ Amy Silberbauer

    Amy SilberbauerAmy is a Solution Architect charged with the definition and delivery of the Enterprise Scaled Agile and IBM DevOps Plan solutions and strategy for the IBM Watson IoT line of business. She is a recognized subject matter expert on scaled agile and software development lifecycle solutions, including enterprise modernization, SOA and collaborative DevOps. She has been with IBM for 29 years and has two decades of software development experience as an engineer, architect and manager. She is a certified SAFe® Program Consultant experienced in leading and consulting on multiple internal and external SAFe (Scaled Agile Framework®) transformations.

    Connect with Amy on LinkedIn.

  • What Can Digital Transformation Do For You?

    What Can Digital Transformation Do For You?

    Contrary to what the massive articles around digital transformation suggest, your business lines’ primary concern isn’t the amazing technology innovations that pop up every day but how well they’re going to deal with your industry’s disruption. Something is sure, your IT organization’s digital transformation cannot be narrowed to technology changes; that’s a simplistic vision which ignores the disruptive impacts on businesses, particularly on the IT function.

    Howard King, writing in “What is Digital Transformation?” published in The Guardian, makes it clear, “Transformation is a whole scale change to the foundational components of a business: from its operating model to its infrastructure. What it sells, to whom and how it goes to market.” He’s right, the business-oriented vision of IT demands a transformation of your IT organization including its infrastructure, applications delivery approach and operating model, and DevOps increasingly is cited as the foundation of the new IT.

    This article shares my experience and lessons learned implementing DevOps as the foundation of digital IT organizations.

    Top Priority: Dealing with Your Industry Disruption

    The belief that implementing a continuous delivery infrastructure is all your business need to survive digital disruptions is a marketing hype DevOps practitioners have grown accustomed to. It’s a myth that prevents you from making informed digital transformation decisions. Emphasizing the technology at the expense of your industry’s disruption forces you to make technology change prematurely; your priority actually should be to figure out how to deal with the disruptions hitting your industry.

    Ask yourself this: What’s the interest in investing in scattered continuous delivery infrastructure implementations, when your favorite vendors and consultants aren’t able to clearly address these concerns:

    • How can you restructure your business value stream to make it lean and agile so that the right services are timely delivered to the right market segments?
    • How can you integrate DevOps, IT as a service, cloud, Internet of Things and big data into a coherent and consistent business capability so that the right services are timely delivered to the right market segments?
    • What’s the architecture of your future information system, including your virtual data center and application delivery infrastructure?
    • What will your role be as the CIO? How will the relationships with your business lines and vendors be managed?

    Make no mistake about it, these are your business line concerns of the moment. Unless you address them, Chef, Puppet, CodeDeploy or any other software application alone won’t help to keep up the pace imposed by Uber, Apple, Google and the like. You’ll need to bring together your business and IT staff in a lean and agile capability, which demands a re-engineering of your IT structure, processes, practices and governance.

    DevOps is the Foundation of Your Digital IT Organization

    Let’s remove the ambiguity: Infrastructure as code isn’t DevOps. The increasing confusion of DevOps with continuous delivery infrastructure is slowly but surely killing it—it’s even disqualifying it as a competitive advantage. That vision focused on tools has nothing to do with the logic underneath DevOps.

    As I’ve been making clients happy with it, DevOps is primarily an application delivery model that seeks to make businesses responsive to market opportunities through accelerating added-value applications delivery and with the goal of continuously generating value. It’s based on three principles helping businesses survive digital disruptions, then grow and prosper:

    • Using continuous delivering infrastructure to speed application delivery doesn’t make sense unless tangible business benefits are generated.
    • Applications are the means by which value is brought to customers and business benefits are brought in.
    • Agile principles across your software development life cycle makes your value stream lean and agile which in turn results in continuous value.
    DevOps Delivery Model - Philippe Abdoulaye
    DevOps Delivery Model – (Credit: Philippe Abdoulaye – ITaaS Now LLC)

    The delivery model uses three levers to implement the above principles and establish DevOps as the foundation of your digital IT, they include:

    • The rules of the game, which are agreed-upon principles, practices, roles and responsibilities and tools institutionalized to ensure agility and accelerated applications delivery;
    • The operating model, which is the lean and agile work environment resulting from the application of the rules of the game; and
    • The continuous delivery infrastructure, which is a platform that leverages infrastructure-as-code solutions to accelerate applications development and deployment.

    The Digital IT Organization: Approach, Principles, Tools

    Success in IT organization transformations often come more from a stream of small wins than from a one-time flood. That’s what I call incremental transformation, illustrated in the following digital five-stage transformation journey:

    The digital IT organization implementation process
    The transformation journey – (Credit: Philippe Abdoulaye – ITaaS Now LLC)

    The following recommendations definitely will make you instrumental to your IT transformation and help your business meet the digital economy’s competitive challenges.

    Strategize

    A CIO who’s expected to help business lines survive their industry’s disruption, grow and prosper first must clarify their business objectives; estimate potential gains, losses, and risks; and create a consensus on the DevOps delivery model to implement in terms of high-level architecture and implementation timing. That’s the purpose of the Strategize stage.

    Never skip this stage—it’s where the executive buy-in is obtained and the vision of the future IT capability is developed. Make sure the following is accomplished:

    • Creating an executive task force co-led by the CIO and a business representative;
    • Scheduling one to three workshops addressing your transformation business case, transformation road map, high-level blueprint of the future IT capability and the DevOps adoption plan;
    • Designating a project manager; and
    • Communication by task force members regarding the whys and wherefores of the transformation initiative.

    Plan

    Digital transformation involves many functions within your business in the form of a cross-functional team: the concerned business and IT staff. Your responsibility as the CIO is to bring the voice of the business across the transformation journey, from business objectives clarification, to design and implementation, to pilot experiment, to deployment.

    Again, do not overlook this stage—it’s about translating the effort into team structure, implementation schedules and operational and strategic reviews. It’s where the risks likely to hamper projected objectives are anticipated. Make sure workshops involving the business and IT are set up to address the following:

    • Team structure and roles and responsibilities;
    • Transformation road map steps translated into operational timelines associated along with expected outcomes; and
    • The project’s operational and strategic reviews, including a risk management plan.

    Design and Implementation

    The third stage is when the three dimensions of the digital IT organization are designed and implemented. They include the DevOps rules of the game, the operating model and the continuous delivery infrastructure.

    Never improvise this stage—use a proven approach to address the dimensions. Favor a collaborative approach that includes workshops or brainstorming sessions with the concerned business and IT staff.

    In the rules of the game brainstorming sessions, make sure the following is addressed:

    • Organization and operational dysfunctions preventing cross-functional collaboration, rapid problem-solving and decision-making;
    • Recommendations to the dysfunctions in terms of agile processes, practices, roles and responsibilities and tools;
    • Recommendations mapping to the appropriate DevOps principles, practices, roles and responsibilities and governance structure; and
    • The IT organization’s DevOps charter, thought to accelerate the rules of the game and the operating model adoption.

    In the continuous delivery infrastructure workshops make sure the following is addressed:

    • A delivery pipeline, including steps and for each of them what follows:
      • Required staff, skills and background
      • Implementation, testing and deployment agile processes and practices
      • Implementation, testing and deployment tools, such as Chef, Jenkins or CodePipeline

    Experiment

    For this stage, set up a pilot experiment involving, in a small-scale real-business setting, a panel of business and IT staff to evaluate the likelihood of the project benefits.

    Neither skip nor overlook that step, it’s where improvements are identified. Throughout the pilot project, make sure the following is in place:

    • Executive task force monitoring the pilot project through bi-weekly reviews;
    • Concerned business lines and IT business lines are involved;
    • Training campaigns are organized around the DevOps charter, the operating model and the continuous delivery infrastructure;
    • Strategic management tools, such as balanced scorecards to monitor progress, risks and achievements; and
    • Measurement of the impacts of the operating model and the continuous delivery infrastructure against the projected benefits.

    Deploy

    This final stage is a Go-No-Go milestone in which your task force, based on the pilot’s outcome, decides whether to deploy your digital IT capability at the scale of the company. Throughout the deployment:

    • Capitalize on the benefits generated by the pilot experiment to launch an information campaign emphasizing the benefits for the company and for the staff; and
    • Train business and IT staff on the DevOps charter, the operating model and the continuous delivery infrastructure.

    Wrapping up

    Today’s DevOps practices focused on continuous delivery infrastructure aren’t likely to help businesses grow and prosper in the digital economy. Thought leader Joe McKendrick in, “The road to digital bliss is paved with service thinking and DevOps,” confirms that fact.

    The approach discussed in this article are detailed in my recent book, written to help make IT leaders instrumental to their business digital transformation:

    Rethink Your IT with DevOps

    If you’re considering a DevOps-focused IT transformation, what will be your first steps? I’d love to hear your thoughts, let’s meet @TheFutureOfYourIT.

  • Dashboards from Hell

    Dashboards from Hell

    Even organizations with a mature DevOps services model inevitably will hear from executive sponsors about the “need” for a dashboard for DevOps information. Organizations just starting to embrace DevOps won’t have the time to construct one yet, but over time will reach the same conclusion. The need is real, and the value is significant—it is how you get there that makes the difference. Vendors are keen to promote the idea that they already have a dashboard built into their respective products. And this is partially true.

    But the problem with dashboards is not a matter of information, it is a matter of perspective and scope. Ultimately, all of the information an audience wants can be found in the various DevOps tooling used in the continuum of services your organization created. However, locating some of it takes profound knowledge of how each tool operates, adequate security profiles and an elephant’s memory as you click on multiple windows to try to correlate the data you ultimately are seeking across the various tools.

    Before you know it, executives and perhaps even practitioners are calling for a consolidated dashboard that would present this information cleanly. The danger here, of course, is an attempt to consolidate every perspective from every individual audience into a single dashboard for your company. Instead, it is wiser to begin such an effort by breaking down the number of audiences into categories (by need, perhaps) and examining what is already available and what is missing to make a dashboard project effective, and not one originating from hell.

    Dashboards: The Practitioner Perspective

    There is a keen difference in what a typical engineer wants to see as it relates to DevOps from what the other audiences will be looking for: namely, detail. If you consider this part of a dashboard effort the foundational section (i.e. providing the right detail information to the engineers who ask), the remaining higher-level presentations can be built on top of this data. Most engineers are content to just “look into the given tool” to get what they want. But sometimes consolidating this information into a dashboard begins to make more sense.

    Ideally, your company has only one build tool and, again ideally, only one deploy tool. If you have multiple types of both kinds of tooling in these disciplines, you would be better served to pick a winner there than try to use a dashboard effort to consolidate build and deployment data across multiple build and deployment tools. Allowing multiple kinds of build and deployment tools across an enterprise usually winds up in the demand for multiple dashboards from multiple audiences, and before you know it, the dashboard project from Hades once again has infested your organization. A clear winner in build, deployment and release orchestration cuts out this kind of problem.

    Most of the time the need for a dashboard emerges when users are trying to see information in a way the native tooling did not plan for. For example, I want to look at a given server, or class of servers, and a list of everything that is on them (by version). Or as an example, I want to look at these technology components of this type and version and find out every server in my entire data center that they may have been deployed to.

    Some vendors offer internal reporting capability intended to meet these needs. However, using those reporting features should be labeled with “caveat emptor” signage. Typically, vendors apply their reporting directly against online or production data within their respective tools. So an engineer presses the report button on his build tool, and the build reporting function begins to look at its own data from a production viewpoint, instead of from an offline reporting copy of the data. When you start running reports against “live” tools, you get slowdowns, sometimes significant slowdowns.

    To avoid this issue, start with determining what data the engineers need, how often they need it and to where you can copy this information (to form the basis of your dashboard system). You also may wish to examine how the base data is inherently constructed and join some of it where it makes sense, so you are not issuing very large queries joining key data from many tiny tables together before anything sensible emerges for your new dashboard system. Speed of presentation will matter still, even though the data becomes only near-real-time instead of real-time.

    Dashboards: The Business Perspective

    Once you have ascertained the detailed information to satisfy your engineering or practitioner community, it usually needs to be married to your business segmentation to meet the needs of this audience perspective. For example, I may have 30 servers that I use regularly in my e-commerce products or line of business division. My engineers may care less about this designation, as they only care which servers are content servers or database servers or application servers. But my e-commerce business teams look at this information differently. They only want to see “their” servers, not every server in my company. So to accommodate the next perspective audience, I need a way to marry servers and technologies to the product lines or lines of business my company understands.

    This may require an outside or additional database that tracks the raw server name and marries it with the business unit that cares about it. It may also track the application names the same ways, marrying them with the appropriate business unit. But this marriage now offers the ability to pull reports (or dashboard screens) by line of business—it is nearly certain your basic build and deploy tools weren’t designed to show information in this way. This kind of business segmentation, or filtering (in some tools), provides an entirely new way to examine the data. The need to drill down to the details may still exist, but a picture of change relative to a product line or business line becomes a high value-add for the business community who looks at DevOps data.

    The business community looks at DevOps information to help it make decisions based on the state of change at any point in time. E-commerce customers, for example, might get special company notifications of upcoming changes or improvements in their capabilities, as the business community is made aware, and stays current with the planned release events and quality of testing underway. Or those same notifications might be delayed based on a quick look at the same information presented the same way.

    The point is, business users are an important community or audience of DevOps information, and the lens they use to examine data is from a product or line of business construct rather than the raw data aggregated by a practitioner.

    Dashboards: The Executive Perspective

    Executives, no matter the level, usually are more concerned about trends in the information than point-in-time data. For example, a given executive may want to know how many deployments per month are happening out of the development class environment for a given line of business. If you have constructed your dashboard with the architecture laid out above, you should be able to provide this kind of information. The discreet data is stored offline (copied from the build and deploy and other DevOps tools themselves), then married with the business segmentation data. Now it can be presented in trend-over-time manners that executives invariably want.

    It is much more likely that a business team member will use the drill-down features of your dashboard than your typical exec. But having it available up to the executive layer is a huge win when demonstrating the detail of what is going on where and potentially framing the why. The other significant win of having a single dashboard for presenting trending data is the consistency in the presentation across the enterprise. Consider for a moment how many Excel spreadsheets are sure to spring up around the company when execs task their people to produce reports that aren’t available in a dashboard at their fingertips. Each minion of the exec likely will grasp certain data in unique ways, often resulting in bar charts that never seem to line up.

    Having one dashboard—or, rather, one method of presentation of the same data—eliminates having multiple unique minion versions that lack context and often misrepresent the real story. In addition, without having a central dashboard you construct and control, minions will pull data requests against the production copies of build and deploy and release tooling, perhaps slowing these tools down to a crawl while data is gathered. The act of looking at a thing will certainly change the speed of that thing, in this instance.

    Impact to the Bottom Line

    Meeting the 80/20 needs of your primary audiences of DevOps data can be done without tying up programming resources forever. Once you have delivered a core dashboard system with mainline capabilities, your executive or business teams can decide how much more resources they are willing to fund to continue to update and enhance the dashboard. As long as they fund it, you can dedicate the resources you need to keep making it better (see agile). But for most shops, this kind of dashboard is a three-to-six-month effort from end to end.

    Once the dashboard is up and running, it becomes “the” tool in the arsenal of transparency for DevOps services. Granting access to this kind of tool (i.e., setting up user IDs) is significantly simpler than doing it in the multiple of underlying tools you have to administer without it. The cost of administration in a large firm can be substantial if a dashboard is not present. Imagine having to set up a typical user ID in your build tool, deploy tool, configuration management database, testing tool(s), release orchestration tool and monitoring tool—and that’s just to start. Within these types of tools that provide a real service or function are secondary concerns such as security or permission profiles, group memberships, etc. A dashboard only needs a valid company ID, and you are ready to “see” change in motion via DevOps.

    The work effort typical executives “task” their minions to produce reports is usually no small effort, either. While this work effort is rarely captured or understood by the executives who request it, it definitively puts a drain on company resources (both man and machine). A functional dashboard can eliminate this work nearly altogether, and produce a consistent work-product output. Your DevOps strategies should be created to reduce costs and facilitate innovation at every turn. If you need help feel free to contact me, I am still looking at the moment 🙂 .

    To continue the conversation, feel free to contact me.

  • 5 Development Trends Reshaping Testing

    5 Development Trends Reshaping Testing

    There’s no shortage of new development trends, but virtual reality, containers, big data, hyperconnected apps and IoT are affecting testing the most today.

    If you like change and challenge, it’s a fun time to work in software. Some technology developments, such as cloud computing, have been gaining speed slowly over many years. That steady progression has meant that developers can stay on top of trends easily and be informed about market changes. Other areas, such as containers, have come to fruition much faster without as much notice. Development teams have had to scramble to stay ahead of these trends, hoping to capitalize on a their efficiency gains while staying ahead of the curve. Here’s a look at five important software trends which are having a profound effect on testing.

    Containers

    Docker made containerization cool and exciting in 2013, even though containers have arguably been around for many years, if not decades. Developers love containers because the technology makes it much easier to set up and install software in different environments, such as in the public cloud or in a private data center. Containers wrap up all of the infrastructure components that otherwise would be installed separately into one easily portable package with a common standard. And since multiple containers can run on the same hardware without creating problems, you can do more with less infrastructure. Typically, containers can move between different environments with relative ease—a nice benefit for multicloud infrastructures. Containers have evolved quickly from an experimental platform to a mainstream infrastructure technology, with many major businesses taking advantage of containers to deliver their product at scale.

    Testing considerations: It’s hard to find much at fault for containers when it comes to facilitating software testing. Because of the ease in setting up test environments and moving code from one environment to another during the production cycle, testers can work much faster. Before containers, there was a lot more waiting around by both developers and testers. When it takes two weeks to set up an environment for a testing cycle that took two hours, teams would tend toward less frequent releases or would do less testing. Now, it is a matter of minutes to set up a testing environment. This is great for companies and development teams that value testing and want to test more frequently with greater access to a variety of environments.

    Another boon with containers is stability. In the past, bugs sometimes appeared due to configuration errors from setting up dev/test environments. Given the plug-and-play nature of containers, there’s little if any concern about the proper configurations being completed before a testing environment can be used. Security has been an ongoing concern with containers, especially in highly regulated sectors such as health care, government and financial services. In those cases, companies may have to support two versions of the code—one in a containerized platform and one in a more traditional virtual or bare-metal environment. We believe this will be a short-term situation as containerization technology matures and becomes adopted across all industries.

    Big Data

    Whether you want to use this vague and overhyped term or not, big data is simply a new way to refer to larger than normal data sets. Big data became a big trend as the hardware and software to store and manage it got a lot more powerful and cheaper, and the cloud shift helped to centralize data sets into one location. Some large companies and research organizations are seeing interesting results from big data projects already, whether in more targeted drug development (precision medicine) or just-in-time environmental data for farmers. Now, every Tom, Dick and Harry is trying to get into the big data market, which means there are too many solutions to count. A natural weeding-out process will occur over the next few years, as the prominent big data players and NoSQL standards converge into two or three leading providers.

    Testing considerations: Big data applications bring value through the way they manage and/or analyze data, not so much through the user interface. Therefore, testing for big data applications is going to require more technical, database-oriented skills. Those skills are also different from the typical relational database skills of the past, given the fact that big data platforms typically are designed to manage structured and unstructured data. These are open source tools and systems such as NoSQL and Hadoop, along with large vendor systems such as SAP HANA. Testers in this field will need to be constantly adapting learning these new technologies.

    Virtual Reality

    Investors have funneled $8.8 billion into the sector since 2012, according to SuperData Research. Google, Facebook, Microsoft, Samsung and HTC are all in the game. The market is segmented between VR, augmented reality (AR) and mixed reality (MR) and consists of both traditional (PC/console) and mobile device applications. VR is a fully immersive, fully virtual experience that mimics reality, while AR and MR are different variations of blending real and virtual environments together. VR applications aren’t just for gaming anymore: We now are seeing the technology being used for product design and collaboration, tourism/travel and entertainment, including film.

    Testing considerations: VR is an emerging technology still, with many different types of platforms, no clear leaders and no real standards in place. That makes it more difficult for developers and testers to make strategic decisions around technology. As well, there aren’t any great tools for automated testing in these environments yet. Therefore, the need for manual and real-world testing will add time and cost to the project. Yet, investing in doing this right is critical for ensuring optimal user experience. In virtual reality environments, it’s difficult to simulate experiences, such as testing for motion sickness, without actually playing with the real devices. This is especially challenging with the newer mobile platforms, such as Google Cardboard, which incorporate a viewing device that transforms the mobile phone into a VR headset. That experience simply can’t be simulated accurately on a PC, a common testing strategy for other mobile apps. Companies testing mobile VR apps may need to invest in the purchase of dozens or hundreds of different mobile devices to gain an accurate picture of how the application performs for most users.

    Hyperconnected Apps

    Web-based and consumer apps comprise the bulk of hyperconnected apps, characterized by having deep, often real-time integration with other applications. Instagram and Facebook, Gmail and Google Calendar, Salesforce and their wealth of apps—these are just a few examples of hyperconnectivity. The advent of REST APIs has facilitated far easier and cheaper integration so any company can build a platform of third-party software in ways they couldn’t have imagined five or 10 years ago. Similarly, with the explosion of the app economy, including open source software, it doesn’t make good economic sense to write 100 percent of code in your application any longer. Product value is increasing the result of how developers can patch together different technologies in innovative ways.

    Testing considerations: Given the importance of APIs in hyperconnected applications, a primary testing consideration will be ensuring the performance validity of those connection points. It’s imperative for both developers and testers to be aware of API update schedules and to test those updates immediately for any negative impact on the application. Beyond API testing, load and performance testing are also important considerations for hyperconnected applications, which typically involve heavy and/or unpredictable transaction processing. Overall, these types of testing are much more technical in nature than manual UI testing and will require testers to expand their skill set to optimize value.

    Internet of Things (IoT)

    The interest in (and hype) for the IoT sector rivals the largest IT trends, such as big data. Initial applications have focused on connecting devices, such as home lighting and thermostat systems, which use sensors to stream data to smartphones. Over time, we will see more device-to-device connections such as a thermostat collecting data from a smoke detector to ensure that the house isn’t on fire if the temperature rises dramatically. Products such as Amazon Echo and Google Nest might be at the heart of many new consumer-based IoT applications, but there is also ample potential in sectors such as utilities, health care, industrial equipment and government (a.k.a. smart city initiatives).

    Testing considerations: As IoT applications mature, connectivity complexity will grow. IoT applications can involve many components: the hardware of the smart device or product and the software that controls it, along with the user applications that interact with the product. When troubleshooting an issue or vulnerability, a tester must investigate the applications that collect, analyze and display the IoT data, the various network connections across which that data travels, the IoT appliance’s hardware and its software. Hardware testing must incorporate typical scenarios such as what happens to the user experience when a device starts to run low on battery or goes offline. Testers will be most valuable when they have experience in both hardware and software testing; alternatively, an organization can have two teams of testers but they must be in very close contact. IoT apps should also be configured to collect and deliver more alerts and logging data than mobile apps, due to the higher number of components involved. This will make it easier to prevent, troubleshoot and fix problems when they arise.

    About the author/Kevin Dunne

    Kevin Dunne picKevin Dunne is VP of Strategy and Business Development at QASymphony.

     

  • Embedding Predictive Analytics

    Embedding Predictive Analytics

    For an enterprise with a deep DevOps maturity model, the initial “wow” factor of automation has worn off. At this point, the sexier concepts involve proving continuous improvement. As an example, our first baseline of metrics might have taken the pre-DevOps timings of a typical build and deployment of a given application. We can measure this time against our post-implementation of DevOps automation and show a significant savings in time. This is opportunity time, meaning an employee must be ready to turn around the decision cycle faster to take advantage of the time saved (not be playing solitaire, for example). But once DevOps automation is old hat, the difference between our initial post-DevOps typical build and deployment time and our current one still using DevOps may not be all that significant. But is there more hidden there than meets the eye?

    Establishing Context & Baselines

    At this point context may be a more meaningful measurement. As an example, if my first version of DevOps automation for a deployment contained 35 steps and my current one contains 160 steps it would stand to reason I am “doing more” now in my deployments than I used to do. These additional steps may involve executing more tests than I originally thought to do. They may be interacting with monitoring systems (turning them on and off as appropriate), which reduces “noise” in the alerts my monitoring teams have to respond to. In short, over time, my deployment processes may become “longer” because I am increasing the capabilities or the sophistication of what I do during these deployments. While from a pure timing perspective this may make them longer, it also makes them more powerful and, so, is worth measuring. To know the delta, however, I must have retained older versions of automation to compare against, not just throw away the old every time I update with new steps.

    So let us assume the context of my automation is now fairly stable. I do the same things, more or less, the same way every time I deploy my application. At this point, measuring the functionality of what I do becomes more meaningful. As an example, my deployment automation may complete four main functional tasks to complete, I deploy the code (the app), I deploy the dataset, I deploy the config files and, finally, I run a series of tests on the above. While I may not ever incorporate too many new kinds of things to do in this automation, understanding how long each function takes to complete is relevant to proving continuous improvement. For instance, if my code takes roughly the same amount of time each deployment, as does my config files, but all of the sudden my dataset begins to grow in each deployment (from normal usage), my overall deployments may be getting slower. This is not due to hardware inefficiency or network inefficiency; it is simply due to a larger dataset that keeps growing.

    If on the other hand, it is my test set that is getting progressively longer to execute, that is an indicator of test script inefficiency (given same code, same config and same dataset). If the number of tests being executed has not increased, but the length of time it takes to run them has, there likely is inefficiency in “how” the testing is being constructed or executed. Understanding the context for what is going on in a deployment or similar kinds of automation must occur before introducing predictive analytics can occur.

    Historical Trends Extended

    Predictively evaluating automation is nothing more than taking the data from a historical perspective and extending the trend lines into the immediate future. If for example, my deployment automation is getting four seconds longer each month for the last 12 months, I can reasonably infer the trend will continue, and I can predict the length of time it takes to execute a deployment in the coming months. Understanding “why” this occurs requires a contextual investigation, but predicting its occurrence does not. This may seem inconsequential as we talk about the length of deployment processes of a given app, particularly if my portfolio of apps is small. But if the trend holds true for a portfolio of thousands of applications, the impacts become more consequential. Or if the phenomenon is directly connected to a particular cloud provider and not seen elsewhere, again, the consequences become meaningful.

    Where having “ground-up” predictive analytics (build and/or deployment automation) becomes much more useful is in the construction of release events. Often, release managers are asked to supply an estimated time duration for a given event to factor into the construction of a new release. If a vendor were to integrate more predictive analytical tooling into the release orchestration (RO) toolsets, “random” or “best guess” timing estimates could be replaced by extending historical data into the timing estimation fields. The RO tooling could provide a default timing estimation based on examining historical trends for a period defined by the user. Estimation defaults could always be replaced by a user (assuming discreet knowledge of other impacting events), but at least the new overall lengths of releases then could become significantly more accurate.

    This has special significance for manual tasks. Let’s say our database admin team traditionally takes 40 minutes to perform a given release event update, but now is running significantly longer than that. Having the RO software aware of the historical trends could alert the release manager, as well as the database team manager, during the event to check out why this is occurring. For manual tasks in particular it may be hard for anyone besides a predictive analytics module to determine what timings “should” be, as compared with what they are in a given release.

    Impacts to the Bottom Line 

    The real questions are, Why should a customer care about this kind of feature set? Why should we be looking at the vendors of our tooling to incorporate forward-thinking architectures such as predictive analytics into the tooling they sell us? Because the efficiency it can bring to our DevOps set of services lowers the costs of administration and increases the speed at which we can deliver innovation in general. Over time, trending combined with context can statistically prove continuous improvement of our DevOps services, which is the promise we made to the executive teams that sponsored and made the investment in DevOps to begin with.

    Metrics analysis can occur (usually with a dedicated investment on our part) without a predictive view of the impacts. But this changes the perspective of our analysis to hindsight thinking instead of forward thinking. In essence, we then are simply “reacting” to what has occurred instead of tailoring our environment to alter the picture of future predicted events to be better than the software predicts it to be. We start tweaking the variables to instantly impact future predictions, instead of only examining them in a post mortem.

    This perspective is yet another cultural catalyst for change your business needs. When I talk about changing how an organization “thinks” about innovation, this is one of the key objectives. We take an enterprise from knowing little about the costs of innovation, to having an extensive knowledge, to having a forward view of what is—or what should be, possible. All of this is possible despite the loss of waterfall controls and in the midst of an agile construction model. All it takes is the integration of forward-thinking architectures into the products we select. If our vendor does not offer them, we begin to enlighten our vendors on the value they would bring to us in cost reduction and to them in market capture. If you need help in these areas, feel free to contact me, I am looking at the moment 🙂 .

    To continue the conversation, feel free to contact me.

  • Tales of DevOps Discovery Part 2: Definitions Matter

    Tales of DevOps Discovery Part 2: Definitions Matter

    Definitions Matter: Step Two in a DevOps Transformation

    The first post in this series on DevOps Discovery discussed the importance of asking fundamental questions of stakeholders to facilitate alignment on priorities and establish a mission statement when beginning a DevOps transformation. Part 2, below, reviews definitions of common DevOps-related terms and provides examples of how failure to agree on such terms can hinder your transformation.

    The industry is full of terms such as continuous integration, agile, continuous delivery, DevOps, etc. The practices described by these terms often are defined differently by various industry “experts,” resulting in stakeholders in the software delivery process having different understandings of what constitutes these practices. Some will state they are already following the practices; others will state they do not. Some will believe the objectives unachievable, while others will be bullish on the prospects; differing interpretations blocking agreement on the organization’s current and target states. If stakeholders cannot agree on the current state (or the start point) and the target state (or the destination), it is impossible to “map a route” for the journey to DevOps.

    For example, let’s look at how some define continuous delivery:

    Continuous Delivery – Definition 1:

    A software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing and releasing software faster and more frequently.

    The approach helps reduce the cost, time and risk of delivering changes by allowing for more incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.

    Source: Wikipedia

     

    Continuous Delivery – Definition 2:

    Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.

    [Continuous Delivery Blog Post by Martin Fowler]

     

    You may read the above and initially think there is no difference in the definitions but, note that the first definition addresses deployment of software with the statement: “A straightforward and repeatable deployment process is important for continuous delivery,” whereas the second definition addresses deployment of software with the statement: “Build software in such a way that the software can be released to production at any time.”

    This distinction may seem trivial, but it is the often the source of confusion between the practices of continuous delivery and the practice that is commonly called continuous deployment. In contrast to continuous delivery, which states that every software build “could be released,” continuous deployment is loosely defined as “the practice of releasing every good build to users.”

    When establishing your target state for the DevOps transformation a seemly slight confusion such as this can result in pushback from stakeholders concerned that continuous deployment is unachievable or too disruptive. These sentiments include:

    • Business: “We cannot gather, spec and track our requirements at that rate.”
    • Operations: “There is no way we can support daily delivery into production, ensure security, uptimes and perform our day-to-day work. It takes us an entire weekend and multiple attempts to deploy today.”
    • Customers: “Our compliance team needs three months to review and approve every new version of our accounting system.”

    The confusion and potential pushback exemplify the need for establishing baseline definitions and alignment at the start of a DevOps transformation, and doing so in the specific context of your organization. The importance of points such as how far it deploys varies with the organization—some organizations can support continuous deployment to production multiple times a week or even multiple times a day, such as Netflix does. Other organizations, such as providers of critical financial systems, may never pursue this cadence of deployment. Be sure to apply these industry practices pragmatically as fit for your organization, not dogmatically based on generic industry definitions.

    Let’s look at a couple of definitions of DevOps:

    DevOps – Definition 1:

    “DevOps (development and operations) is an enterprise software development phrase used to mean a type of agile relationship between Development and IT Operations. The goal of DevOps is to change and improve the relationship by advocating better communication and collaboration between the two business units.”

    [Webopedia]

     

    DevOps Definition 2:

    “DevOps (a clipped compound of “development” and “operations”) is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software, can happen rapidly, frequently, and more reliably.”

    [Wikipedia]

     

    Like the example with continuous delivery, these two definitions also are very close. Yet on detailed observation, it is clear the first definition describes DevOps as being focused on relationships (i.e., culture and collaboration). The second definition extends this to include the “automating the process of software delivery and infrastructure changes,” giving consideration to tools and technical practices such as continuous delivery.

    Does this matter? Yes. Often if the executive sponsors and key stakeholders do not account for the need to establish tools and technical practices and only focus on relationships, there is a high chance for unrealistic expectations for timeline, budgets and results. Conversely, if the need for changes in culture and process are not considered and the DevOps transformation focuses only on tools and technical practices, the efforts are also unlikely to yield the expected results. A DevOps transformation must account for the trinity of “People, Process and Tools.”

    When considering DevOps, it objectives and the objective of its component practices it is useful to establish a clear idea of the relationships between theses practices. It is often useful to visualize them as below:

     

    DevOps Transformation Triangle
    The DevOps Transformation Triangle

     

    DevOps is the “What,” the target state as defined by a set of guiding principles. Agile development, continuous integration and continuous delivery are the “How,” a collection of process, practices and tools that together build the foundation for a sustainable DevOps implementation.

    In the next section, we will review some of the initial discovery questions asked when beginning on a DevOps transformation, provide real-world examples of answers from various stakeholders in real organizations, and analyze those answer to better understand the journey to DevOps.

  • The Automation of Compliance

    The Automation of Compliance

    Consider how much money your enterprise spends on audits. Smaller firms not subject to external regulation should consider how much money is spent on quality assurance—or the lack of it. A business that suffers from a customer-facing quality incident will, at minimum tarnish the brand of its services or at the high end, lose market share altogether. Larger firms that fail audits can suffer from brand tarnishing as well as high fees, not to mention further scrutiny by regulatory agencies.

    Failure to perform or failure to follow proscribed processes inevitably leads to a black eye in front of customers. The entire discipline of governance is premised on avoiding both. Whether your governance practices are efficient or not—and whether your “process” as it sits today leads to quality or not—is not the subject of this review; it is how DevOps and automation can shine a spotlight on both and radically reduce the costs you spend today to achieve them.

    The Spotlight of DevOps

    The spotlight of DevOps begins with consistency that leads to predictability. Automating any given kind of process or technology the same way every time begins to give insight on any “holes” from things that do not work (or work well). If all that changes in your deployment process, for example, are pointers inside a configuration file and the kind of testing executed against this version of the code, you have achieved a build-once, deploy-many state of deployment automation. Operations staff who support the deployment of change or maintain the environments in the software development life cycle (SDLC) begin to  isolate any issues faster because they become familiar with “how” a given kind of technology is automated to deploy. This reduces the total number of deployment errors and, in tandem, reduces the length of time it takes to fix them. Any “process” problems or weaknesses in your governance model become apparent quickly and can be fixed quickly. Limiting variation is the medicine to keep them away in future.

    The spotlight of DevOps next turns to transparency. Offering your entire enterprise visibility or “read only” access to your entire DevOps tooling suite has a profound impact on the transparency of change. As an example, consider that I am a humble accountant in your firm who has submitted an agile story to improve our billing system. I may or may not, given the nature of my job, be co-located near the programming team who will fulfill this request. If our firm specializes in accounting software then perhaps I will co-locate, but if I am only requesting a change to an ancillary or support system, likely not. Without direct access to the programmers through traditional agile co-location, my visibility into the progress of change is limited to the tracking system I submitted the request within (assuming you have one, and don’t use email to do this).

    If I know a programmer in MIS, perhaps I ask him at lunch how my change is coming along. But if I have read-only access to the testing systems, or to the release orchestration (RO) tooling, I may get a much better up-to-the-minute view of exactly where my change is, how it is doing from a quality assurance (QA) perspective, and when it is scheduled for release into production. Based on this news, I may make decisions about how we bill our customers in what time frame to increase overall revenue to the company. I would notify my boss of same (who also has access, but pays less attention), and of any changes that impact the new timeline I create. The marketing team (who also has access) begins to work with public relations (PR) and the advertising guys on customer announcements, etc. etc.

    The net impact of having universal transparency is that audiences who are not traditionally notified or interested in the inner workings of change movement now make decisions based on it. Consider that prior to this systemic access, and the reliability of DevOps to ensure delivery, the only news you got before was from “the IT guy,” who was often wrong. It does not take long before IT loses credibility from being wrong about delivery timelines, and these other audiences begin to pay less attention (if at all). At that point, an “I’ll believe it when I see it,” attitude emerges. And this same attitude infects internal auditors because of what they hear and see from disaffected internal customers who have been misled repeatedly by the failed manual methods of the past (pre-DevOps).

    Processes that are disinfected over time by the consistency of DevOps begin to change the internal company thinking about the delivery of innovation—namely that it can be counted on as accurate. Transparency offered to all audiences of a company increases the visibility into all aspects of innovation delivery and better ideas emerge, both about what to do with completed efforts and how to improve the innovation process itself. And before you know it, your governance and QA processes become efficient, reliable and predictable. Variation is eliminated or greatly reduced, and the costs of operation and innovation begin to decline.

    The Forgotten Stepchild of Automation

    What is rarely a matter of focus in most companies is the secondary ability to automate the compliance history of a given application. Most firms are content (translate, only able to) work with their testing tools to produce logs about a given version of an application now residing in production. A test log does show an auditor the majority of the quality related information about a given version of an application. But the log is a highly limited view of what has taken place. What is needed is the full context of innovation. Enter compliance automation.

    Consider for a moment, that requirements drive the train. Why are we doing the innovation we are doing? Is this a bug-fix, a new feature, a client request, a regulatory requirement? Being able to trace the code itself back to the requirements system that drove the need adds significantly more context than is possible from mere test logs. Next, think about the number of attempts that finally resulted in this version of the application. An automated build tool (part of our DevOps suite) has this information. At this point you could evaluate how many builds and deployments took place in development before a worthwhile candidate emerged to traverse the remainder of the SDLC. An automated deployment tool (part of our DevOps suite) can trace the number of environments this version of the app was deployed into for testing and how long the overall process took to get something moved from development to production.

    And finally, testing is often done from a variety of tools, with results compiled in a variety of result repositories. This information may be available from a solid RO tool that tracks the gated progress of this version of the application, but if it’s not, it still should be sitting in each of the testing tools used throughout the process. Depending on the level of inherent DevOps tool integration (i.e., using a vendor suite) much of this information may “flow” from one tool to the next. But if not, the information does still exist in the logs produced by each of the DevOps tools.

    It may be well worth the application of internal programming resources to put together a tool that gathers all this information (automates the collection and presentation of) for audit purposes. Imagine the reduction in cost of supporting an audit that can gather a simple push-button report (or dashboard) of the entire life cycle of a given version of change. At that point, you have the entire context for a piece of innovation. You know what drove it (requirements). You know what it took to create it (build and deploy). You know the entire history of testing it (testing tools). And, finally, you have the entire governance process used in its deployment to production (RO tooling). But instead of examining multiple tools to collect and present it, you have created a push-button system to do so.

    Impacts to the Bottom Line

    The investment in adding automation to the compliance history of innovation pays off in more than one way. It is not just the ease with which history can be recited. It follows the same principles as DevOps itself. Automation introduces consistency in how audit information is gathered and presented (reducing variability from one audit to the next). Automation introduces transparency if the entire company is given “read only” access to this new system. Automation shows auditors we are not afraid to learn from our mistakes and proud of our accomplishments. And in effect, it reduces the need for audit altogether.

    Typically, regulatory bodies investigate and audit firms where they are uncertain of how the results were obtained. An automated system like this one fully eliminates that need. Transparency as could be offered here, may convince auditors that attention is not warranted here. Keeping in mind the spotlight of DevOps has already improved quality and streamlined process, add to that full transparency of audit materials and the confidence of auditors regarding your company may increase significantly. Routine becomes the only reason to audit, rather than suspicion.

    Invariably the “new” will sometimes result in an error that your current systems and processes (automated or not) will not catch. But this truism is recognized, and accounted for, in how DevOps enables innovation. And if you have this kind of automation in place, it can be demonstrated in how we fixed what we fixed (whether governance or QA). Your strategies in DevOps should help you achieve this level of confidence. (If you need help, feel free to contact me, I am currently looking.)

    To continue the conversation, feel free to contact me.

  • The War with Human Nature

    The War with Human Nature

    What is the difference between “leading” edge and “bleeding” edge? Being the first human being to advocate change.

    It is a virtual war with human nature to accept change. For all the innovation introduced into our society, most of it is a product of invention over evolution. Don’t get me wrong: Once we find something that works, we inherently work tirelessly to make it better, thus evolving it over time. But the genesis of the “new” is easier to accept as creation—a new idea, a new innovation of our own—than a change in direction, a forced acceptance of the ideas or creations that were “not invented here.”

    In the workplace, most employees are looking for personal security, for a feeling derived from consistency and predictability in the day-to-day work activities. Anything that threatens an employee’s ability to continue to secure compensation becomes something that inherently meets our resistance. The No. 1 culprit: you guessed it … “change.” As long as what I do remains the same, over time I will get better and better at it. Practice makes perfect. But when something “new” is introduced in the actions I am supposed to perform, there is a risk that my value, my expertise, will be reset to zero and perhaps I will not be valued as I once was. This threat to my compensation is real; therefore, my resistance to it is equally real.

    Programmers Found Guilty

    You would think that engineers and those working in the software industry would be exempt from this phenomenon. But most are not. Finding an ex-Cobol, ex-Clipper, ex-Windows, current mobile Apple iOS programmer is like finding a unicorn in Central Park. Most successful Cobol programmers are still Cobol programmers, looking for jobs in mainframe shops, content to see the new generation of programmers not competing for their declining job markets. The early generation of programmers minted after the adoption of the PC computer tend to know less about Cobol and RPG-3 and more about PC-based programming languages such as GW-Basic and Pascal (at least the older ones). Programmers minted after the Windows graphical adoption know more about that generation of coding languages, and so on, and so on.

    The basics of coding have remained the same since the advent of this discipline. We create a list of instructions, in sequential order, that the computer must execute to produce a desired result. There are nuances to this, and an explosion of capability in this, but the basic principles remain much the same. Yet, programmers well-versed in a particular language are shy about learning a new one, unless the platform is similar, the features are similar (albeit improved) and the belief is high that they will be successful in the new language quickly. There is no reason why a Cobol programmer could not be successful creating mobile apps. Or, for that matter, a mobile app developer could try his hand at Cobol. But it seems neither do, at least not very often.

    What happens instead is that each programmer begins to establish his/her own ecosystem of support: simulation models that imitate the platform responses, highly evolved text entry tools that auto-format code and check for omissions such as missing parenthesis and scripted tools that perform the mundane tasks we need to build (or construct) new versions of code or deploy a build into a new environment. This same ecosystem of support will exist for database, network and middleware engineers as well. The tools will be decidedly different, but the reasoning for them tends to be identical: to make things easier to do.

    Looking for a Magic Bullet

    There is no cure for breaking down the resistance to change, except constant exposure to it. However, forcing employees to face perceived threats to their value—and their compensation—will, at a minimum, produce exorbitant stress at work—or, worse, an employee turnover rate that would rival donuts next to a police station. The goal is to introduce change that will ultimately increase the value of an employee, as well as potentially increase the compensation they worry about. This is where the DevOps value proposition must be created for each impacted audience. A generic value statement is not enough, nor is a mandate by top company executives. You cannot “force” people to adopt something new, anything new.

    Creating the right value proposition must be specific to the audience you seek to adopt the “new” system, tooling or process. DevOps is a continuum of all three (people, process, technology). So for the database team, for example, changing out the ecosystem it has evolved into a new set of tools and processes, may seem like a hard sell at first. The advocate of DevOps (whether the exec teams or the service teams delivering it) must be able to articulate the value proposition clearly to the database teams.

    “Why” should they move to a newer ecosystem? A stubborn engineer can always point out the holes in your tool and the beauty of his own. He/she will likely have evolved his/her own support ecosystem to a state of near utopia … but only for himself. Usually the processes and tooling engineers use to make their own lives better do not scale to others in the department. And God forbid some other engineer updates “their” scripts or makes changes to “their” process. While this is wonderful for the engineer who creates it, it is miserable for anyone else attempting to use it.

    Constructing a Value Proposition

    The value statement must trade prima donna utopia for scale. Using common DevOps tooling, it is possible to support many more task requests than most individually constructed ecosystems can handle in a given day. And believe me, those task request increases in volume are coming. For example, an Ops engineer may be able to handle provisioning five systems a day with his own tooling, but when requests come in to provision 50 or 100, he is dead in the water. Having the Ops engineer help manage the deployment template or framework allows users to self-serve on provisioning requests and keeps him from having to do everything for everybody. His job is different but every bit as important.

    The value statement must trade current skills proficiency for next-generation, market-valued skills. In effect, a database guy is valuable, but a DevOps database guy is considerably more valuable, or a “big data” database guy, or a IoT database guy … pick your poison. Whatever you are today, whatever skill set you have, the jobs that pay more money are the jobs that fewer people know how to do. At some point, a lack of DevOps proficiency will relegate an engineer to the pages of history. On the other hand, adopting and embracing DevOps will allow engineers to get the training they so desperately need to keep their career “fresh.”

    Finally, the value statement must trade today’s earnings for tomorrow’s increase in earnings. Companies that embrace DevOps are going to either save extreme amounts of overhead by staff reduction remaining at the same level of innovation productivity. Or, they are going to increase innovation productivity levels radically, making them leaders in their respective space by meeting customer demand much faster with higher quality (the path most taken). Either option should translate into higher compensation for employees who embrace DevOps. If companies do nothing to increase employee comp, choosing only to increase profits, competing firms will be happy to poach them and pay them more instead. This forces more costs in recruiting, training and employee turnover when it happens again.

    A Human Strategy that Works

    Your strategy for training employees to embrace change should be particular to the kind of work they do, and reflect the things that will matter to them. If you need help crafting a strategy let me know, I am looking at the moment 🙂 . Not every team will be as critical to the overall success of DevOps as perhaps your engineers are, but the value proposition should adjust for this fact. Keep in mind, if the change is less, then the resistance will be less as well. Your accounting, HR and legal teams, for example, may not feel the impact of DevOps as rapidly or as severely (in terms of change); thus, their resistance may be less, but it is likely more than zero.

    DevOps is about people, process and technology. It is in that order, with a heavy emphasis on people and a relatively light impact from tooling. And it will succeed or fail based on the embrace of people, process and technology. Process and technology do not care about change. People do. Since people are the largest priority or component of DevOps, it is their fears that must be addressed and their value that must be enumerated to succeed.

    To continue the conversation, feel free to contact me.