devopsy.kinsta.cloud

Tag: devops integration

  • Bridging the Gap Between ITSM and Software Engineering

    Bridging the Gap Between ITSM and Software Engineering

     

    Considering the individual roles of IT service management (ITSM) and software engineers, it seems obvious they should work closely together. ITSM tracks internal and external customer issues, and engineering fixes code-related issues (such as defects) that IT cannot fix on its own.

    However, more often than not, coordination between these two departments is poor, which—critically—undermines customer response times and damages application satisfaction. It’s important to remember that the quality of customer response isn’t just about speed, it’s about giving customers insight into the state of the issue.

    The disconnect between ITSM and engineers—caused by individual priorities and semantic differences—makes this real-time insight difficult. Both teams have individual and shared concerns, and separating these while simultaneously improving collaboration is a big challenge.

    Addressing this relationship is an important step in optimizing ITSM and DevOps integration, alongside focusing on IT help desk enhancement requests to product management, and tracking code from build to release and back.

    The Flow of Information on Things that Matter

    Most of the tickets generated by the IT help desk can be handled internally with no help from engineering, but it’s the tickets that do need help that require consideration. For these issues, there needs to be an open flow of information on the things that matter between these two teams.

    IT help desk must give engineering enough information to understand the defect and its impact on the customer. Meanwhile, engineering needs to be able to give the help desk a sense of when to expect the fix, when the fix is done and be able to request more information when required.

    Information that should be shared from the help desk to engineering includes the following:

    • Severity – Engineering needs a sense of the importance of the defect. If everything comes in as “High Priority,” nothing is high priority. The help desk must understand that engineering already has its own internally generated set of defects, as well as defects found by QA and other sources. For these customer defects to be prioritized and fixed in a timely manner, engineering needs to know how impactful they are to the customer.
    • Description – This is perhaps the most important piece of information the help desk can pass along. The more contextual information that can be shared, the better. It’s immensely important to provide engineering with the necessary steps to reproduce the specific defect, which version of your tool the customer is using, the general environment, etc. Sometimes this is best in structured fields, whereas other times it’s important to simply write out everything.
    • Screenshots and other visual information – Along with the written description of the problem, engineering often requires logs and/or screenshots. As the saying goes, “A picture is worth a thousand words.”

    Now that engineering has the vital information it needs to fix the defect, it has a responsibility to provide useful information back to the help desk:

    • Planned release – This should not be a firm commitment, but rather an estimate to give the help desk general timing. While the actual date of the fix may vary, it’s still useful to be able to tell a customer,”This is coming soon” vs. “We realize this is a problem and it’s on our backlog to fix.”
    • Engineering status – While this is a single defect to your customer and to them only has one status, there are really two statuses your organization will care about:
      • Status of the ticket in your ITSM tool
      • Status of the defect in your defect tracking/agile planning tool

    It’s very important that the ITSM tool contains both statuses to provide more information and context to the help desk. Imagine the benefit of knowing that the defect is “In Triage,” “In Progress,” or “In Verification” and being able to provide that information to a customer.

    • Actual release/released – As opposed to the planned release information, this lets the help desk know that the customer’s issue has been resolved. Imagine running a report from your ITSM tool indicating the defects fixed in your most recent release. Your IT help desk can proactively reach out to customers to let them know to upgrade, enabling you to offer more proactive customer communications.

    Finally, and perhaps most important, you allow your teams to talk about the “same thing” even when it exists in different tools and they speak a different language. There are four core ways to do this:

    • Cross tool tracking – Each artifact in each system should have the ID (and ideally URL) of its twin artifact in the other system. If a help desk representative needs to talk to an engineer about a defect, the rep needs to let engineering know which defect it is in the engineering tool, not just in the ITSM tool.
    • Routing – IT help desks often support multiple products, each of which have multiple components that may belong to a different engineering team. The help desk needs some indication which component and/or team should work on the defect. This information will help the IT help desk know which team to reach out to for help.
    • Filtering – Not every ticket in the ITSM tool needs engineering help (that’s a good thing). However, there should be some indication on the IT ticket if it has been sent to engineering. You do not want to send the same defect to engineering twice (confusing) or to assume it’s been sent when it hasn’t (poor time to resolution or issue overlooked).
    • Communication – IT and engineering need a shared way to talk about a specific defect that they both care about. The closer this communication can be to the native commenting that already occurs, the better. Ideally this is not via email as that communication vehicle does not stay attached to the artifacts in question, meaning information can be hard to find or get lost.

    The Benefits

    Customers will love it – Improved communication means happier customers. Customers can plan their upgrades with more certainty, and their trust grows since your organization is able to deliver on its promises. No longer will they feel like their issues are being thrown into a black hole.

    Better engineering focus – Engineering gets pulled in a variety of directions, contending with feature work as well as internally generated defects. The more information it has, the easier it is to prioritize and deliver more value to customer. Without the ability to share the severity of a defect, you risk two undesirable outcomes:

    • The defect is fixed instead of more important work, which hurts all your other customers.
    • Other less important work is prioritized, which can threaten your relationship with this customer.

    Better trust across your organization – Open communication fosters trust and understanding. Providing insight into prioritization and status let’s your help desk feel part of the development process. No longer will staff feel like they’re sending problems into a black hole with no idea when they will be fixed. Enhanced cross-team communication also helps engineering to avoid feeling like work is simply being thrown over the fence. Obtaining more context around a defect is invaluable.

    Next Steps for Total DevOps Integration

    To fully integrate your ITSM and engineering teams, there are still (at least) two other processes to implement:

    • A method for the IT help desk to provide enhancement requests to your product team: These enhancement requests are similar, but distinctly different from the defects described above. Consequently, they need to be prioritized and triaged by a different team.
    • The other process to implement is the traceability from task to code to build and back again. In order to properly inform customers that their problem is fixed in this version of your product, it’s important to correlate features, stories and bug fixes with the actual version of your product that includes them for the first time.

    Bridging the gap between ITSM and engineering teams brings your company closer to customers. While making your organization work as a unified whole is an incredibly complex problem, addressing this relatively small—but vital—piece of the puzzle helps you on your journey to integrating your whole software value stream. This change of mindset—combined with connecting all teams, tools, disciplines and processes in an enterprises’ software value stream— optimizes collaboration, productivity, efficiency and, most importantly, value delivered to the customer.

    About the Author / Trevor Bruner

    Trevor Bruner is a Product Manager for Tasktop Technologies. His background includes experience in financial services, oil and gas and as a Navy submarine officer. Connect with him on LinkedIn and Twitter.

  • Interoperability’s Impact on Transparency

    Interoperability’s Impact on Transparency

    We fix what we see is broken. Providing “transparency” into the DevOps-enabled software development life cycle (SDLC) has tangential goals of quickly illuminating what is broken, running slower than it should or working flawlessly.

    Automated testing systems, for instance, are generally good at highlighting pass/fail results from integrated testing in the development process. Once they have proven their reliability, they become a de facto part of the development automation and rarely are thought of again. However, when a post-DevOps legacy build begins to take longer and longer to complete, are we able to quickly examine the runtime of our entire body of integrated tests to ensure they continue to perform within expected execution timing parameters (i.e., could the code be fine and the tests themselves be the culprit)? And is that problem exacerbated when the number of different testing tools begins to explode on the same coding unit of work?

    Connecting the dots has always been the goal of DevOps. In the ideal world, the business requirements—or the epics and stories—are part of the “dots” that remain connected or associated with each change in coding, builds, deployments and ultimately release. But are these “dots” easily transparent in each part of the DevOps continuum?

    Transparency from multiple perspectives

    To think engineers are the only audience of DevOps is too restrictive. Product and project managers are also interested in knowing “the state of change” as it relates to the business issues they are tracking, and in real time. Whether the requirement originated from a bug-fix or as a new feature, the audience of people who want to see what is where within the DevOps set of tools is not confined only to people within the development or operations of IT. The ability to provide sensible, near real-time presentation of what changes are where gives the entire organization transparency to the progress of change. It impacts every organization from customer service, to program management—executives want to know as much as casual viewers in many different business functions.

    Audit is another key driver for full transparency. Regulatory compliance is at one end of this need; corporate quality standards should be on the other. Organizations always should be more concerned with quality standards than the regulations they are subjected to. Therefore, the process of audit always drives the need for greater transparency. Imagine the ability to have “push button audit” enabled by DevOps for any change under inspection. Imagine what further cost delta this represents to organizations that are still bound to only manual methods of examination. With DevOps, not only is this possible, it also should be normal. The entire history of any change entering production should be stored by each of the DevOps tools within the journey, from ideation to implementation. All test records should be immediately accessible (and automatically associated with the particular version of change in question). Everything from failed builds to the exact time it was released into each impacted data center around the world can—and should—be available for a consolidated, push-button recollection for audit. However, the ability to make this happen often is directly proportionate to the interoperability of the DevOps tools used.

    Transparency of key data and processes

    When point solutions are selected for the DevOps tooling (defined as best-of-breed tools, with no inherent interoperability with other tools in the process), the particular functions they perform are thorough and generally complete. However, the only way to integrate key information between different tools is through APIs and/or a homemade amalgamation of the data from each point solution tool into a common dashboard. Amalgamating data is always a challenge, particularly when you consider the point of view of the audience varies by function and business discipline. If you eliminate or reduce the points of view down to only engineers (as is what usually happens), you destroy the value of DevOps transparency for every other business discipline in your organization. If you spend the labor to create a very functional, multi-perspective dashboard, your investment can be much higher than what you make in the tools themselves.

    Integrated solutions—generally pushed by the same vendor as the preferred (albeit more license-cost-heavy) solution—are not the ultimate panacea, either. While a given vendor may be able to offer tools that track requirements, construct builds, integrate testing, execute deploys and orchestrate releases, more often than not, the vendor is stronger in one discipline over the others. They may ask you (the customer) to use a limited number of tools designed for one thing to do more functions than the tools were intended to perform—which creates a recipe for constant customization and lucrative service deals for the vendor.

    What becomes important to measure in any of these instances, is the level of inherent interoperability that will exist between each type of DevOps tool in the offering. If the “connecting of dots” is inherent, and the state of change can be viewed in any of the tools, that is a win. If the point-of-view offered extends to more than just an engineer, but includes other business audiences who may have interest, that is a win. If the reporting is push-button, that is a win.

    Transparency of tool usage and roles

    But beware the dark side. Whether your organization relies on point solutions or integrated solutions, both can be corrupted by an inconsistent methodology of user profiling within the tools themselves. For example, let’s assume we have a business requirements tool, a build tool, a deployment tool and a release tool. Generally, a programmer, who we’ll call “Dave,” has a very specific role in the SDLC process that doesn’t include releasing code (programmers usually don’t release code into production). Generally, the level of granularity in defining tasks and roles within a given tool is proportionate to the rules your auditors require (whether in-house or regulatory). So for a given bank, we might define the role of a programmer “Dave” to be able to see business requirements, execute builds all day long, deploy them to the test class (or first level) of environment in the SDLC and then simply watch what happens as it progresses to production, never directly interacting again.

    One of the biggest problems DevOps tooling has is that each tool handles user management separately. That may allow me to define the roles of a programmer properly within each tool, but it does not enforce that my guy Dave is properly mapped to that role. I could make Dave a business analyst in the requirements tool, a programmer in the build tool, a deployment manager in the deployment tool, and a release manager in the release orchestration tool. This would allow Dave god-like abilities in each tool within the process. He could, in fact, create the requirement to move all our money to his account, build the code to do it, deploy it and release it into production. Dave would not have triggered any alarms in the tools, because each tool thought he was doing what he was supposed to be doing in that role. And I have never seen tools audit what role Dave is supposed to be mapped to given his job in the company or even spot-checking what role he carries in other DevOps tools to see if there is an inconsistency. Perhaps abstracting user management from the entire set of tools may be the answer—but then, who offers that capability today?

    Transparency, when properly managing access controls and particularly at the push of a button, can provide solutions—even if only after the fact—to a great number of problems that exist in our organizations and in our SDLC process. We fix what we can see is broken. If our DevOps solutions can highlight transparency from ideation to production release, we can have confidence that what we see is real and worth our attention. When our transparency varies by tool, or loses connection within the process, our confidence of the state of change diminishes and our certainty about what needs our attention goes away. Using DevOps to shine a light on our true priorities is yet another cultural change enabler we have the responsibility to pursue.

    To continue the conversation, feel free to contact me.

  • How to accelerate your production integration activities

    How to accelerate your production integration activities

    IT Operations is often accused to be the bottleneck in application release processes and a general constrain to agility. But why is that, what are the root causes of this issue and how can we address them?

    There is indeed a very big gap between the desired world of DevOps with Infrastructure-as-Code, Docker, Microservices Architectures, Private Cloud and the Software-Defined Data Center and the realities of most software delivery in enterprise IT today.

    Ideally tools are shared between dev and ops, information flows in real-time, everything is measured and visualized through dashboards and release pipelines are fully automated.

    In reality, however, each major release of enterprise software requires a myriad of preparation activities from IT operations and infrastructure to ensure the operational readiness of the release in production. We call the set of these activities production integration.

    We put these activities into four groups:

    1. Batch automation. New application releases often require new jobs and changes to jobs and job plans. Their impact on existing production chains must be assessed and controlled.
    2. Data Migration automation. Operations and infrastructure frequently has to support the data migration process and provide the technical foundation and automation.
    3. Deployment automation. Changes to deployment plans and application components will impact existing deployment scripts and routines. They need to be updated and re-tested.
    4. System Provisioning and Configuration. In the course of a new release, it is often required to add infrastructure such as servers, storage and network bandwidth. On top of it, middleware and databases need to be prepared for the new release.

    Through over 300 customer interviews we identified the main reasons:

    • Lack of software engineering methodology: Employees with sound software engineering skills are very rare in ops departments. However, sound software engineering practices are critical to build re-usable and modular automation frameworks and libraries. Without that, the risk is high that automation is created again and again for similar purposes without cleverly re-using what is already available.
    • Priority Conflicts: In many organizations, the same teams work on production integration and production support. Urgent production issues cause shifts in priorities and delays in integration activities. The problem is exacerbated, because specifications for production integration tasks are often received late in the process, not leaving enough of time for the teams to deliver.
    • Organizational Constraints: Over-specialization causes bottlenecks. We all know this database administrator, who is the only person in the company able to identify and find the root causes of performance problems, or this application server expert, who exclusively owns the deployment scripts. Also, dislocation of teams causes communication overhead and slows down the process.

    We have identified a set of measures to address the most obvious obstacles to more agility and speed in the production integration process and increase its transparency. We call this the DevOps Production Integration toolkit. It is based on three pillars.

    Agile Enablement of ITO: We recommend organizing the delivery of production integration tasks through a tailored version of Kanban. We have developed a methodology to integrate Kanban-driven ops with SCRUM-driven development in a scaled organization.

    Kanban is based on managing a pipeline of tasks and visualizing their flow. With Kanban, team members pull from a prioritized backlog, and available delivery capacity is managed to avoid the occurrence of bottlenecks. Kanban allows for constant re-prioritization of tasks in the pipeline.

    All activities of a team – including both the analysis and resolution of production problems and production integration tasks – are managed through a single pipeline of tasks. Color-coding helps to separate different releases and production issues, as well as identifying dependencies. The Kanban board is organized by swim lanes to account for different roles and skill sets in the team, for example database administrators, middleware administrators and production control experts.

    A single prioritized backlog may be implemented either each Kanban team or across multiple Kanban teams. The backlog contains all items these teams are responsible for: all production integration requirements (we call them automation stories) and all production issues, which need to resolve.

    The backlog is managed by a product owner (PO), who has to collect and maintain requirements from all stakeholders – development, operations, security, first line support and others – and is responsible for their constant (re-) prioritization to create and maintain a fully ranked list of items. Requirements from development are typically retrieved in the development sprint planning meetings.

    The PO writes automation stories, which cover the “what” and the “why”, adds them to the backlog and works with the Kanban team on the creation of tasks.

    Version Control for ITO: While advanced version control is part of every application development team today, it is not yet wide-spread in IT ops and infrastructure. However, with an increasing release velocity, the number of script versions and other automation artifacts, which are created, used and modified within different environments, increases significantly and automated version control becomes very important.

    Modern version control systems support storing, tagging, branching, comparing and merging of multiple or many versions. The most recent generation of these tools such as git is based on the concept of de-centralized repositories and pull rather than push.

    However, IT administration UIs and consoles of ITOM (IT Operations Management) software often do not support out-of-the-box integration with git or other version control systems. They usually store objects in proprietary databases. Quick and easy integrations of version control with these UIs and consoles become elementary.

    Agile Metrics and Dashboards for ITO: Today, availability indicators such as MTBF (Mean-Time-Between-Failure) are the most commonly used performance indicators in ITO. However, to incentivize and motivate your teams towards more agility, it is important to combine them with agile indicators. Examples of agile indicators are:

    • The wait time of a requirement in the backlog until it is processed
    • The cycle time of a task through the development pipeline
    • The relationship of idle time to processing time for tasks in the pipeline

    Most agile collaboration tools make it easy to measure and extract this data. It is useful to

    1. Combine selected agile and availability indicators into KPIs; their relative weights depend on the context and the objectives of the organization
    2. Measure their current baseline
    3. Set realistic KPI targets
    4. Implement an automated measurement system
    5. Extract and visualize consolidated data through a dashboard
    6. Perform regular performance reviews or improvement meetings to analyze performance and performance trends, identify root causes for weak performance and measures to improve it

    The DevOps Production Integration toolkit will not solve all agility constraints in ops, but complemented by automation tools and other measures it can be a powerful answer to many DevOps challenges. It can be implemented locally in ITO and does not require a major replacement of tools or organizational re-design.

    Click here to learn more about implementing a seamless and efficient DevOps Product Integration ToolkitClick