devopsy.kinsta.cloud

Tag: CI/CD

  • The Essential Toolbox for DevOps Analytics

    The Essential Toolbox for DevOps Analytics

    With DevOps teams under constant pressure to improve software delivery cycles while maintaining—and improving—quality, many have turned to continuous testing (CT), implementing it as part of their overall development strategy. CT offers a number of benefits to DevOps teams, but it also introduces new challenges, specifically infinite amounts of data that require analysis. As more teams integrate CT into their strategy, it is clear that analyzing, understanding and filtering the results data quickly is critical to preventing bottlenecks in the DevOps process. The problem is, teams simply don’t have the time needed to analyze data.

    Current research indicates that teams spend anywhere between 50 and 72 hours per regression cycle analyzing test results. This includes filtering out noise and assessing failures that may impact their software releases. However, the amount of time DevOps teams have to review and qualify test results has shrunk dramatically in recent years—from days to hours or even minutes, in some cases—leaving teams scrambling.

    With pressure to accelerate velocity and deliver consistent, high-quality experiences, DevOps team require new analytics solutions to help them manage large test-results datasets in a CI/DevOps environment.

    Let’s explore five essential tools that enable DevOps teams to quickly and efficiently analyze data, triage issues and act upon failures with the best possible insights.

    Executive Dashboards

    Dashboards have evolved immensely over the years, allowing dev managers and QA managers to easily examine the pipeline, see CI trends related to time, build health and more. When looking for the best solution, make sure it includes quality heat maps and CI dashboards, which make it easy to spot an anomaly in the CI pipeline and allow managers, DevOps teams and practitioners to quickly drill down into the single test report and the issue.

    Single Test Report Visibility With Advanced Reporting Artifacts

    With a single test report, practitioners have access to the entire flow of test steps, video, logs, screenshots and Jira–even the ability to drill down into the source code. This makes it quick and easy to locate an issue. For teams, these means more time fixing and less time searching.

    Cross-Platform Reports

    These analytics reports show the UI/UX simultaneously across multiple screen sizes, resolutions and platforms. Since the boundaries between the digital platforms need to be seamless, whether on desktop browsers or mobile devices, having the ability to see the UI/UX on multiple form factors (and layouts) at the same time is essential to assess overall quality.

    DevOps Noise Reduction Tools Powered by AI and Analytics 

    Today, finding which error classifications are true application bugs is critical to efficiency. Noise reduction tools that are powered by AI allow teams to filter out problems caused by various issues that are not software defects—for example, device connectivity issues and wrong scripting. Having a tool with the ability to locate each failed test execution, the root cause and the classification category is a huge productivity boost for both the test engineer and the developer.

    DevOps and Actionable Insights

    Analyzing data is one thing, knowing what to do it with it is another. Today, there are a number of analytics solutions on the market that provide teams with both capabilities. Once an issue is detected, classified and reported, having deep insight brings practitioners halfway toward the resolution of the issue.

    To succeed in software delivery, teams not only need better automation as part of their processes, test flows and CI/CD workflows, but also an analytics platform to manage all of their test results data. Since DevOps involves a collection of team members from all parts of the software delivery lifecycle (SDLC) process, the central platform needs to meet the needs of all team members. As you work to build your next test analysis toolbox, consider the five features above to efficiently evaluate the data, act upon it and deliver iterations and features with confidence.

    — Eran Kinsbruner

  • Survey Finds CI/CD Adoption Has Long Way to Go

    Survey Finds CI/CD Adoption Has Long Way to Go

    At the KubeCon + CloudNativeCon Europe 2019 conference, Codefresh published the results of a survey of 130 developers that finds not only are nearly a third (32%) not using any type of continuous integration/continuous development (CI/CD) platform at all, but 60% said their organization has yet to take full advantage of automation to accelerate application development and deployment.

    In fact, according to the survey, it takes more than two weeks for two-thirds of developers (64%) to push a commit into a production environment. Additionally, the survey finds that for 39% of respondents, less than 10% of their company’s processes are automated from Git commit to code to production.

    Not surprisingly, only 19% of developers said they deploy code at least once daily. Roughly a third said they release code into a production environment only once a quarter. At that rate, it may be difficult for those organizations to justify investments in a CI/CD platform.

    Dan Garfield, chief technology evangelist for Codefresh, which develops a CI/CD platform, said that while much progress still need to be made in terms of spurring adoption of CI/CD platforms and best DevOps practices, the shift toward microservices-based applications inevitably will force the issue. The inherent complexity associated with building and maintaining applications consisting of hundreds of microservices based on containers, which regularly are ripped and replaced, will require all development teams to adopt a CI/CD platform. Right now, however, the shift to microservices is even less nascent than DevOps—21% of survey respondents said they are not employing cloud-native architectures because they are overwhelmed by the prospect.

    Garfield said once organizations more aggressively embrace microservices to increase the rate at which applications are developed and deployed, Codefresh is betting most will opt to standardize on a CI/CD platform based on Kubernetes. Most organizations have come to realize they will be building and deploying applications across multiple clouds. A CI/CD platform based on Kubernetes, such as the one offered by Codefresh, enables organizations to consistently apply the same set of best DevOps practices, he said Garfield. As such, competition among vendors that provide CI/CD platforms based on Kubernetes is sure to be fierce in the months ahead.

    In the meantime, a 10-year effort to convince organizations to employ best DevOps practices more broadly appears finally to be on the cusp of becoming mainstream. Organizations of all sizes have embraced DevOps unevenly. In fact, while most organizations have been able to achieve continuous integration with a handful of development teams, very few organizations are anywhere near close to achieving continuous delivery of application code within a production environment.

    The shift to modern applications based on containers, however, will require most organizations to standardize on a set of best practices across the enterprise. Pressure to deliver applications faster in the age of digital business transformation is mounting with each passing day. The only unknown at this point is to what degree will organizations be able to automate those processes.

    — Mike Vizard

  • ‘DevOoops’ Moves: Deploying Code Out of Step

    ‘DevOoops’ Moves: Deploying Code Out of Step

    ‘Everything as code’ can create opportunities for misadventure for DevOps teams

    DevOps’ ultimate aim is to create efficiencies in the software delivery process. But it doesn’t always work out that way. Sometimes DevOps initiatives lead to mistakes—something we lovingly call “DevOoops.” CloudBees recently polled some colleagues and came up with five examples of a DevOoops in action. Following is a discussion submitted by Director of Engineering Laura Frank Tacho about deploying code out of step.

    Scenario:

    Running in parallel during CI/CD can speed up your automated testing and shorten your feedback cycles. Just make sure deployment is not executing as a parallel step. That misconfiguration will cause your code to deploy at the same time the tests are running, regardless of their exit codes, pretty much defeating the purpose of automated testing before a deploy.

    This scenario touches on a theme we’ve seen throughout most of the “DevOoops” stories our colleagues have come up with. The theme is how the emerging concept of “everything as code” can create opportunities for DevOps teams to slip on proverbial banana peels.

    In this case, we’re talking about sequencing problems in a process we can call pipeline as code.

    To be successful, teams need to extend a quality-first approach in automated testing throughout their continuous delivery life cycle—from code to commit to test, as well as through the various tiers of the development life cycle. And with a quality-first approach, you need to apply testing and quality management to your environment configurations and infrastructure as code.

    Laura’s example highlights how teams need to treat their CD pipelines as applications themselves. They need to check for and eliminate bugs and errors in CD pipeline code to ensure that code is operating as expected. Failure to properly manage the creation and evolution of complex CD delivery pipelines will result in significant cost or risk.

    So, if you don’t have proper test coverage on application code, you could end up releasing an application that hasn’t been tested at all. If the app doesn’t pass through a security gate, as it’s supposed to, you could release unintended preproduction data.

    While there are always risks with untested code, we have to give unique consideration to untested or unmanaged CD pipelines. We need to consider the risks of what missed gates could mean for our developers, our organization and our customers.

    Teams should do the following as they release their CD pipelines:

    • Build: Start from the left-hand side of process and work through the defining and coding of CD pipeline left to right in the direction your apps and development artifacts flow: from dev to test to deployment. While we may not be able to define an ideal automation solution completely on the first pass, going left to right encourages teams to focus on what the life cycle of an artifact looks like.
    • Manage: Define your CD pipeline using an everything-as-code or pipeline-as-code approach. For this to add value, you must manage development of your CD pipeline as you would your application code and apply software best practices. Check your CD pipeline code into a source code repository and leverage language-specific technologies that can check your syntax. (If you’re using Groovy-based code or Jenkins-based code, there are versions of Lint that can perform these functions.)
    • Create staging environments: You should establish staging environments or test pipelines to validate your CD process. These pipelines should go through all checks and gates against your real tools, using reference code that is representative of the applications your pipeline would be servicing but stops short of production into environments.

    This will give you an opportunity to iterate changes against your application code, do some initial scans and checks to collaborate to define your CD pipeline in code-managing SCM. Execute initial validation on scans as pipeline as code and then validate expected operations in a staging environment before you deploy it as the basis of the pipeline for your production of business-critical applications.

    Key Takeaways

    • This story highlights how crucial it is to ensure that your application code and environments pass through certain gates and checks before being deployed out to the customer.
    • You need to construct your CD pipelines so your parallel testing is managed as a gate to production, ensuring that properly tested and validated changes are not automatically deployed.
    • Deployment happens at the end of all the processes within a pipeline, and all required tests, validations, checks and approvals have been done before any change is deployed to the customer.

    To read our previous “DevOoops Moves” post, click here.

    — Brian Dawson

  • Monitoring the DevOps Toolchain

    Monitoring the DevOps Toolchain

    DevOps has grown in popularity among many companies seeking greater agility and higher-quality code. When using different tools for automation, these organizations need monitoring and tracking of these tools and metrics from different portals on a common dashboard. Toolchain monitoring helps to bring all reports into a common portal.

    Enabling monitoring tools from a centralized platform can save a lot of effort and time. These tools have many in-build capabilities that bring value to the business, such as predictive alarming, graphical representation of data and integration with other tools including email, ticketing system and auto remediation. These added features become game-changers in today’s competitive SLA-driven and standard environments.

    Monitoring can be categorized in three ways:

    • Infrastructure monitoring: Monitoring the environment, machine and resources used to create a platform for an application.
    • Application monitoring: Monitoring the application’s performance, resource consumption, processing time and responsiveness.
    • DevOps monitoring: Monitoring the pipeline and toolchain from continuous exploration through release management. DevOps monitoring tools have the capability to monitor job health, job status, performance and quality.

    Why DevOps Monitoring is Essential

    In DevOps, we use many tools to create a toolchain and pipelines. In some big teams with a long release train, monitoring a project’s status from requirements-gathering to deployment can be difficult, as you have to track a number of different tools. And even then, you might not get the status in a desired format. DevOps monitoring tools bring everything to one single dashboard, where you can see the status and other metrics, including useful information about the project quality and progress.

    DevOps Monitoring: Not the Same as Tools Monitoring

    Tools monitoring is specific to the health and performance of tools; DevOps monitoring focuses on things such as the status of sprint, code quality, code branch status, test results and deployment status. All of which are required by project management for continuous improvement in productivity and quality.

    What to Monitor?

    There are a number of things we want to monitor in DevOps orchestration, and all these metrics provide a perspective of the current status of the project or product development and productivity:

    • Performance: Performance of the application.
    • Development (sprint or Kanban progress) and defects status from the application lifecycle management (ALM) tool.
    • Code repository monitoring for a project specific branch.
    • Build job or pipeline monitoring from the continuous integration (CI) tool.
    • Code analysis and security test results and reports.
    • Deployment and upgrade status in different environments.
    • DevOps maturity using the tools’ usage and quality data.
    • Artifacts (versioned artifacts).

    These are the basic and mandatory areas any project would monitor on a single dashboard.

    Monitoring Use Case

    In agile projects, CI/CD tools are implemented following the best practices of DevOps. An ALM tool is implemented to manage the backlog and development sprints. On this project a CI tool is used to build code and all the code is getting committed in a code repo, using a code review and analysis tool. A binary repository has been implemented to store all the binary of every version.  An automated deployment tool is being used to deploy the application in QA, staging and production environment.

    Now the struggle comes when monitoring each tool, the status of the build and the progress on different portals (CI tool, code repo, analysis tool, deployment tool, etc.).

    Using centralized monitoring tools can help project management to monitor the metrics mentioned below. These metrics give the overall picture of build, quality and deployment status on one dashboard.

    Available Tools for DevOps Monitoring

    Below are few available products that provide similar solutions:

    • Hygieia: Freeware tool that supports all DevOps tools, but metrics are very limited.
    • KlipFolio: Has lots of tool support, but it’s not free.
    • DevOps Insights: An IBM tool that supports IBM cloud platform only.
    • DataDog: Has a wide range of tool support, but it’s not free.
    • Prometheus: Freeware tool that monitors lots of tool-using exporters.

    Build vs. Buy

    Deciding which tool to use is always a tough task—you need to compare all the available tools and their offerings from different perspectives. Below are a few factors that play an important role when deciding on available options or to build your own solution:

    • Range of supported tools (plugins or data collectors).
    • Freeware or paid.
    • Support to customize data collection.
    • Rich UI for data representation and customization of fields and views.
    • Range of metrics supported.
    • Ease of installation and configuration.
    • Licensing and distribution policy.

    Solution Approach for Building Your DevOps Monitoring

    If you have decided to build or extend any existing tool, below are some considerations:

    • Identify the tools you have to monitor: List the DevOps tools you have to monitor; the list should cover all the areas of DevOps (build, repo, code quality, deployment, monitoring, etc.) to get the complete picture.
    • Identify the metrics: Once you have the list of tools, identify the important metrics from each tool. Your metrics should be capable of providing KPIs of DevOps areas.
    • Identify API or command-line interface to fetch the metric data: You need a good understanding of tools and their implementation to identify the best way to fetch the data. It could be predefined APIs, REST APIs or any command line interface. Make sure you are getting all the desired data.
    • Correlate the metrics with the business value or define the DevOps KPI: Your metrics should represent the KPIs and business value of all the tools involved in achieving DevOps. The dashboard should cover all the aspects and represent the current status and gaps.
    • Determine how to store and process those metrics: You need to define the schema to store the data in defined database; your schema should be generic enough to cater any data from different tools. To process the data, you need a middleware component between database and UI to process in a presentable format.
    • Build a scoreboard based on metrics: Use a rich GUI to display the scoreboard in different formats such as table, graphs and charts.
    • Support high availability: The tool should be designed to support high availability to avoid downtime and loss of data points.

    How Monitoring Tools Collect Data

    Most of the tools we use for implementing DevOps and CI/CD are web-based tools such as Jenkins, Sonar, Git, SVN, UDeploy, Gerrit, Jira, Confluence or others. These tools provide REST APIs, from which we get the data in a described format—either in json or xml. Some tools support predefined APIs or CLIs to connect and fetch the data. Once you identify the communication mechanism with the end system, the next step will be writing robust code that can fetch the desired metrics data and handle all the cases to parse the data and put it in a relational or non-relational database.

    Once you have the data in the database, you need a GUI to display those in tabular and graphical formats and a middleware component that can fetch the data from the database and provide it to GUI in the desired format.

    Getting the data from the system you want to monitor is the real challenge in this whole exercise, as everything else depends on it. Below are a few challenges you might face while developing a toolchain monitoring system:

    • Identify the API or any other mechanism to fetch the data.
    • Identify the metrics to monitor.
    • Identify how you want the data to be presented on the GUI—whether a trend chart, pie chart or just tabular presentation.
    • Determine the authentication mechanism for the monitoring system, as each supports a different kind. Sometimes it needs a predefined token while other times a token needs to be generated at runtime.

    Simple Architecture of DevOps Pipeline Monitoring

    Below is the simple generic architecture, on which most of the monitoring tools work. In the picture below, the different DevOps tools are on the left side; these tools are the end system from where we want to bring the data. Only a few tools have been shown as an example.

    A data collector brings the data from each end system and communicates with the end system using a REST API or any API exposed by that tool or a CLI, depending on the case. Using the defined communication mechanism, it will bring the data and pass it to the parser to fetch the required data and put it in the database in the desired format.

    While developing data collection, you need to have good a understanding of the tool you are going to monitor as well as available APIs including REST, CLI, SMIs or any pre-defined API built within the tool.

    The data parsing layer will fetch the desired metric value from the raw data collected by the data collector. Raw data could be in any format, including key value pair, json, xml or csv. The parsing methodology will differ in each case depending on the type of data. You can use any generic API for the parsing available.

    You must identify a good API to parse the data available in a different format and complexity.

    The database will hold the metrics data of each interval for each tool and user credentials.

    The GUI will request the data from the parsing component. It will represent it in tabular or graphical user-friendly format.

    Other features: Once we have data, we can have the trend chart to make some critical decisions, we can generate alarm on breaching any defined thresholds and integrate it with email or SMS to take immediate action.

    Monitoring Tool High Availability

    If we are relying on our monitoring tool to get reports of all the tools used in DevOps, we cannot afford downtime. We may lose data points and trends if the monitoring tool goes down, and management may not get the actual status of an entire project without having to gather the data manually.

    To avoid such situation, high availability (HA) should be implemented on the monitoring tool. In case of a HA setup, we will keep gathering the data in case the monitoring node goes down.

    To implement the HA, there are two aspects we need to consider: the HA of GUI and middleware (business logic component) and the HA of database.

    HA in the GUI and middleware could be achieved using an Apache or Nginix cluster. In the case of containerized environments, we can use any orchestration platform’s default feature to achieve high availability, such as Kubernetes. Similarly, in the database we can implement a clustered database feature provided by the most of available databases.

    Conclusion

    DevOps toolchain monitoring is very useful. It provides a good snapshot of the health of a project to management and is able to identify any blocker issues that might impact the release so they can be corrected. Monitoring can help identify pain points to focus on in defining the road map. Organizations following DevOps practices and using DevOps tools should have a tool to monitor the DevOps pipeline.

    — Prateek Kumar Asthana

  • CloudBees Allies with Atos to Accelerate DevOps Adoption

    CloudBees Allies with Atos to Accelerate DevOps Adoption

    In a move that should accelerate adoption of best DevOps practices in the enterprise, CloudBees and Atos announced this week that the two are collaborating to advance adoption of continuous integration/continuous development (CI/CD) platforms on the Google public cloud.

    Announced at the Google Cloud Next 2019 conference, the alliance will result in the IT services giant delivering a managed instance of a CloudBees CI/CD platform running on the Google cloud, in addition to providing traditional consulting services.

    Francois Dechery, chief strategy officer for CloudBees, said the alliance with Atos will accelerate adoption of DevOps processes in traditional enterprise IT organizations as Atos consultants engage customers looking to modernize their application development and deployment processes. In fact, many of the principles first pioneered by proponents of agile development methodologies are now being applied to business process management, he noted.

    Atos plans to add a DevOps practice as a complement to its existing processes. Michael Kollar, senior vice president for cloud engineering, said the company expects that, for the foreseeable future, most of their clients will be trying to meld some degree of DevOps and existing ITIL-based approaches to IT services management (ITSM). The alliance with Google will go a long way to accelerating that transition because it will allow Atos to deliver access to a CI/CD platform based on a consumption model that reflects the way public cloud infrastructure is consumed, he said.

    Kollar noted enterprise IT organizations are under more pressure than ever to deliver applications faster as more organizations recognize they are a software company that happens to make something. Most organizations today need to create a superior customer experience through software to differentiate themselves from their competition, he said.

    Many of those companies will need help setting up the processes that will enable them to function as a software company, he added. The biggest challenge, however, is finding a way to meld new ways of building and deploying applications with investments in legacy systems on which the business still depends, he said.

    Naturally, Atos won’t be the only IT services firm to launch a managed DevOps process. But as more IT services firms invest in developing managed services, there will be need for a balance between the managed services provided by IT services providers and cloud service providers such as Google and the IT services that will continue to be managed and delivered by internal IT organizations. The degree to which organizations will rely on external IT services firms for DevOps will be dictated by how mature they are today. But it does not follow that an organization which sees a need to rely on external expertise today will be that position tomorrow. As more organizations develop internal expertise, a managed service provider will need to earn its seat at the table every time its contract comes up for renewal.

    — Mike Vizard

  • Apply Balance and Critical Thinking for Greater Agile Success

    Apply Balance and Critical Thinking for Greater Agile Success

    In an ideal world, implementing DevOps practices and Agile methodologies would be the key building blocks organizations need to realize rapid, high-quality releases.

    The ability to iterate on a product with a high volume, velocity and level of quality is the ultimate goal of the modern enterprise; every CEO wants a faster time to market. But keeping business practices agile is much easier said than done. The term “agile” is broad. While The Agile Manifesto does a good job of defining the concept, there are myriad technical and non-technical aspects to the idea. At the end of the day, simply saying you are “agile” isn’t some silver bullet that delivers faster cycle times, a better product and more satisfied users.

    Instead, success with Agile methodologies means thinking more critically about how they are deployed and if your current DevOps practices are strong enough to support that strategy.

    What is Agility, Really?

    In practice, agility is a complex combination of things, perhaps best understood by what it’s not. Being agile with a product is not simply thinking about how fast you can complete a collection of features. Yes, it’s great when the backlog becomes smaller, faster. But Agile, when applied correctly, is really about having a true understanding of the user’s business problem and a real idea of their envisioned, desired outcome—and what it might take to achieve that outcome.

    This requires a shift in thinking when it comes to the product backlog. Too often, backlogs are viewed vertically, meaning features are built that may not have independent value. As a result, release efforts grow in development effort and elapsed time, limiting the ability to quickly release in small, incremental stages. As an example, consider search functionality as a feature set. On its own, search isn’t necessarily an independently valuable feature. What are users searching for? Why do they need this information and what are they doing with the results? A vertical view of agility fails to ask these questions and focuses only on the myopic feature set at hand. Consequently, even the most mature Agile practices are limited in what they can accomplish. Who really cares if you can get a feature out of the backlog quickly if it doesn’t matter to users?

    Agile thinking should instead be applied horizontally; the user’s journey is the center of everything. What do they need to accomplish? What are they unable to do now and how quickly can we give them that ability? In the search example, this means thinking beyond simply building a search function and instead developing a new product release that eases a specific pain point of users. Quick and iterative changes based on direct user feedback is the goal.

    So Where Does DevOps Come In?

    Agile is hardly a foreign concept for software engineers—86 percent use an Agile methodology (such as scrum) day to day. But the tools, processes and practices of DevOps need to be honed constantly to support iterative development, testing and release of code.

    As with the development of any product, software developers have much to gain from building and testing functionality in small, iterative releases. Smaller tests allow for more intensive studies of accuracy and problem-solving before allowing an entire organization to rely on a solution. While these pilots may not be fully rolled out, the teams involved must be integrated to include voices from a variety of stakeholders, including but not limited to:

    • Architecture
    • Experience design
    • Development
    • Security
    • Quality assurance
    • Operations

    Teams must also keep these DevOps best practices front-of-mind when developing a software solution:

    • Employ microservices. A solution built from these separately deployable services that perform specific business functions is much easier to scale out or expand. Microservices also remove single points of failure (SPOFs) by preventing issues in one area of an application from affecting others.
    • Blend continuous integration (CI) and continuous delivery (CD). These two DevOps strategies support each other in much the same way as integrated human teams that consider stakeholder needs. While CI automates the build and testing processes, CD automates the deployment cycle from development to production. The automation behind both results in a vastly increased deployment frequency.
    • Use infrastructure-as-code. Modern cloud platforms allow for infrastructure to be software-defined. This allows DevOps practitioners to bring best practices from the field of software engineering to bear against systems challenges. Think about the new features AWS announced for CloudFormation this year, for example: configuration drift detection, dynamic references, macros (functions with iteration support) and linters. All of these are in support of treating infrastructure orchestration as a software engineering activity, allowing DevOps teams to use one common set of tools and practices.
    • Refine observability. As architecture becomes ever more complex, the practices of monitoring and logging are evolving. No longer can you independently monitor the uptime of a server and simply log OS and app issues. Application performance management (APM) becomes paramount, measuring aspects of the user experience as they flow through the app, as does identifying and publishing relevant business metrics. The distributed nature of microservices and the various cloud services enabling them, demand new tooling to measure potential bottlenecks in the form of distributed tracing. And the DevOps best practice of “hands-off production” (e.g. no shell or attachment access) means you’ll need to provide real-time visibility into the health and inner workings of your application via log shipping, adding your centralized logging solution into the critical path of application troubleshooting.
    • Focus on communication. This is perhaps the most beneficial point for your team and should not be overlooked. The cross-functional, collaborative teams supported by both Agile and DevOps culture break down traditional silos and unite members around a common goal. Formal structures around these teams should start early and continue through retrospectives to assess and celebrate everything you’ve accomplished in a short time. You’re not working in a vacuum, and this project isn’t the only one your team will ever produce. Keeping careful records of successes, failures and areas for improvement will only make your work more efficient and Agile in the future.

    Agile is the perfect process for the modern tech landscape—releasing quickly in small bites that are then perfected based on customer feedback. In living up to that ideal, teams who turn to Agile in managing their projects shouldn’t take a set-it-and-forget-it approach to their processes. Instead, bolstered by DevOps tools, culture and practices, they should embrace the ongoing refinement and reflection required to support both customer needs and business goals.

    — Adam Hanson

  • 5 Steps for Successful Continuous Testing Implementation

    Continuous integration (CI) calls for code to be always working. Continuous delivery (CD) and continuous deployment mean that we must be able to roll out a working configuration to production at any time. As DevOps becomes the norm, bugs in production are now a developer’s personal problem. Developers (and their managers) want to know immediately if code changes they are working on can break something.

    What is Continuous Testing?

    The only guaranteed way to prove code and configurations are working is to test them. That’s why we need continuous testing. In organizations making this transition, performance engineers and test automation experts shift from minding a backlog of test-building work, to facilitating a democratization of testing. Everybody builds their code and everybody tests their code, continuously.

    Continuous testing is becoming the “new normal.” In this modern and agile testing approach, developers are equipped with low-friction testing capabilities and tools and are expected to unit test any new or updated code. With testing shifted all the way left to developers in pre-production, and all the way right to post production monitoring, organizations are just seconds away from proving code is ready and working to acceptable service levels.

    Although continuous testing is becoming a more standard practice, it certainly presents unique challenges for execution.

    Here are five steps to help any company successfully implement a continuous testing policy:

    Define Tests Early

    Lack of clear requirements can result in wasted time and delays. Behavior-driven development (BDD), acceptance test-driven development (ATDD) and model-based testing using tools should be used so that all requirements are clearly documented and communicated.

    Test cases need to be clearly defined and test scripts created ahead of time, to enable continuous testing of code at all stages of production.

    Optimize Testing Processes and Test Coverage

    Save time, money and resources by testing only what you need to test. Visual models allow various paths to be explored and optimized so that only your tests provide maximum coverage, without unnecessary testing.

    It is also recommended to import test cases, remove duplicates and distribute and allocate optimized test cases in existing tools.

    Shift-Left Your Testing (and Shift It Right as Well)

    What do we mean by shift-left testing? Tests should be run earlier in the development cycle. Developers test as they go, with test automation including functional testing, performance testing, monitoring and security testing. This requires developer-friendly tools for easy adoption and use.

    By shift-right testing, we continue to test through production and post-production, including continuous monitoring. Gartner recently released a report on the shift-right testing, calling it a “must” for adopting DevOps practices.

    Provide Complete Test Environments

    The ability to provide virtualized test environments is critical to achieving continuous testing. Eliminate blocks and reduce wait times by providing complete test environments on demand, with dev-friendly tools (as code, CI/CD integrations, supported open source).  These environments should include test data on demand to ensure that teams can perform comprehensive tests with production-like data. These environments should be ephemeral; ready on demand, decommissioned after use.

    Get the Right Test Data

    The inability to get robust test data causes significant delays in many application release cycles. To accurately test new functionality, test data should be as close as possible to what the application will encounter in production. If the test data lacks certain real-world characteristics (i.e. actual fields, data specifications, negative scenarios, etc.), the tests are unlikely to find many potential issues or break the application where there are weak points. Understanding data models in order to extract the right data is a special skill in and of itself.

    Production data is clearly the best data to use for testing, but data privacy regulations and team silos often restrict its availability. The answer to this dilemma is a robust test data management tool that lets you copy production data and mask it to protect sensitive information, while maintaining the characteristics that make production data desirable for testing.

    Remember, continuous delivery and continuous testing are processes that can take time to implement successfully, but teamwork, continuous feedback based on data and the right tool set will help you on your journey. You can read more about implementing a continuous testing policy here.

    — Sushil Kumar

  • How CI/CD Can Fast-Track Next-Gen Media Rollouts

    How CI/CD Can Fast-Track Next-Gen Media Rollouts

    Using CI/CD can help digital service providers move as quickly as over-the-top content providers

    Online video consumption has been increasing tremendously with rapid change in consumer expectations to have seamless viewing experience across various digital devices. To capture this growing demand, it is critical for digital service providers (DSPs) to deliver fast-track rollout of next-gen media services. Moreover, to counter the growing influence of over-the-top content providers (OTTs), DSPs must deliver media services that are high quality, right the first time and consistent, along with creating competitive advantage through speed to market. However, unlike OTT players, DSPs are challenged by extremely long development and rollout timelines for new services and offerings, primarily because of vendor-specific hardware and software applications that do not support rapid changes.

    Figure 1: Rapidly changing landscape of consumer video consumption.

    Embracing an effective continuous integration and deployment (CI/CD) setup can enable DSPs to achieve the same service innovations and delivery agility that OTT providers are offering. An important characteristic of an effective CI/CD setup is to create a fully automated deployment that is easy to operate and visualize. The setup also should have remote accessibility to ease parallel deployment at multiple geographic locations and a robust monitoring toolchain to monitor each step in the deployment process.

    However, implementing a CI/CD setup with these characteristics for next-gen media landscape comes with its own challenges; it requires end-to-end CI/CD orchestration, which is operationally complex. This involves stitching together different types of video services such as live TV streaming, VoD, electronic program guide (EPG), 4K/HDR, user personalization and profiling. These services need to be rolled out in multi-geographic locations and multi-tenant environments, catering to multiple vendors. The service must offer a seamless viewing experience across all digital platforms including TV, mobile, tablet and desktop.

    In setting up such complex processes, cross-functional and cross-geographic teams own different parts of the process, which can create gaps in design, infrastructure setup, configurations and eventually deployment. The CI/CD pipeline created can be costly and ineffective.

    Setting Up Effective CI/CD

    One-click automated delivery pipeline tool for predictable rollouts/rollbacks

    Create a one-click automated delivery pipeline tool that can facilitate automated change requests (CRQ) to initiate CI/CD jobs. The tool triggers the entire deployment of all the app components in the respective tenants across multiple geographies.

    Key considerations in setting up a one-click automated delivery pipeline:

    • Track CRQ approvals over systems such as JIRA to trigger the automated delivery pipeline.
    • Build pipeline as a code with intermediate CI/CD stages using open source tools such as Jenkins.
    • Design a master job to accept component and configuration details. This performs multi-job configuration and triggers contextual deployment/rollback simultaneously across tenants.
    • Interconnect different aspects of networks, infrastructure, components and applications to run the entire process smoothly.

    Setting up an automated delivery pipeline provides greater accuracy and predictability by reducing the human error factor.

    Automated snapshots, pre- and post-validation scripts to ensure consistency

    Create a fully automated snapshot management process that includes automated snapshot creation, rollback and deletion across respective tenants. Automated snapshots before deploying the new version of application facilitates rollback to the previous consistent version with zero downtime, if any failure occurs.

    Ensure automated multiple validation checks at every deployment stage to ensure consistency. These checks include validation of network connectivity, memory usage, CPU utilization, running applications, error and debug log files, license files, binary file size and configuration files. These validations can be done via tools such as groovy, ansible, Jenkins, python or shell scripts with proper libraries that match industry standards.

    Auto-scaling tool to optimize resource usage

    Ensure the application always has the right amount of capacity to handle traffic demands. This requires integrating an auto-scaling tool with predictive machine learning (ML) models and change request process to dynamically scale up/down the infrastructure.

    Key considerations in setting up auto-scaling tool:

    • Identify external and internal influencing factors to analyze system throughput.
    • Leverage insights derived from a monitoring dashboard to manage changes in current payloads.
    • Leverage triggers from a ML model to plan future expected system throughput.
    • Create automated configurable scripts to support dynamic scaling based on the inputs received from change request and ML model.

    Creating such tool enables better infrastructure availability with efficient cost management.

    Tool-agnostic and on-demand environment creation framework built on open source

    Leveraging open source tools provide flexibility and control over the entire exchange-to-exchange deployment process. Using customized framework with a layered approach makes the delivery pipeline tool-agnostic. It gives flexibility to easily replace/plug in any tools in the DevOps chain without much dependency.

    This ensures increased reusability, as the same toolchain can be used for multiple environment setups. Further, a customized framework dynamically handles the resources and utilizes the infrastructure effectively.

    Service assurance dashboard for stability and monitoring end-to-end processes

    A service assurance dashboard can provide glimpse of end-to-end CI/CD performance in a single window. It provides real-time graphical insight into failures/blockages in the pipeline, highlighted in the context of the overall flow. The dashboard should dynamically fetch data from the deployment pipeline for respective components and monitor how the upgrade/rollouts are processing. It should provide insight into resource usage, which can trigger dynamic auto-scaling and give a visual comparison of previous firmware with the new one.

    Implementing such a service assurance dashboard can provide greater visibility and help in easy diagnosis and error prevention.

    Benefits

    • Accelerated time to market with nearly 60 percent reduction in waiting time with parallel deployment across multiple tenants.
    • Improved predictability with significantly reduced fallouts and downtime/failure rates.
    • Optimized cost with approximately 50 percent lower IT spends in the deployment process and resource usage.

    This article was co-authored by Sumit Thakur, manager, strategic insights at Prodapt; Vijay Ramachandran, senior project manager at Prodapt.

    — Lakshmiprasad A

  • ‘DevOoops’ Moves: SCM Conflicts

    ‘DevOoops’ Moves: SCM Conflicts

    DevOps’ ultimate aim is to create efficiencies in the software delivery process. But it doesn’t always work out that way. Sometimes DevOps initiatives lead to mistakes—something we lovingly call a “DevOoops.” CloudBees recently polled some colleagues and came up with five examples of a DevOoops in action. Following is a discussion submitted by solution architect Will Refvem about siloed departments.

    Scenario:

    One of the biggest challenges in DevOps is SCM. Early in my career, being given access to Git repositories was a bit like handing a belt-fed machine gun to a drunken toddler. My biggest DevOoops moment came when I was trying to master the basics of Git in a project that used Git submodules—don’t, just don’t—and had a series of bash scripts that implemented a primitive CI/CD workflow by running in the environments we were deploying to (seriously). I can’t tell you how many merge conflicts and detached HEADs I encountered, or how I got out of them. Fortunately, we were using the fork-PR model, so I did no real damage.

    Software configuration management (SCM) is key not only for versioning, but also for providing a shared view of code and other assets, from development through to production. Giving proper consideration to your SCM structure, branching and merging model, access controls, etc., will support collaboration, reliability and traceability—all key for DevOps success.

    In the 2019 DevOps and Jenkins Community Report, 66 percent of organizations respondents reported the use of modern solutions SCMs, specifically GIT-based solutions GitHub or BitBucket. But the older SCM solutions aren’t going away completely—45 percent of all organizations still use pre-GIT, pre-DVCS (distributed version control systems) such as Perforce, Subversion and CVS. While it’s possible to apply DevOps practices using these legacy solutions, it is important (and more challenging) to design a branching/merging model and surrounding workflow, which supports the key aspects of DevOps such as peer review, collaboration and short-lived branches.

    Modern SCMs such as GitHub have features that inherently support collaboration, including the game-changing Pull Request, making it much easier to implement processes that best support CD and DevOps.

    Poorly designed CD workflows or inadequate CD systems create friction multiple ways. If the interactions are too cumbersome, users will not commit code to the pipeline frequently enough. If it’s difficult to access the pipeline and create visibility into the process, it can lead to unresolved conflicts.

    You need to pick the right tool. If you don’t have the option to get the modern tool, you need to take the time to design a legacy tool.

    It’s also important to consider two other issues in relation to this DevOoops example: branching and acquiring the appropriate level of DevOps skills.

    • Branching: A proper branching and merging model is required to ensure you are feeding the appropriate inputs into your DevOps pipeline, such that you are able to continuously integrate, test and ultimately deploy changes. A simple guide here is encapsulated in the KISS (keep it simple, stupid) principle. If you need a full-wall whiteboard or a spreadsheet fit for an accountant to list your active branches, you need to rethink things. Get as close to the ideal of mainline trunk development as possible, with short-lived feature branches, minimal application versions, etc.
    • Skills: As organizations seek to implement DevOps, most likely new tools and practices will be implemented simultaneously. It’s important to understand and accommodate the fact that there will be bumps and bruises as you learn to use these new tools and practices. For instance, shifting from centralized version control such as Apache Subversion (SVN) to distributed version control such as Git is a paradigm shift that can be difficult to fully grasp and result in destructive mistakes, such as deletion of nodes from the change history, overwriting changes or misconfiguration access control. Consider hiring for these new skills and/or account for time and training.

    Takeaways

    • A proper branching and merging model are required to ensure that you are feeding the appropriate inputs into your DevOps pipeline to let you continuously integrate, test and ultimately deploy changes.
    • If you are using a pre-Git legacy SCM solution, develop a branching and merging strategy within the legacy solution to achieve appropriate velocity in your DevOps model.
    • Consider that you will need to develop new skills to adopt new SCM tools and practice and account for this in your plan.

    — Brian Dawson

  • Digital Transformation Technology and Process: More Similar Than You Think

    Digital Transformation Technology and Process: More Similar Than You Think

    Right around this time of the year, the gym starts to get a little less crowded. Throughout January, it’s packed to the gills with folks freshly motivated by their New Year’s resolutions to get in shape. But by mid-February, the commitment seems to have all but dissipated.

    Why is it so hard to just commit?

    During my last workout, it occurred to me that the set of vacant treadmills next to mine was eerily reminiscent of the state of enterprise DevOps. According to Forrester, 50 percent of digital transformation efforts stalled in 2018. It seems commitment and follow-through are difficult in both personal and professional arenas. Admittedly, adopting and scaling DevOps is no easy feat. While reflecting on what does work, I found some parallels between the transformation process and DevOps itself.

    What Success Looks Like

    According to DORA’s “2018 State of DevOps report,” organizations with the highest level of DevOps execution, as measured by throughput and stability, also achieved the highest level of business performance in terms of profitability, market share and customer satisfaction. In a true case of going fast without breaking things, elite performers realized impressive performance gains:

    • 46 times more frequent deployments (throughput)
    • 2,555 times shorter lead times (throughput)
    • 7 times lower change failure rate (stability)
    • 2,604 times shorter time to recovery (stability)

    These metrics underlie a key tenet of DevOps transformation, which is breaking up large chunks of work into many small pieces.

    Shrinking the Change

    Smaller changes mean getting work out faster, with less risk. They take less time to plan, less effort to execute and are easier to troubleshoot. But, smaller code changes alone aren’t meaningful unless they are coupled with shorter release cycles. In essence, agile without CD doesn’t work.

    Stockpiling a few months worth of agile code iterations into a single monolithic release reintroduces an enormous amount of risk and negates much of the work done to adopt agile software development in the first place. Big queued-up releases often mean someone is stuck working the weekend to iron out all of the things that inevitably go wrong and no one should be stuck with weekend release duty. As Jez Humble put it, “The most important metric for continuous delivery is the number of person hours spent outside of business hours doing releases… and the goal for that is zero.”

    In practice, CD is what brings agile development to fruition; it’s why Gartner finds that 40 percent of IT departments have already invested in CI/CD with a full third planning to invest in 2019. CD is all about small pieces. In the same way agile enables smaller code changes, CD enables smaller releases.

    Not Just Smaller, but Iterative

    At GitLab we use the phrase, “just commit” as a shorthand for continuous delivery and deployment. The idea is that when you have a completely automated CI/CD pipeline, all a developer needs to do is commit code to their branch, and everything else needed to get that code running live in a production-like environment—from containerization to security testing—is automated. Similarly, getting code to production is as easy as merging the feature branch back into trunk with the entire production deployment automated. The message to the developer is, “Just commit your code and let your DevOps tooling do the rest.” This is CD in a nutshell.

    With this type of automation in place, the cost of making a change significantly decreases. This means the cost of getting started is very low, but so is the cost to keep going. Instead of spending a massive effort to map an entire path to your solution, you simply need to figure out your next step and keep iterating. According to GitLab’s “2018 Global Developer Report,” 67 percent of developers report the top challenge to getting work done is unclear direction. By shrinking the change, business leaders will find it easier to set direction for the next step without having to map the entire path. This brings us back to digital transformation.

    Commit to Digital Transformation

    Just like a monolith being decomposed into microservices, the practice of breaking large chunks into small pieces applies to driving digital transformation as well. Like Creighton Abrams said, “When eating an elephant, take one bite at a time.”

    How can enterprises shrink the change?

    For most organizations that achieve successful digital transformation, the initiative usually starts with a single team or innovation unit. This group proves the success for the rest of the organization. It develops that culture, tools and processes that other parts of the organization can then adopt. One example of this principle in practice is the transformation that Chris Hill, Head of Systems Engineering, Infotainment, led at Jaguar Land Rover. When Chris came on board, only three build engineers had the ability to release to production and it took four to six weeks to complete each release. Chris gradually scaled a CI/CD platform, step-by-step over two years, until every team across the organization was able to complete a release in less than 30 minutes. Today, the Jaguar Land Rover completes 50 to 70 deployments per day. This iterative pattern is seen over and over again with enterprise digital transformation. It’s encouraging because it means successful digital transformation can be driven by either executives top-down or engineers bottom-up.

    Keep the Fuel Burning

    According to Argyle, “91 percent of companies have at least started on digital transformation, but only half have created an organization-wide strategy.” This seems to align with Forrester’s metrics on half of the organizations that stall.

    So, how can organizations keep going and avoid the stall out?

    If I can make a confession, I am also one of those New Year’s resolution makers who just started working out in January. How it is that I’m still hitting the gym while others have dropped off? I have found that taking a “DevOps” approach is a key to my success. Instead of broad sweeping changes, I make small incremental improvements one day at a time.

    It turns out the strategy for getting started—“just commit”—is the same strategy to keep going.

    By starting small with a single project and gradually expanding scope in iterative steps, an organization can continue to improve over time at a rapid pace. Small, single-commit code changes coupled with CD increase throughput and stability. In the same way, small, incremental process improvements bring about the greatest success with digital transformation.

    — William Chia