devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

  • Buildkite Tightens CI/CD Integration With GitHub

    Buildkite Tightens CI/CD Integration With GitHub

    Buildkite has tightened the integration between its namesake continuous integration/continuous delivery platform and the GitHub source code repository.

    Fred Showell, a senior product manager for Buildkite, said the integration will accelerate application development efforts using the Buildkite application in GitHub. Any connected GitHub account can be selected as a source when creating Buildkite pipelines, he noted.

    Repository webhooks can also be created automatically, making it faster to kick off builds on a Buildkite pipeline from pull requests and then feed build information back into GitHub using the Status application programming interface (API).

    The bulk of organizations that employ Buildkite are software vendors that are continuously updating software-as-a-service (SaaS) applications. Most of those organizations routinely employ software artifacts that are stored in the GitHub repository.

    While GitHub continues to expand its DevOps capabilities in lockstep with its parent company Microsoft, Showell said there’s plenty of opportunity to extend capabilities of frameworks such as GitHub Actions, a set of APIs that DevOps tools can invoke to automate workflows.

    In general, Showell noted the number of SaaS-based platforms continues to expand as organizations embrace cloud applications to drive digital business transformation initiatives. Some of those efforts are now also expanding to include use cases involving edge computing initiatives, which is expanding the reach and scope of the Buildkite platform, he said.

    Of course, many enterprise IT organizations are now applying many of the same best DevOps practices employed by SaaS platform providers to their own application development efforts. Many of those organizations are realizing they are essentially software companies that also manufacture something or provide a service within the context of a specific vertical industry. As such, the number of organizations embracing best DevOps practices should substantially increase in the months ahead.

    Most of those organizations are already trying to manage a large portfolio of applications. Digital business transformation initiatives may allow them to in time retire some of those applications, but chances are good legacy applications will need to be maintained for some time until the digital process that has been put in place proves effective. It’s likely then that sound software development lifecycle management practices will be more relevant than ever.

    Competition among providers of CI/CD platforms to provide those capabilities is already fierce. Fueled by massive amounts of capital, providers of these platforms are expanding the scope of their offering to address a market that is expected to be valued in the tens of billions. Buildkite itself raised $28 million in additional funding last summer.

    It’s too early to say how the next phase of the DevOps platform wars might play out. However, it’s apparent that when it comes to DevOps platforms the level of investment required just to keep pace is substantial.

  • CircleCI Report Finds DevOps Teams Adjusting to New COVID-19 Normal

    CircleCI Report Finds DevOps Teams Adjusting to New COVID-19 Normal

    An annual State of Software Delivery report published today by CircleCI suggests that after a spike in developer activity last spring at the beginning of the COVID-19 pandemic, the pace at which code is moving through a continuous integration/continuous delivery (CI/CD) platform has returned to normal levels even as developers continue to work from home.

    CircleCI CEO Jim Rose said after any initial rush last spring, it appears maintaining that level of productivity is simply unsustainable for most organizations. If heroic measures are continuously required, Rose noted it’s only a matter of time before DevOps teams start to burn out.

    Based on an analysis of 55 million data points gathered from 160,000 projects spanning more than 44,000 organizations, the CircleCI report finds that despite the ability of a handful of organizations that can deploy code multiple times per day, most organizations on average run workflows less than once (0.7 times) per day.

    However, Rose noted, there is too much focus on the number of times per day code is being deployed. The more important metric is duration. Half of the 11 million workflows measured by CircleCI finished in less than four minutes. Three-quarters finished in less than 11 minutes, with 95% of all workflows measured finishing in less than 35 minutes.

    Rose said durations are longer than in previous years, which suggests more testing is being included in workflows. In an ideal world, Rose said workflow durations should be in the range of five to 10 minutes.

    The report also finds 50% of the workflows went from failing to being successfully deployed in 55 minutes on average, with the median success rate 61% of workflows. Splitting the workflows by the ones run on default branches versus the ones run on non-default provides more insight, and the median rates are 80% for the default branch and 58% for the non-default branch. Failure rates for the non-default branch are higher because that’s the branch most developers employ to experiment in a way that doesn’t impact the primary workflow.

    Rose said the report also suggests there is a correlation between larger team size and better overall engineering performance. In times of uncertainty, IT leaders should consider increasing the size of teams to reduce burnout and maximize flexibility. Depending on the scope of the project, Rose said the CircleCI data shows that somewhere between five and 20 code contributors is the optimal number of team members. IT organizations would also be well-advised to consider how those team members are geographically distributed to maximize the number of hours in a day any project is being worked on without having to overwhelm a small number of developers working in the same time zone.

    As has always been the case, DevOps success over the long term requires organizations to balance multiple factors and metrics. After all, the primary goal for most organizations is not so much to increase the amount of code being deployed as much as it is to increase the quality of the code that ultimately gets deployed in a production environment.

  • GitLab Makes Building Secure Mobile Apps Easier

    GitLab Makes Building Secure Mobile Apps Easier

    GitLab has made 60 enhancements to its namesake continuous integration/continuous delivery (CI/CD) platform to address a range of things from DevSecOps processes to collaboration, at a time when most DevOps teams continue to work remotely.

    With help from external contributor support for the Mobile Security Framework (MobSF), version 13.5 of GitLab adds an all-in-one tool for penetration-testing, malware analysis and security assessment that runs both static and dynamic analysis of code running on Android, Apple iOS or Windows platforms. GitLab is also making available a user interface to configure its existing static analysis security tools.

    On the collaboration front, GitLab has added support for Group Wikis to provide a central place to collaborate as well as an enhanced Snippets tool to facilitate code-sharing among group members. Snippets with multiple files are now supported inside a single Snippet.

    GitLab is also making available additional templates for deploying to Amazon Web Services (AWS), as well as a GitLab CI/CD template for Terraform and integration with Gitpod, a tool that makes it easier to spin up development environments.

    Finally, GitLab has made feature flags a core part of its platform and added support for a Timeline view to make it easier to navigate incident discussions. It also included the ability to store other binary types in GitLab that are not yet supported via raw package feeds and attach binary assets to releases.

    Brendan O’Leary, senior developer evangelist for GitLab, the COVID-19 pandemic has accelerated digital business transformation initiatives that often depend heavily on secure mobile computing applications that rely on rapid development. As the DevOps teams tasked with building these applications are working from home to help limit any spread of the pandemic, the need for tools to facilitate collaboration across distributed development teams has become critical, he said.

    It’s not clear to what degree the need to work from home is influencing the selection of CI/CD platforms just yet. The trend toward working remotely was well underway prior to the pandemic. At this juncture, however, it’s clear most development teams will be working remotely well through 2021. Most IT organizations will need to revisit some early assumptions about how to optimize DevOps practices across a distributed workforce now that many of them have had firsthand experience, noted O’Leary.

    The challenge is some developers may thrive while working from home while others struggle. In the absence of collaboration tools, it becomes more challenging for managers to ascertain what specific issues need to be addressed. Of course, even if a vaccine is discovered. there will be no going back—the way applications are built and deployed in a modern enterprise has been forever altered. The issue now is determining how best to move forward as the COVID-19 pandemic continues waxes and wanes.

  • Progress Reorganizes Chef Portfolio

    Progress Reorganizes Chef Portfolio

    Progress today announced updates to the portfolio of DevOps offerings it gained with the acquisition of Chef earlier this month.

    A Chef Infrastructure Management offering now combines Chef Infra and Chef Automate into a single offering, while Chef App Delivery combines Chef Habitat, Chef Infra and Chef Automate into a single offering that creates an artifact that can run anywhere.

    Sundar Subramanian, senior vice president and general manager for core products at Progress, said the goal is to organize the Chef portfolio around a set of distinct offerings for automating the management of infrastructure and continuous delivery (CD). Instead of simply offering a set of tools for power uses, Subramanian said Progress is moving toward creating a set of platforms.

    At the same time, Progress announced a dedicated content delivery service that provides access to a library of curated Chef Premium Content for compliance scanning and remediation that complies with the best practices defined by the Center for Internet Security (CIS).

    Finally, Chef has added a Learn Chef Skills Validation Program based on the online Learn Chef platform and the OpenBadges Standard, a specification designed to promote certifications that teach skills in a way that are both verifiable and portable.

    CD and infrastructure management are typically handled by different members of the same DevOps team. By segmenting its offering around federated responsibilities, the Chef software portfolio will become easier for DevOps teams to consume based on specific tasks, Subramanian said.

    Progress expects its offerings in the months ahead to have greater appeal among midmarket IT organizations that are accelerating multiple digital business transformation initiatives in the wake of the COVID-19 pandemic, he noted. The company has been focused on providing application development tools to these organizations for decades, and as their IT operations mature, many are beginning to embrace best DevOps practices as they seek to reduce the total cost of IT operations.

    The challenge those organizations face today is they depend heavily on ITIL-based frameworks for managing IT. Beyond acquiring DevOps tools and platforms, most of those organizations are at the beginning a what has been shown to be a transformation to DevOps processes that requires major changes to the culture of an IT organization. The cultural transition often involves years of effort as organizations try to balance different approaches to managing IT based on ITIL and DevOps processes.

    It’s too early to say what path to automation of IT those organizations will take. The Chef tools still require a level of programming expertise to employ that is often lacking among traditional IT operations staff. Not to mention the fact that Progress is not the only DevOps platform provider looking to expand its reach into the midmarket.

    Regardless of the path chosen, the number of organizations that embrace DevOps principles is about to expand significantly.

  • Harness to Apply AI to DevOps

    Harness to Apply AI to DevOps

    Harness, at its {Unscripted} 2020 conference today, announced its plans in the fourth quarter to make available as a beta a module that leverages machine learning algorithms to optimize build and test cycles on the Harness Continuous Integration (CI) Enterprise platform.

    At the same time, Harness is adding a beta of a Continuous Features module to enable DevOps teams to employ feature flagging to simplify the application development process. A visual dashboard for feature verification and analytics monitors all flags, including active, inactive, live and unrequired flags, as well as customer issue tickets, to simplify the management of new capabilities being added to an application.

    Harness is also updating its continuous delivery (CD) platform to include a revamped user interface, GitOps and pipelines-as-code processes, bi-directional synchronization and conflict management, and templates to standardize deployments across teams.

    Finally, Harness is adding a Harness Next Generation Continuous Verification tool to provide visibility into the impact changes have on IT environments. This tool can be deployed standalone or used in conjunction with any CD platform and is intended to make it easier for DevOps teams to discover the root cause of issues that typically stem from recent changes made to an otherwise stable IT environment.

    Harness CEO Jyoti Bansal said artificial intelligence (AI) in the form of machine learning algorithms are poised to play a major role in taking DevOps automation to the next level. The Harness AI module can reduce test cycle time by up to 75% by correlating and isolating tests to changed code rather than requiring all tests to be executed with every change. The Harness approach means only relevant test cycles are run, he said.

    Continuous Integration Enterprise will also be able to identify gaps in test plans by highlighting code changes that are not covered by existing test cycles. Library caching and container image size optimization will also be added in the future to make builds more efficient.

    Harness also announced its intent to apply machine learning algorithms more broadly. Algorithms will enable DevOps teams to control, optimize and accelerate the flow of software pipelines rather than requiring engineers to manually verify builds, tests, deployments and releases. Harness will also apply data science techniques to observe the quality, performance, cost and reliability of code as it flows through each pipeline step and, if need be, automatically abort or roll back the delivery of software.

    As each build is committed, the opportunity to collect even more metadata about the workflow process only increases, Bansal added.

    Obviously, Harness is not the only provider of a CI/CD platform looking to apply machine learning algorithms. Thanks to machine learning algorithms, the rate at which applications are developed and deployed could increase by several orders of magnitude in the months ahead. The issue now is to what degree the rest of the business will absorb that potential rate of change.

  • Opsera Simplifies Building of DevOps Pipelines

    Opsera Simplifies Building of DevOps Pipelines

    Fresh off raising $4.3 million in funding, Opsera today launched a namesake platform that enables IT teams to orchestrate both the tools employed by developers as well as the pipelines that make up a DevOps process.

    Company co-founder Chandra Ranganathan said the Opsera platform automates setup of DevOps pipelines using a declarative approach that doesn’t require DevOps teams to standardize on a specific set of tools.

    Ranganathan said the Opsera platform also acts as a repository for pipelines that can be reused by DevOps teams. To facilitate that effort, Opsera also includes a set of pipelines that DevOps teams can employ immediately, he said.

    Opsera is designed to be a layer of abstraction that can be deployed across multiple continuous integration/continuous delivery (CI/CD) platforms, he said, noting that capability makes it possible for IT teams to reuse pipelines without having to standardize on a specific CI/CD platform.

    Developers, meanwhile, can still choose among any number of third-party tools that have been integrated into the Opsera platform. The goal is to automate what currently is a manual process for setting up DevOps environments, which often can take months, in a few hours using a declarative process, said Ranganathan.

    Once deployed, the Opsera platform also provides access to analytics tools that enable DevOps teams to optimize those workflows, he added.

    While much progress has been made in terms of accelerating application development thanks to the rise of DevOps, the setting up and maintaining of a DevOps platform often requires a dedicated IT operations team in part because each development team wants to continue to use the tools they prefer. However, platforms that automate the DevOps process have thus far required IT organizations to standardize on specific development tools. The Opsera platform reduces the size of the IT staff needed to support DevOps workflows without forcing developers to employ a specific set of tools, Ranganathan said.

    Each development team is also automatically given its own virtual data center environment to ensure isolation between projects, he noted.

    In the wake of the economic downturn brought on by the COVID-10 pandemic, the need to automate IT processes has increased sharply as organizations look to reduce the total cost of IT. DevOps processes are no exception. In fact, DevOps processes by design should lend themselves more to automation than legacy waterfall-based approaches to application development. As DevOps processes become more automated, it also should free additional resources to develop applications faster.

    Regardless of the motivation, automation of DevOps processes will need to become more automated to keep pace with increasingly complex IT environments. Development teams are increasingly being required to build and then regularly update hundreds—sometimes thousands—of microservices. As the tools for building those microservices become more diverse, the DevOps team needs to find a way to embrace tool innovations without requiring a DevOps platform to be entirely reconstructed. Striking that balance requires a level of abstraction that minimizes the impact any change in tooling preferences can have on the underlying CI/CD platforms.

  • JFrog Adds Free DevOps Tier in the Cloud

    JFrog Adds Free DevOps Tier in the Cloud

    JFrog announced today it is making a free tier of its namesake DevOps platform available on all three major public cloud service providers.

    Stephen Chin, senior director for developer relations at JFrog, said a free subscription to JFrog Platform running on Amazon Web Services (AWS), Microsoft Azure and Google Cloud Platform (GCP) is now generally available across 18 cloud regions. DevOps teams are provided with up to 2GB of storage, 10GB of monthly data transfer and 2,000 CI/CD pipeline minutes per month free of charge.

    The subscription spans JFrog Artifactory, an integrated software package management tool for binaries and container registry; JFrog Xray, a vulnerability scanning tool; and JFrog Pipelines, a continuous integration/continuous delivery (CI/CD) platform.

    Chin said the best way to further DevOps adoption is always going to be to make it as easy as possible to get started. There is no substitute for hands-on experience, he said.

    Fresh off raising additional capital via an initial public offering (IPO), JFrog is looking to expand its customer base. JFrog has more than 5,800 customers currently. The free tier will make it easier for JFrog to achieve that goal in a way that allows a nascent DevOps team to upgrade to a paid tier as their project scales, he said.

    It’s not clear to what degree organizations that have already embraced DevOps might be willing to replace their CI/CD platform or, in some cases, consolidate the multiple CI/CD platforms they already employ. However, as the number of organizations that are embracing best DevOps practices continues to grow, there appears to be plenty of opportunity for providers of CI/CD platforms to expand as organizations move to accelerate various digital business transformation initiatives in the wake of the economic downturn brought on by the COVID-19 pandemic.

    Less clear as the competition among DevOps platform providers heats up is the degree to which the total cost of DevOps could decline. Most IT organizations are under more pressure than ever to contain costs even as they are being required to accelerate software development. The challenge is striking a balance between open source tools and platforms that IT teams need to deploy and support versus relying on DevOps platforms that are managed on their behalf. In theory, the fewer IT staff dedicated to managing infrastructure, the more money there is to hire developers.

    DevOps adoption historically has been driven from the bottom up within most organizations. However, as more organizations appreciate the importance of software within their organization, senior IT and business leaders are revisiting software development processes. As part of those efforts, more organizations are looking to standardize on a specific set of DevOps platforms across their entire organization regardless of what cloud platform they may be deployed on.

    The challenge, is as often the case with any journey, is making that first step.

  • Feature Flags: The Cure for DevOps Remote Work Stress

    Feature Flags: The Cure for DevOps Remote Work Stress

    Feature flags can help reduce the stress associated with DevOps during WFH

    The world has drastically changed. As a result, we have had to adapt the way we shop, socialize, educate and work. Businesses have had to adapt to keep operations going, and the biggest change they’ve had to make has been the move to remote working. For most, this has meant bringing a laptop home from the office and setting up at a home desk or dining table. For teams used to collaborating in-person, it may feel like the world has turned upside down.

    DevOps teams have been thrust into a decentralized environment at the very time they are facing record traffic and the pressure to return to or even exceed their prior cadence. The current climate has accelerated demand for fast and efficiently running online services, such as e-commerce, streaming services and video conference platforms. Digital platforms not only are facing greater demand, but they are also seen as more mission-critical than ever.

    The stress of all this can be daunting, but it’s useful to remember that the very idea of DevOps emerged as a way to move faster with greater resilience. We need to trust the patterns and tools that were emerging as the “new normal” even before the current crisis emerged.

    One of those patterns, which is sometimes referred to as progressive delivery, is the idea of decoupling deployment from release. The tool many have chosen to accomplish progressive delivery, especially to meet the needs of decoupled and decentralized teams, is feature flags.

    Feature flags allow teams to deploy new features in production while restricting access to only a subset of users. The ability to control exposure without requiring further deployments or infrastructure changes in turn makes testing in production (exposing the feature to internal dev and test users only) and canary testing (exposing the feature to a small number of real users before ramping up to 100%) straightforward and repeatable processes.

    This “fewer moving parts” way of promoting features eliminates the need for the large-scale coordination that an all-or-nothing “big bang” release requires. That’s why feature flags are particularly well-suited for teams to work remotely, in decentralized environments.

    Feature flags allow for time to make sure everything is working properly before the switch is flipped. It is the equivalent of changing the plane while you’re flying, without the passengers falling. For this reason, DevOps teams that were using feature flags prior to global shelter-in-place orders were much better positioned to continue development efforts without interruption while working remotely. Feature flags allow for DevOps teams to operate in gradual stages with the ease of experimentation and rollbacks, significantly reducing the stresses of remote work.

  • CircleCI Brings Insights to CI/CD Platform

    CircleCI Brings Insights to CI/CD Platform

    CircleCI has added an Insights dashboard to its namesake CI/CD platform that makes it easier to surface metrics such as duration, mean time to recovery and throughput change over time as part of an effort to optimize DevOps workflows.

    The Insights dashboard makes it easier to discover, for example, which jobs are failing, which workflows are failing tests and which workflows are taking the longest to process. DevOps teams can also gain visibility into credit spent, throughput, success rate and mean time to recovery.

    CircleCI CTO Rob Zuber said that while it’s relatively easy for a single developer to ascertain issues with a build, the Insights dashboard shines a light on how an issue is creating a cascade of issues throughout a development project. The Insights dashboard also makes it easier for managers to track the interdependencies that exist between software projects, notes Zuber.

    After nine years CircleCI now has more than 1 million users employing its CI/CD platform. A recent report published by the company shows 80% of workflows running on its platform finish in less than 10 minutes. The company also reports that the adoption of its integration modules, dubbed orbs, has increased 200% since becoming available in 2018. There are now more than 1,900 orbs listed in the registry, with more than 58,000 organizations using orbs within their CI/CD pipelines. Strategic partners include Hashicorp, Plandek, Salesforce and others.

    Competition among CI/CD platform vendors is intensifying. It’s unclear, however, if CI/CD platforms will continue to expand in ways that subsume additional functions or if much of the focus is going to be on further integrating CI/CD platforms with other platforms and tools as part of a software development strategy that revolves around best-of-breed tools and platforms. Regardless of approach, more organizations will be embracing best DevOps practices to cope with software development projects that require more discipline to successfully build, deploy and maintain. The issue now will be determining which CI/CD option to standardize on.

    Overall, Zuber noted software development in the last few years has become more complex with the rise of microservices and containers. Complexity is increasing because the way one problem is solved tends to create others, he said. The challenge now is to find ways to provide more visibility into complex software development environments, as over time more artifacts and platforms span both monolithic and microservices-based application development projects, noted Zuber. In fact, that challenge is driving many organizations that have already embraced one CI/CD platform to reconsider their options as software development becomes more challenging.

    It may be a while before most organizations are able to exercise as much control over software development as they might like. However, the one thing that is certain is no one can manage what they can’t see.

  • How Logs Can Empower Your CI/CD Delivery Pipeline

    How Logs Can Empower Your CI/CD Delivery Pipeline

    Change is the currency of software success. This need to change and grow is the basis for continuous integration and continuous delivery (CI/CD). Each year, new industries embrace this philosophy of constant change, but without the necessary data underpinning their decisions, every deployment is a risk. When you enrich your software delivery with log data coupled with best practice log management, you mitigate this risk and enable rapid, safe change in your organization.

    Logs are often the most common source of information about an application’s behavior. They provide context as well as result, opening up far more information than a simple count metric will. Despite the immense insight that they can offer, organizations regularly overlook them. Many leave their logs in files on servers, far away from the CI/CD pipeline that needs them. It doesn’t have to be this way; with some best practices, every part of your business can collect, analyze and utilize the hidden knowledge within your logs.

    Add Life to Log Data With JSON

    Logging in an unstructured format dramatically increases the complexity of detecting patterns in your logs. By logging your application output in JSON, you open up the possibility of analysis of all of your logs. This gives you a top-down view of your entire system while still maintaining readability. This empowers your CI/CD pipeline by allowing you to query and filter your logs, which then enables you to focus in on the problem and precisely diagnose any unwanted side effects of your latest change.

    Create Actionable Alerts

    If alerts are defined and channeled correctly, have context and can be interpreted easily, they will be actionable, add context and offer greater value. It is difficult to provide context when a single metric exceeds a threshold, but log lines are more sophisticated. They can include additional information to provide context and allow us to pinpoint the source of the issue fast. This is an essential capability for a company embracing CI/CD.

    Prioritize Your Logs

    Structured logs often come attached with a severity, which indicates how seriously we should investigate an event. An INFO level log signals business as usual, but an ERROR demands immediate attention. Once your logs have these tags, you can make intelligent, automated decisions that will automatically respond to unwanted changes in your system.

    Benchmark Each Version to Understand ‘Normal’

    “Normal” is a complicated idea when working in a microservices architecture. It may be safe to ignore a minor slowdown of a single service, but an unfortunate combination of events can lead to disaster. Benchmarking enables us to see these disasters well before our users. Benchmarking is the act of recording what the normal behavior of an application is. After each deployment from your CI/CD pipeline, compare your new behavior with your previous benchmark. If something is out of the ordinary, logs provide the information you need to act decisively. They provide an outstanding baseline signal for your benchmark.

    Analyzed Logs Can Level up Your CI/CD Pipeline

    Building a CI/CD pipeline is not the most difficult part. Neither is the deployment of new features. In fact, the greatest challenge facing any organization wanting to deliver their changes via a CI/CD pipeline is observability. With a best-practice approach to the preparation, curation and analysis of application and system logs, we can overcome this challenge and confidently change at a pace that propels us to the forefront of our market.