devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

CircleCI Brings Insights to CI/CD Platform

CircleCI has added an Insights dashboard to its namesake CI/CD platform that makes it easier to surface metrics such as duration, mean time to recovery and throughput change over time as part of an effort to optimize DevOps workflows.

The Insights dashboard makes it easier to discover, for example, which jobs are failing, which workflows are failing tests and which workflows are taking the longest to process. DevOps teams can also gain visibility into credit spent, throughput, success rate and mean time to recovery.

CircleCI CTO Rob Zuber said that while it’s relatively easy for a single developer to ascertain issues with a build, the Insights dashboard shines a light on how an issue is creating a cascade of issues throughout a development project. The Insights dashboard also makes it easier for managers to track the interdependencies that exist between software projects, notes Zuber.

After nine years CircleCI now has more than 1 million users employing its CI/CD platform. A recent report published by the company shows 80% of workflows running on its platform finish in less than 10 minutes. The company also reports that the adoption of its integration modules, dubbed orbs, has increased 200% since becoming available in 2018. There are now more than 1,900 orbs listed in the registry, with more than 58,000 organizations using orbs within their CI/CD pipelines. Strategic partners include Hashicorp, Plandek, Salesforce and others.

Competition among CI/CD platform vendors is intensifying. It’s unclear, however, if CI/CD platforms will continue to expand in ways that subsume additional functions or if much of the focus is going to be on further integrating CI/CD platforms with other platforms and tools as part of a software development strategy that revolves around best-of-breed tools and platforms. Regardless of approach, more organizations will be embracing best DevOps practices to cope with software development projects that require more discipline to successfully build, deploy and maintain. The issue now will be determining which CI/CD option to standardize on.

Overall, Zuber noted software development in the last few years has become more complex with the rise of microservices and containers. Complexity is increasing because the way one problem is solved tends to create others, he said. The challenge now is to find ways to provide more visibility into complex software development environments, as over time more artifacts and platforms span both monolithic and microservices-based application development projects, noted Zuber. In fact, that challenge is driving many organizations that have already embraced one CI/CD platform to reconsider their options as software development becomes more challenging.

It may be a while before most organizations are able to exercise as much control over software development as they might like. However, the one thing that is certain is no one can manage what they can’t see.

October 6, 2020
How Logs Can Empower Your CI/CD Delivery Pipeline

Change is the currency of software success. This need to change and grow is the basis for continuous integration and continuous delivery (CI/CD). Each year, new industries embrace this philosophy of constant change, but without the necessary data underpinning their decisions, every deployment is a risk. When you enrich your software delivery with log data coupled with best practice log management, you mitigate this risk and enable rapid, safe change in your organization.

Logs are often the most common source of information about an application’s behavior. They provide context as well as result, opening up far more information than a simple count metric will. Despite the immense insight that they can offer, organizations regularly overlook them. Many leave their logs in files on servers, far away from the CI/CD pipeline that needs them. It doesn’t have to be this way; with some best practices, every part of your business can collect, analyze and utilize the hidden knowledge within your logs.

Add Life to Log Data With JSON

Logging in an unstructured format dramatically increases the complexity of detecting patterns in your logs. By logging your application output in JSON, you open up the possibility of analysis of all of your logs. This gives you a top-down view of your entire system while still maintaining readability. This empowers your CI/CD pipeline by allowing you to query and filter your logs, which then enables you to focus in on the problem and precisely diagnose any unwanted side effects of your latest change.

Create Actionable Alerts

If alerts are defined and channeled correctly, have context and can be interpreted easily, they will be actionable, add context and offer greater value. It is difficult to provide context when a single metric exceeds a threshold, but log lines are more sophisticated. They can include additional information to provide context and allow us to pinpoint the source of the issue fast. This is an essential capability for a company embracing CI/CD.

Prioritize Your Logs

Structured logs often come attached with a severity, which indicates how seriously we should investigate an event. An INFO level log signals business as usual, but an ERROR demands immediate attention. Once your logs have these tags, you can make intelligent, automated decisions that will automatically respond to unwanted changes in your system.

Benchmark Each Version to Understand ‘Normal’

“Normal” is a complicated idea when working in a microservices architecture. It may be safe to ignore a minor slowdown of a single service, but an unfortunate combination of events can lead to disaster. Benchmarking enables us to see these disasters well before our users. Benchmarking is the act of recording what the normal behavior of an application is. After each deployment from your CI/CD pipeline, compare your new behavior with your previous benchmark. If something is out of the ordinary, logs provide the information you need to act decisively. They provide an outstanding baseline signal for your benchmark.

Analyzed Logs Can Level up Your CI/CD Pipeline

Building a CI/CD pipeline is not the most difficult part. Neither is the deployment of new features. In fact, the greatest challenge facing any organization wanting to deliver their changes via a CI/CD pipeline is observability. With a best-practice approach to the preparation, curation and analysis of application and system logs, we can overcome this challenge and confidently change at a pace that propels us to the forefront of our market.

September 22, 2020
JFrog Sets Future Course Post-IPO

Another milestone in the march toward mainstream adoption of best DevOps practices was reached this week, with JFrog becoming a public company.

JFrog CTO Yoav Landman said the funding that the provider of a continuous integration/continuous development (CI/CD) platform is receiving from investors is a clear vote of confidence in DevOps. The next major DevOps challenge will be finding ways to further automate the distribution of software across what is rapidly becoming a much more extended enterprise now that IT organizations are embracing various types of edge computing platforms, he said.

JFrog CTO Yoav Landman

The injection of funding will be used in part to enable JFrog to address that challenge through a combination of internal product development and inorganic acquisitions, said Landman. The last few years have witnessed a wave of consolidation as providers of DevOps platforms expand their reach into adjacent categories.

At the same time, Landman said IT teams should expect to see JFrog investing more in various forms of artificial intelligence (AI) to further accelerate software development and deployment.

JFrog, as a leading proponent of DevOps, has been making a case for focusing specifically on application binaries rather than just source code. The building and deployment of application binaries is at the root of any digital business transformation initiative, noted Landman. As such, any organization that now views itself as some type of software company needs to master DevOps processes to succeed.

That requirement is also accelerating the rate at which CI and CD processes need to converge, he added. Most organizations that have embraced DevOps tend to master CI. CD has proven more challenging because each platform on which software is deployed is unique. JFrog addressed the challenges associated with CD first via its JFrog Artifactory platform before adding on the CI tools required to automate DevOps processes on an end-to-end basis.

In general, Landman noted there is now more focus on software supply chains as more organizations come to appreciate how dependent they are on applications to differentiate themselves from competitors. That issue is one of the main reasons the senior leadership of more companies are becoming more aware of the processes required to build and deploy quality applications consistently across an expanding number of platforms. In fact, it’s conceivable there might one day be more software running on edge computing platforms than in either local data centers or public clouds.

JFrog reportedly has more than 5,800 customers currently. It’s not clear to what degree organizations that have already embraced DevOps might be willing to replace their CI/CD platform or, in some cases, consolidate the multiple CI/CD platforms they already employ. However, as the number of organizations that are embracing best DevOps practices continues to grow there appears to be plenty of opportunity for providers of CI/CD platforms to grow as well.

September 17, 2020
CircleCI Looks to Advance DevOps Best Practices Adoption

CircleCI today launched a developer hub as part of an effort to further advance adoption of DevOps best practices.

Company CEO Jim Rose said CircleCI developer hub provides access to a set of resources for optimizing continuous integration/continuous delivery (CI/CD) pipelines. In many cases prior to the onslaught of the COVID-19 pandemic, organizations had only partially implemented best DevOps practices, noted Rose. The CircleCI developer hub will make it easier for those DevOps teams to extend the scope of those processes while serving to educate IT teams that have had limited exposure to DevOps workflows, he said.

The CircleCI developer hub provides access to, among other things, CI/CD configuration optimizations and pre-built packages, migration guides and documentation. There are also reusable snippets of CircleCI configuration code, customized open source Docker images that can be used on any platform, and command-line interfaces (CLIs) and application programming interfaces (APIs).

Later phases of the developer hub will include an orb development kit to help users quickly create new orbs using CircleCI CLI commands. An orb is a software development kit (SDK) that simplifies integrating the CircleCI CI/CD platform with third-party tools and platforms. CircleCI also plans to provide a much more streamlined search engine experience for CircleCI documentation.

The CircleCI developer hub was built in part using $100 million in additional funding the company raised earlier this year. Overall, CircleCI claims it processes nearly 2 million jobs across 100,000 projects per day.

In general, Rose said DevOps is becoming more challenging as organizations attempt to orchestrate the building and deployment of monolithic and microservices-based applications constructed using container and serverless computing frameworks. There are plenty of instances in which deploying a monolithic application is still more appropriate, so IT organizations need to be able to create DevOps workflows capable of spanning multiple classes of frameworks and platforms, he noted.

Making matters more complex is the testing of microservices now needs to be integrated within production environments, added Rose. It’s not really possible to completely test a microservices until IT organizations see what impact it might have on other microservices that have already been deployed in a production environment, said Rose.

It’s not clear what impact COVID-19 has had on the productivity of DevOps teams. It might be easier for developers to generate more code working from home than in the office. However, generating quality code at the right time during a project requires a lot more orchestration when members of the application development team are highly distributed. More than a few organizations either already have or soon will be tuning DevOps workflows to align with the so-called new normal.

Of course, there are plenty of places to find tips and advice when it comes to building and extending DevOps workflows. The challenge is creating workflows that are not only flexible and resilient but also simple to implement and maintain.

August 25, 2020
CircleCI Adds to Salesforce CI/CD Orbs Collection

CircleCI today announced it has extended its CI/CD support for DevOps teams building applications that extend the software-as-a-service (SaaS) platform from Salesforce.

Company CEO Jim Rose said interest in building applications that complement the Salesforce platform is rising sharply in the wake of the economic downturn brought on by the COVID-19 pandemic. Rather than build entire applications from scratch, IT teams are advancing digital business transformation initiatives by leveraging the core suite of applications Salesforce makes available as a cloud service, he noted.

Many of those applications are built by professional developers that have embraced best DevOps practices, Rose said.

To reduce friction for those DevOps teams, CircleCI is now adding support for a Salesforce Apex orb. An orb is a set of pre-configured connectors that CircleCI makes available for its continuous integration/continuous delivery (CI/CD) platform. The company has created more than 1,600 CircleCI orbs. Salesforce Apex is a proprietary object-oriented language that Salesforce makes available to developers.

The Salesforce Apex orb automatically deploys changes to a production application, runs unit tests that are displayed using the Salesforce Platform user interface and tracks version control on commit.

CircleCI has also updated its existing Salesforce command-line interface (CLI) orb to tighten integration.

Rose said that as the pace of application development continues to accelerate, more IT teams are applying best DevOps practices to how they build and deploy applications as part of an effort to make the whole organization more resilient to change. Others are simply looking to modernize existing DevOps workflows that currently run on legacy CI/CD platforms.

In either case, Rose said most end customers would prefer providers of CI/CD platforms to focus on making DevOps teams more efficient rather than moving into adjacent market segments to increase the size of their total addressable market. Those latter strategies are, in many cases, trying to drive consolidation of the overall DevOps market to woo potential investors rather than focusing on customer interests, he noted.

It’s too early to say how the DevOps platform wars will play out. The size of the overall market continues to expand as the rate at which mergers and acquisitions are occurring accelerates. Regardless of the outcome, more IT organizations are under pressure to deliver more robust software faster, said Rose, noting the Salesforce application developer community represents only one segment of that expansion.

Each organization will, of course, need to chart its own DevOps course. It’s not uncommon for different teams with the same organization to have standardized on multiple DevOps platforms. There may come a day when the need to reduce costs may force those organizations to rationalize DevOps platforms. However, as long as the rate at which high-quality software is delivered continues to improve, just as many organizations may conclude it’s just as well to leave something alone that doesn’t appear to be broken.

August 11, 2020
Harness Acquires Drone CI Project

Harness announced it has acquired Drone.io, creator of the open source Drone continuous integration (CI) project.

Company CEO Jyoti Bansal said Harness, which provides a continuous delivery (CD) platform delivered as a service, will offer Drone to organizations that want a CI platform alongside a CD service from a single vendor. However, Harness will also continue to facilitate the integration of its namesake CD platform with other CI platforms.

Drone will continue to be freely available as open source software and will be re-christened Harness CI Community Edition. Drone Enterprise will be sold as Harness CI Essentials Edition. Later this year, Harness plans to introduce Harness CI Enterprise Edition, which will extend the core platform.

The two companies claim there are currently more than 50,000 active Drone community users for a project that is supported by 250 contributors. Organizations that have adopted Drone include eBay, CapitalOne, Cisco Systems and VMware. Instead of requiring scripts, continuous integration pipelines created in Drone can be declared and managed as code in Git. That approach creates a standard syntax for pipelines that are easier to create, use, maintain and troubleshoot. The two companies claim that reduces the time and cost of continuous integration by a factor of five to 10 over rival platforms.

Analyst firms are projecting the DevOps software market will generate $15 billion in revenue by 2023, signaling an opportunity to provide a more modern CI platform that is easier for organizations to employ, Bansal said. Jenkins, the most widely used CI/CD platform today, traces its lineage back to 2004.

While the overall market continues to grow, however, there’s been a wave of consolidation among vendors attempting to construct end-to-end DevOps platforms. Many enterprise IT organizations tend to prefer to rely on as few vendors as possible to reduce integration costs, which is an issue Harness in part is trying to address by acquiring Drone.io.

Harness is a member of both the Continuous Delivery Foundation (CDF) and the Cloud Native Computing Foundation (CNCF). Drone is already available on a wide variety of platforms, including Kubernetes, and it’s not clear if either arm of the Linux Foundation will play a major role in advancing the development of Drone. The CNCF has thus far accepted Argo, an open source CD platform for Kubernetes, as an incubation-level project.

In the meantime, competition among providers of CI/CD platforms is intensifying. Jenkins, originally developed by CloudBees, is now being advanced under the auspices of the CDF along with Jenkins X, a CI/CD for Kubernetes environments, and Spinnaker, a CD platform originally developed by Netflix. GitLab, Microsoft and its GitHub subsidiary, JFrog and CircleCI all have established major rival CI/CD ecosystems of their own.

Ultimately, it will be up to each IT organization to decide which CI/CD platform makes the most sense for them. It’s even possible many departments within the same organization will make use of multiple CI/CD platforms. Over time, however, most enterprise IT organizations are expected to adopt a CI/CD standard to better facilitate collaboration across multiple DevOps teams.

August 5, 2020
Top Pressing Concerns for CI/CD in 2020
The tech landscape is changing rapidly. With the need for increased development velocity, CI/CD has become ubiquitous among most organizations. Yet, with the growing number of microservices, CI/CD may be tough to wrangle without the right strategy.

At the same time, it’s difficult to keep pace with the constant evolution around CI/CD. So, what is the state of the CI/CD space in 2020, especially in these turbulent times? What do stakeholders see as top pressing concerns companies should consider to advance their release fluidity?

For this article, I interviewed two stakeholders on the latest trends and industry cooperation around CI/CD. Read on to discover what two CI/CD thinkers, Tracy Miranda, board chair, Continuous Delivery Foundation (CDF), and Lachlan Donald, CEO, Buildkite, see are pressing concerns for CI/CD in 2020.

More Microservices, More Problems

With the number of microservices rising to the hundreds (or thousands) for some enterprises, maintaining consistent release strategies across the board could easily become a nightmare. If the goal is to replicate a unique CI/CD pipeline per microservice, pipeline automation and configuration management become top concerns.

“Automation will be key,” said Miranda. “We can no longer afford to be writing each CI/CD pipeline by hand; we will need new approaches.” She added that recent industry discussion shows “configuration management is a top concern for CI/CD practitioners.”

Tools like Jenkin X could enable this sort of automation, she noted. Companies should consider evolving their CI/CD pipelines with better configuration management.

Platforms Must Evolve

The recent “State of the Cloud Report” from Flexera found 39% of IT leaders saw implementing CI/CD in the cloud as a top priority. According to Miranda, the global crisis seems only to accelerate the need for cloud and CI/CD automation.

But, traditional CI/CD platforms aren’t always suited for modern development workflows with numerous pipelines. For Donald, an evolution in new tooling is absolutely necessary to help companies keep pace.

“We’re at a point where organizations need to completely rethink their approach to CI/CD,” Donald said. “Traditional CI/CD platforms are not suited for modern development workflows.” He cites slow speed, unreliability, scalability limitations and security risks as common hangups among many CI/CD tools.

Lack of Clarity in CI/CD Market

The market has expanded significantly to address the gap in CI/CD tooling effectiveness. According to Miranda, this has led to a continuous deployment tooling ecosystem wrought with interoperability and competing styles.

“My biggest concern is the proliferation of tools and platforms with a lack of interoperability and lack of clarity of how to best use them or whether they are fit for purpose,” Miranda said.

These differences make it difficult for developers to find the right integrations that work for their environments, leading to wasted time and effort. “Developers will experience growing pains with their CI/CD platforms as they work to keep up with the constant demand for quality software,” acknowledged Donald.

Lack of Standards

Part of the lack of clarity resides in the fact that there are little to no standards emerging in the CI/CD space. Aside from Accelerate rising as the “bible” for software delivery, Miranda sees a lack of standards in the CI/CD space.

“Not having standards or de facto standards is a problem,” she said. “For example, why is there no de facto standard for reporting pipeline test results?”

Advancing Open Source CI/CD Tools

To improve Interoperability throughout the CI/CD industry, bodies such as CDF are working on setting general guidelines. Miranda described how CDF is accomplishing this, by supporting a CDF Interoperability working group, sharing CI/CD practices and maintaining open source CI/CD projects. Two CDF projects she finds quite compelling are Tekton, a pipeline execution engine, and Jenkins X, a new CI/CD platform for pipeline automation.

Cloud vs. On-Premises: A False Dichotomy?

In analyzing the CI/CD tooling ecosystem, Donald recognizes a recurring dichotomy; that companies must choose between the security of on-premises and the speed of cloud-native. In reality, this dichotomy may be false.

“It is possible to have the best of both worlds,” he said. “A hybrid strategy [is] the solution for testing and deploying software at scale without compromising security or performance.’

As opposed to cloud tooling, self-hosted CI/CD control planes could grant developers more code control and help meet security standards. “I believe self-hosted agents are always a good idea,” Donald said.

As we recently covered, a hybrid cloud approach is picking up steam. Similarly, Jenkins X is designed to run on any Kubernetes cluster, regardless of on-premises or cloud, demonstrating industry movement toward tooling neutrality and hybrid motivations.

Moving Beyond PASS or FAIL

There is a reductive tendency in CI/CD to simply check whether a build passed or not. But, this mindset misses an enormous amount of useful information and undervalues the test and release suites themselves.

“Teams should get granular about what worked and what didn’t,” Donald said. DevOps teams must evolve beyond a “PASS or FAIL” mindset to analyze more minute aspects of the build process itself.

For Donald, this also means shifting to treat test code performance “as a first-class citizen, applying modern development and scaling practices to it and allocating the necessary resources to it.”

Future Evolution: Progressive Delivery and Hybrid Approach

In closing, I asked Miranda and Donald their thoughts on the future CI/CD evolution. They shared their perspectives on where CI/CD needs to evolve.

Miranda views progressive delivery as the next milestone. With progressive delivery, teams deploy services in a staged manner, she said. “Tools, technologies and processes that fall under the ‘progressive delivery’ umbrella will be the next milestones for improving deployment speed and efficiency.” Building blocks such as services meshes could help advance this process, she noted.

Donald stressed the need for a hybrid approach to CI/CD. A neutral control plane could grant better control and future-proof continual release strategies.

Assessing the CI/CD Landscape

In summary, our panel shared thoughts on the top concerns for CI/CD in 2020:
- Platforms must evolve: Teams should rethink their approach and emphasize automation in pipelines.
- Lack of clarity: There are few standards and a lack of clarity among tools. This confuses developer adoption.
- Interoperability: A focus on interoperability is critical toward advancing the industry.
- Unhelpful dichotomies: The cloud-native versus on-premises divide may not be helpful for a hybrid strategy; thus, it’s recommended to seek out platform-neutral tools.
- Move beyond PASS or FAIL: Switch the mindset to “Test code = application code.”
High performant organizations must scale their technology tremendously. With this desire comes a mounting strain on how companies adopt a continuous deployment strategy. Without a proper CI/CD plan, business agility is sacrificed.

As an ending note, I absolutely love Donald’s statement: “A good litmus test for how well a company is run is their approach to CI/CD.” Perhaps what all can agree on is that companies must prioritize CI/CD, as it is now business-critical. Otherwise, deployments may be numbered for those organizations that don’t adapt.
August 4, 2020
How to Use Microservices To Evolve DevOps Pipelines
Enterprises are quickly becoming an intricate mesh of many applications. As companies create more and more microservices, their deployment environments are becoming increasingly elaborate. Without proper configurations, a microservices road map could quickly become unmaintainable.

According to DeployHub CEO Tracy Ragan, if development teams are pushing points to a cluster continuously, application development can suffer without high visibility into their dependencies and application interrelationships.

In her recent talk, “Evolving Your Pipeline for Microservices with DeployHub,” presented by the Continuous Delivery Foundation (CDF), Ragan, also Governing Board member of CDF, discussed ways in which teams can obtain improved insight into microservices delivery pipelines.

Microservices Tracking Issues

A 2018 study conducted by O’Reilly found that 86% of respondents reached mild success in adopting microservices in production. Only 15% reported “massive success.” These results demonstrate both the excitement surrounding microservices and the rarity of flawless execution.

“I’m a huge fan of microservices,” said Ragan. “This is an exciting time to be in software development.” Microservices break from the traditional software delivery model, she said, forcing teams to emphasize the runtime. However, with this shift, it becomes easy to lose the logical views between applications.

According to Tracy, dependency tracking and version control are top issues in maintaining microservices arrangements. With continuous updates, it can be challenging to keep track of which version a service is using. “Microservices development falls apart without a coherent management approach,” she said.

Losing Visibility Moving From CI to CD

Part of the loss of visibility stems from the move from CI to CD. Continuous integration (CI) involves routinely updating source code and starting the compile process. This is often done every hour, as opposed to every week or month. With traditional CI, the system created a package and scanned for dependencies at compile. This was the first step in the continuous journey.

However, nowadays, microservices are typically linked at runtime or in the Kubernetes deploy file as part of a continuous delivery (CD) processes. In doing so, we’ve lost a certain amount of visibility that CI granted. Ragan likened it to “hammering a wine glass”—we still have the constituent parts, but they lie shattered in pieces.

Large microservices suites of 50, 100 or even 300 are not uncommon these days. Such large ecosystems require logical tracking of versions and dependencies. Ragan argued that development teams need a workaround layer to replace the critical configuration management lost in CI compile reports. Visibility into interrelationships is necessary to trace how changes in one service will impact other applications.

Enter Ortelius

Ortelius is a package that helps with continuous microservices management. Ortelius, named after Abraham Ortelius, the inventor of the first atlas, helps trace the connections between services to improve an ecosystem’s visibility.

To get an idea of how it behaves, consider an e-commerce store. To reach an evolvable microservices pipeline, each microservice should be independently deployable with its own workflow. For example, an e-commerce store may require a product microservice, payment microservice and an authentication microservice. These reusable components are independently deployable but rely upon one another to function as a whole.

At a macro scale, an enterprise with thousands of inter-application dependencies would visually resemble a constellation of nodes. An extreme example of this may resemble the Netflix Death Star, a mass conglomeration of intersecting points. Ortelius grants transparency into an otherwise tangled mess as an “internal marketplace for microservices,” said Ragan.

Paring Ortelius With DeployHub

Ortelius is the open source core of DeployHub, an open source SaaS for sharing and releasing microservices. Ortelius, paired with DeployHub, enables teams to see the logical application map, exposing the complexity of microservices dependencies.

Developers add DeployHub to any existing microservice development workflow prior to release. Ortelius is then called to increment the service and record dependency information. DepoyHub then uses for Helm or Ansible for the actual code push, and adds the record metadata and adds the tracking and mapping layer. Having such information is still critical for reporting like dif reports or a bill of materials report, noted Ragan.

This sort of solution, Ragan proposed, is also vital for continuous configuration management—when a microservice changes, the team will always be aware of the new application version.

The New Microservice Pipeline Model

As noted above, in the traditional model, DevOps teams map dependencies in the CI stage. When an API developer publishes to the main catalog, they define specific attributes based on the microservices. A versioning engine will then typically map to visualize dependencies.

Ragan outlined steps for a new microservices pipeline:
1. Build image: Compile and build the container image.
2. Put into registry: Quay or Docker registry. Specific for container registries.
3. Track service-application configurations: This is where we collect configuration data to map the structural relationships.
4. Push to cluster: DeployHub helps to standardize this process across an organization.
5. Collect feedback: Configuration reporting and impact analysis.
Core to this process is being able to independently deploy microservices and aggregate useful information. Of course, this is an incomplete model—a complete pipeline would utilize other steps including testing and automated security scanning as well.

Benefits

By streamlining the microservices journey in this way, repos will be broken out and teams no longer go through one big long workflow. According to Ragan, the visibility garnered from this method could help organizations in multiple other ways:
- Reduce time: Ragan estimated automated application relationship mapping could replace one to two hours of manual work per week per development team.
- Reusability: Decomposing applications into their domains aids reusability across an organization.
- Avoid redundancy: Code sharing can significantly reduce duplicated code, thus avoiding redundancy and increasing flexibility.
- Visibility: Detailed application to service relational knowledge could empower SREs to make better data-driven decisions.
- Centralize: Helps standardize container deployment processes regardless of cluster type or collection, aiding both traditional and containerized microservices deployments.
- Versioning: Improved insight into versioning helps when you need to share services externally.
Lastly, what’s nice is Ortelius is open source on GitHub. Ortelius adopts an open community format to accept feedback. “Democratization is the core reason we have open source,” said Ragan. Ortelius can be used for free, or with added features from DeployHub pricing for enterprise cases.

A Confluence of Software Delivery Styles

Managing continuous software delivery across many microservices leads to unprecedented issues. In solving these issues, the industry appears to be fracturing slightly. For example, the model promoted by Ragan is in contrast with a recent presentation I covered from Codefresh.

Where Ragan suggested improving visibility into independent microservices deployment pipelines, Dan Garfield of CodeFresh suggested adopting a single CI/CD pipeline to operate all microservices from a single control plane. As it goes so often in software tooling, the use case will likely dictate adoption.

Whatever the software delivery process, when decomposing applications—or “taking the hammer to the wine glass,” as Ragan called it—we probably shouldn’t sacrifice visibility into the metadata associated with application versions and service to application interrelationships. If such data is lost or not managed correctly, we could lose a lot of potential value.

For more information on evolving DevOps pipelines, watch the CDF-presented webinar here. DeployHub also published an adjacent article on this subject here. You can also contribute to Ortelius on GitHub.
July 1, 2020
JFrog Extends Reach and Scope of CI/CD Platform

At its online JFrog swampUP Conference today, JFrog announced it has extended the reach of its ability to distribute software to include options based on peer-to-peer and content delivery network (CDN) mechanisms.

In addition, JFrog expanded the reach of the JFrog Pipelines continuous integration/continuous delivery (CI/CD) platform, enabling DevOps teams to build their own library of reusable pipeline steps via an Extensions Development Kit (EDK). DevOps teams can encapsulate frequent and common actions in their pipelines that can run sequentially or in parallel. That approach also serves to reduce the need to rely on plugin modules to extend a JFrog delivery pipeline.

Finally, JFrog has added another free public artifact center that provides a means to manage Helm charts, which are widely employed to deploy applications on Kubernetes clusters. ChartCenter is intended to make it easier to deploy microservices based on containers.

JFrog CEO Shlomi Ben Haim said all three additions to the JFrog portfolio extends the ability for users to manage the entire DevOps workflow from integration to delivery regardless of what type of software is being deployed or where it is installed.

As DevOps teams look to deploy software on edge computing platforms within an internet of things (IoT) environment, Ben Haim said they will need to be able to employ either a peer-to-peer or CDN-based approach to manage that process efficiently. Existing mechanisms are not designed to handle large volumes of artifacts that need to be delivered in a specific order. The new mechanism added to the JFrog CI/CD platform enables DevOps teams to achieve that goal without having to acquire a separate platform, he said.

In fact, Ben Haim noted, efficiently delivering software to edge computing platforms represents the next great leap forward for DevOps. Achieving that goal, however, will only be practical if the entire CI/CD workflow process keeps track of all the dependencies and metadata end-to-end, he said, noting that approach will enable endpoints to pull only the delta code required to update the software after an application is initially deployed.

There is no doubt extending best DevOps practices to edge computing platforms will be a major challenge. Most DevOps teams already struggle with CD because each platform software is being deployed on is unique. Because of a general lack of standardization in terms of potential endpoints that might be deployed at the edge, that “snowflake” issue is likely to become even more exacerbated as IT environments become more distributed than ever.

In general, Ben Haim said far too much of the DevOps focus continues to be on source code management when in reality the biggest issue IT organizations face is how to efficiently manage binaries. At a time when organizations are making massive investments in digital business transformation, Ben Haim noted the rate at which software is deployed and updated is about to accelerate dramatically.

As such, Ben Haim said it’s now only a matter of time before DevOps workflow processes will need to be able to scale well beyond their current limits.

June 24, 2020
CD Foundation Touts Spinnaker CD Progress

At an online Spinnaker Live event today, the Continuous Delivery (CD) Foundation revealed the open source CD platform is gaining significant momentum since being contributed by Netflix late last year.

There were on average 842 merged pull Spinnaker requests a month in the first quarter of 2020. A total of 1,183 contributors participated in the project in the last year, with 40% of those contributors participating in the first quarter. Since the first quarter, the project has added at least two new contributors a week.

Rosalind Benoit, chair of the CD Foundation Outreach Committee and director of community at Armory, a provider of an enterprise edition of Spinnaker, said the bulk of the increased contributions to Spinnaker are coming from Amazon Web Services (AWS), which the cloud service provider is already employing to orchestrate software delivery across multiple platforms.

Benoit said Spinnaker represents an effort to finally address CD issues that have been difficult for most organizations to solve. While continuous integration (CI) is a solved problem, Benoit says CD remains problematic. Each platform code is deployed on within enterprise IT environments is a snowflake; as such, not much progress has been made toward automating the delivery of code across different platforms, said Benoit.

The goal for Spinnaker is to create a foundation for CD that will achieve groundswell support that was achieved for Kubernetes by the Cloud Native Computing Foundation (CNCF), she noted. Both the CD Foundation and CNCF are arms of the Linux Foundation.

Last year, the CD Foundation was launched to advance development and adoption of Spinnaker alongside Jenkins, an open source CI/CD platform; Jenkins X, an opinionated open source CI/CD platform for Kubernetes; and Tekton, an open source project and specification for creating CI/CD components.

While many DevOps teams have attempted to address CD challenges using Jenkins, progress has been uneven at best. Spinnaker was built by Netflix specifically to automate software delivery across multiple clouds using a framework that can be integrated with any CI tool. As such, it serves to provide a clearer line between where CI ends and CD begins.

Historically, most application code is pushed from a CI tool out to a deployment platform. As Spinnaker and other cloud-native platforms such as Kubernetes continue to evolve, the process of pulling images from CI tools to achieve CD goals will become much more automated.

It’s not clear just how much depth of support for Spinnaker there is among providers of DevOps platforms. Members of the CD Foundation in addition to Armory include CloudBees, CircleCI, Google, Huawei, IBM, JFrog, Netflix, GitLab, Harness, Puppet, Rancher Labs, Red Hat, SAP and Sysdig. AWS has yet to formally join the CD Foundation. However, few of these vendors have detailed how they intend to incorporate Spinnaker in their respective platforms.

In the meantime, DevOps teams should be paying close attention to how best CD practices will evolve alongside existing CI processes because it’s clear the next major phase of their continuing DevOps adventure is only now just beginning.

June 18, 2020