devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

  • 5 DevOps Trends Worth Knowing in 2019

    5 DevOps Trends Worth Knowing in 2019

    DevOps today is more than the buzzword it was 10 years ago. Back then, the idea of combining development with production in your IT infrastructure was not common; the logistics were nonexistent and most of the time businesses decided it was far too much effort.

    However, if the last nine years has taught us anything, it’s that DevOps is worth every penny businesses invest. It helps companies streamline production, keep customers happy and provide agile services to attract new audiences.

    But as with anything in the IT field, there is a paradigm of change. This article discusses some of the top DevOps trends happening now and how to implement them in your own system.

    Automation Will Be a Bigger Focus

    Automation has always been the way for businesses. Whether it’s speeding up the pipeline with machines, providing better response times for angry customers with chatbots or better inventory management with tracking and logging system, automation is behind it all.

    DevOps is no exception.

    But as with everything, there can be too much automation. The best thing to do is break down processes and find simple, redundant tasks that can save time if automated and apply the changes.

    The most important aspect of automating DevOps is to understand the six pillars in the DevOps cycle, which are:

    • Continuous business planning
    • Collaborative development
    • Continuous testing
    • Continuous release and deployment
    • Continuous monitoring
    • Collaborative customer feedback and optimization

    DevOps Assembly Lines: The Next Big Thing

     

    To track the development of your application, we have pipelines. These visually represent the product from its initial stage right through until final development and release.

    However, continuous integration (CI) is no longer the priority focus when it comes to effective monitoring and implementation. Continuous delivery and automation are a major part that businesses have already started delving into in 2019.

    This is where DevOps assembly lines enter the picture.

    Assembly lines are focused on bridging the gap between manual and automated tasks, ensuring there is no redundancy while providing several tools for teams to implement, to ensure apps are hitting the production lines as reasonably quickly as possible.

    Microservices Architecture Adoption Taking Off

     

    Microservices architecture may as well be that new kid in school trying to fit in. These smaller services compound into agile, implementable tools that DevOps teams can use to scale and test software over the course of weeks—not years.

    As you can imagine, this can help some businesses streamline their development and production lines, setting them way ahead of their original production estimates—which is never a bad thing.

    Implementing microservices into your DevOps team can also help companies focus on the runtime and efficient delivery of new development sources, without the hassle of creating dependency errors when something decides to bug out for no apparent reason.

    Security Needs to Step Up

    Cybercrime is on the rise—we all know that’s old news. But as a business, cybercrime can either make or break your sales and weigh down your DevOps team.

    That’s why more companies are hopping onto enterprise security options. Although there are third-party programmers for almost every situation you need, none of them have the same security levels as the tools from the enterprise version provides. Safeguarding both your company and your clientele.

    Riding the Artificial Intelligence Wave

    Finally, we have artificial intelligence (AI) and machine learning (ML). When you think about it, these two elements are literally the sum of everything we want in the DevOps community—the ability to crunch out hours upon hours of menial labor and saving your teams hours to focus on higher-priority tasks.

    The combination of AI, used to predict algorithmic influencers, and ML’s ability to adapt to new situations will be key for DevOps teams to get to the next level in 2019.

    — Gil Artmoore

  • Using Jira Software Search to Accelerate DevOps Workflows

    Using Jira Software Search to Accelerate DevOps Workflows

    Imagine driving down the highway and seeming to get a traffic jam at a distance. You realize your car is equipped with GPS to have clear visibility into what’s next and can show exact directions for alternative routes and estimated travel time. It is very easy to pick the fastest and most favorable route to reach your destination by looking ahead proactively and having the set of right tools for collecting information.

    The combination of Dev and Ops teams that are working together to build a faster and more reliable released pipeline is defined as DevOps. Similar to GPS, Jira Software acts as a singular source of truth for information development across your DevOps workflow.

    Complete traceability and visibility of work across different teams can get you surface information and data from an integrated toolchain to the right people anytime. No matter if you’re a team lead, release manager or a tester, the workflow visibility can ensure everyone by letting them know what’s in development and what’s being supported.

    In the latest release of Jira Software version 7.9, there is support added for the dev cycle to show development data from Bamboo, Bitbucket and GitHub in Jira Software search from your DevOps practice. It makes it a lot easier to identify development issues by making informed decisions and faster shipping with software development services.

    Get Answers to Development Questions via Basic Searches

    A new column called Development is added to the default view of Jira’s Issue Navigator in the latest release of Jira Software Server (v7.9). It allows viewing specified information pulled out from Bitbucket Server without clicking into the individual issue. Instead of limiting yourself to the single-issue information, you can see a high-level snapshot of issues coupled together by filtration, to help surface larger trends or problems.

    If this gets stopped at their source, does it really need to move the needle for improving your dev process within DevOps? For instance, if you wish to know how many issues under development, are they in the review?

    If you ever find that one or two developers are getting loaded down with the bulk of pull requests, you can immediately access if redistribution of work is required or not.

    Filter Out Your Development Information Using JQL

    What happens if you need to answer too many complex or open-ended questions? This can become monotonous after a time. You can use Jira Query Language (JQL) to create search queries that are based on development information to uncover key information for avoiding all pending problems.

    The latest version of Jira Software Server comes updated with the syntax to align the JQL query formatting. The users of Jira Software power use JQL for a long time to look at a subset of issues for driving reports and focus on Scrum and Kanban boards. With the help of cross-reference data from developer tools, Jira Query Language seems to be more powerful now.

    Examples of Jira Query Language in Action

    The teams who practice DevOps need critical internal and customer feedback, which helps them to inform features by making them higher quality and more valuable to the end user. Hence, focusing on feedback often creates tension when you decide if the first version is ready to deploy.

    You may ask: What number of issues are done but have not been deployed?

    From this question, you can determine whether the feature is ready to be released and starts collecting feedback or is blocking any issues that need to get resolved prior to release.

    Often, the team’s test builds locally before jumping off to deployment. If the same is the case for your team, then it is a good practice to regularly check the status of the builds.

    You may ask: What number of issues are failing builds?

    If you figure out a lot of failing builds, it may be a signal that the developer environments do not match the deployment. Generally, this is a problem you want to catch and fix right away, so conversely, the set of issues can be the areas where the testers can check acceptance criteria and begin exploratory testing.

    Boost Up Your Search

    You need to surface the information when and where it is needed as most of the teams get the most out of DevOps by bringing development, QA, high-quality releases, more-informed decisions and happy customers. Keep learning!

    — Kibo Hutchinson

  • Using Jenkins Configuration as Code

    Using Jenkins Configuration as Code

    Using the following code, we can configure Jenkins Master to integrate Sonar, Nexus and Jira and manage them as a code. Code as configuration is a plugin in Jenkins, which is very popular and simple to use.

    First, we need to install plugins from Jenkins Market place.

    Manage Jenkins > Manage Plugins > Advance

    After installing, got to Manage Jenkins and you will find Configuration as Code settings.

    After opening this link, it will give us the Jenkins Configuration as Code screen.

    Here, you have two steps:

    1. URL or Path.
    2. Export the Existing configuration.

    Provide URL or Path

    We have two options for storing Jenkins.yaml file:

    1. Any directory Jenkins is running
    2. Version Control (such as Git)

    If we go with option one, then we need to place Jenkins.yaml file in JENKINS_HOME directory, place that directory path and click on “Apply new configuration.”

    If we go for version controlling, we need a place for the Jenkins.yaml file and then we need to open the file in a raw format in GitLab, as shown below:

    Now, place that link and click on Apply new Configuration.

    Export the Existing Configuration

    Export the current configuration into Jenkins.yaml, click the Export Configuration button to download the Jenkins.yaml file.

    After opening this file, you may see some errors. Keep in mind, this is still in under development so the export function is now working properly.

    We can write Jenkins.yaml file in simple YAML format.

    Plugins

    This is what the YAML format looks like:

    Let’s move to next step tool’s installation.

    Tool installations

    This is what it would look like during the tool installation:

    Tools Integration

    The next step is the pipeline job configuration.

    Setting Up Job with JCasC

    Now, let’s bring everything together and see what happens in Jenkins:

    After applying the new configuration, it will show the date, time and path of the configuration.

    If there is an error in syntax format, it will throw the error as shown below:

    Rest API Method to Submit the Configuration

    Here, officially we have JCasC. There is a way to apply changes through CURL.

    Here we have two CURL commands:

    1. Validation
    2. Applying the new

    Validation

    Here, I am using Jenkins YAML to store in version controlling tool with public repo:

    Response will be as shown below:

    Applying new Configuration

    — Santosh Rahul Goru

  • Chef Commits to 100 Percent Open Source Code

    Chef Commits to 100 Percent Open Source Code

    Chef has decided to open source its entire portfolio of IT automation software as part of an effort to make it easier for organizations to construct a DevOps pipeline using the company’s software.

    A part of that effort, Chef also launched the Chef Enterprise Automation Stack—which combines Chef Infra for managing infrastructure, Chef InSpec for maintaining compliance, Chef Habitat for managing applications, Chef Automate for managing hybrid clouds and Chef Workstation, a starter kit for launching Chef—within a single distribution of Chef software. Chef Infra is the original Chef project around which the company was launched.

    Corey Scobie, senior vice president of product and engineering for Chef, said the company previously offered a mix of open source and proprietary software that made engaging with some customers overly complex from a licensing perspective. This approach aligns the entire Chef business model around providing support for open source software, he said.

    At the same time, Chef is now open to collaborating more closely with developers who want to contribute code to its various projects. That’s critical because code contributions are now the primary means through which many IT organizations help shape the direction of products as they continue to evolve, Scobie noted.

    Of course, many IT organizations today either mandate the use of open source software or have instituted an “open source first” policy that makes it challenging for any proprietary software to be adopted.

    Chef expects that by making its software easier to consume via a unified open source transition it will be able to further accelerate adoption of what is described as a “Coded Enterprise” to managing IT. Rather than manually configuring IT assets in isolation, that approach encourages organizations to adopt best DevOps practices that treat all aspects of IT, including infrastructure, as code, said Scobie, adding that capability is more important than ever as IT organizations are increasingly require to manage heterogenous cloud computing environments.

    Chef claims that approach has already resulted in more than half of the Fortune 500 having adopted at least one module of its software and, in the fourth quarter of 2018, the highest level of bookings in the company’s history.

    While Chef helped pioneer the IT automation category, competition across this sector has intensified. Chef and its primary rivals Puppet and Red Hat are the leading proponents of frameworks for automating IT operations. Although adoption of those frameworks has increased substantially in recent years, use within IT organizations often remains fragmented, often resulting in islands of automation. Chef is clearly hoping that IT organizations will increase adoption of its software stack now that it’s  based entirely on open source software. Many organizations face challenges teaching their IT operations teams enough programming skills to automate their processes. In absence of those skills, adoption of IT automation frameworks within the context of a set of well-defined DevOps best practices often remains uneven, at best.

    — Mike Vizard

  • Integrating Maven and Nexus for Continuous Delivery with Jenkins

    Integrating Maven and Nexus for Continuous Delivery with Jenkins

    This article demonstrates how to integrate Maven with Nexus repositories and deploy snapshot and release artifacts automatically during the build process. It also helps to understand version uptick using Maven.

    In this article we’ll cover:

    • Creation of a hosted repository for snapshots and releases.
    • Creation of a proxy repository pointing to Maven Central.
    • Creation of a group repository.
    • Integration of Maven with Nexus repositories.
    • Configurations in the project pom.xml file.
    • Configuration of the Jenkins release build.

    Proxy Repository

    A proxy repository refers to a remote repository. The initial request for a component is forwarded to the remote repository. The component is then retrieved and stored locally in the repository manager, which acts as a cache. Further requests for the same component are fulfilled from the local storage.

    Hosted Repository

    A hosted repository is a repository that stores components in the repository manager as the authoritative location for these components.

    For release, follow the same configuration done for snapshots except Name, Version policy and Deployment policy.

    Group Repository

    The combination of multiple repositories of the same format into a single item is a repository group.

    This feature of the Nexus Repository Manager lets developers rely on a single URL for their configuration needs.

    Create a new Maven (group) repository and configure it like:

    Integration of Maven with Nexus Repositories

    Nexus repositories are integrated with Maven using the Mavens settings.xml file. This file is usually present in ($HOME/.m2, in case of Jenkins /var/lib/Jenkins/.m2) for user-level configuration. Configurations in this file are applied for all the Maven builds initiated by the user.

    There are two ways Maven can utilize the Nexus/other artifactory in general. One is to download the dependencies and use the artifactory as mirror during the build, and the other is to upload the built artifacts into the repositories.

    Each repository in the Nexus and the access credentials are configured in this settings file, so that the repository can be referred in pom.xml and utilized to build the project artifacts and upload them.

    <?xml version=”1.0″ encoding=”UTF-8″?>

    <settings xmlns=”http://maven.apache.org/SETTINGS/1.1.0″  xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”  xsi:schemaLocation=”http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd“>

    <servers>

    <server>

    <id>nexus-snapshots</id>

    <username>admin</username>

    <password>*******</password>

    </server>

    <servers>

    <servers>

    <server>

    <id>nexus-releases</id>

    <username>admin</username>

    <password>*******</password>

    </server>

    <servers>

    <mirrors>

    <mirror>

    <id>central</id>

    <name>central</name>

    <url>http://hostname:8081/repository/maven-group/</url>

    <mirrorOf>*</mirrorOf>

    </mirror>

    </mirrors>

    </settings>

    Configurations in the Project pom.xml File

    The following configuration is for downloading the dependencies from maven group repository:

    <project …>

    <repositories>

    <repository>

    <id>maven-group</id>

    <url>http://hostname:8081/repository/maven-group/</url>

    </repository>

    </repositories>

    </project>

    And this is the configuration for publishing snapshot and release artifacts to Nexus:

    <project …>

    <distributionManagement>

    <snapshot>

    <id>nexus-snapshots</id>

    <url>http://hostname:8081/repository/maven-snapshots/</url>

    </snapshot>

    </distributionManagement>

    </project>

    <project …>

    <distributionManagement>

    <releases>

    <id>nexus-releases</id>

    <url>http://hostname:8081/repository/maven-releases/</url>

    </releases>

    </distributionManagement>

    </project>

    Add the tag to the SCM for the Maven build:

    <scm>

    <connection>scm:git:http://gitlab.com/user/project.git</connection>

    <url>http://gitlab.com/user/project</url>

    <developerConnection>scm:git:http://gitlab.com/user/project.git</developerConnection>

    </scm>

    Run – clean deploy in Maven goals for Jenkins snapshot build.

     Configuration of Jenkins Release Build

    In the Jenkins build configuration, enable the a Maven release build with required release goal. Also, make sure to select Preselect ‘Specify SCM login/password,’ which is required when performing Maven release with authentication.

    After the snapshot build completion, the below inputs are needed to perform the Maven release. Here, username and password refer to the GitLab admin credentials.

    Conclusion

    By following the above steps, Maven and the Nexus artifactory can be integrated at the user level and the Maven execution for that user inherits this setting. A similar setting can be done at the system level with the Maven global tool setting.

    These steps help project teams with one-time configuration of snapshot and release artifacts upload to Nexus, and a version uptick of pom.xml in GitLab. Similar settings can be performed for other artifactories such as JFrog.

    — Mohamed Azeem

  • How DevOps Orchestration and Feedback Loops Can Enhance Your DevOps Pipeline

    How DevOps Orchestration and Feedback Loops Can Enhance Your DevOps Pipeline

    DevOps orchestration and feedback loops can significantly impact and improve your organization’s delivery pipeline. In this post, we’ll define and illuminate the importance of DevOps orchestration and feedback loops, and how together they can make a difference in your pipeline.

    DevOps Orchestration

    Although people widely use the terms “automation” and “orchestration” interchangeably, their scopes vary with significant process differences between them. DevOps automation automates a single function or a task in a DevOps pipeline that is routine and monotonous. Orchestration describes a string of such automated tasks consolidated into a streamlined workflow. In environments where automation would not work because of barriers among the groups, orchestration can be used to help organizations solve issues despite such barriers. Hence, DevOps orchestration simply brings together already existing automation networks into a cohesive, consolidated platform.

    The Need for DevOps Orchestration

    DevOps teams deal with constant navigation between development and operations. It would be highly beneficial if a single platform could pull together the tools teams typically use as various automation pieces from multiple DevOps toolsets. This is where DevOps orchestration services come into the picture, as orchestration enables teams to achieve that.

    The primary goal of DevOps orchestration is to ensure seamless and quick delivery of new builds into production. It also aims to achieve streamlined and error-free software releases. Different organizations use various coding standards, guidelines and policies. DevOps orchestration plays a major role in efficiently managing and ensuring adherence to such policies. DevOps orchestration also is pivotal in properly defining and maintaining third-party software and their multiple versions across different environments.

    DevOps Monitoring

    Effective DevOps monitoring is vital. It is the key to understanding and evaluating various performance metrics in the production environment. When done right, DevOps monitoring can provide visibility into different critical aspects such as deviations in performance, application functionality issues and service availability problems. In any DevOps environment, the ability to identify and pinpoint such issues proactively can be highly beneficial, and DevOps monitoring enables executives to do so.

    Feedback Loop

    Feedback loops are the essence of any process improvement and DevOps is no exception. Typically, such loops initially function with an organization’s policies and progress all the way through implementation, policy adjustments (if any) and usually ending with reimplementation. They are not limited to specific process cycles in a DevOps environment—for example, it’s applicable to CI/CD practices, infrastructure monitoring and application development

    Types of Feedback Loops

    Broadly, there are two categories in feedback loops: reinforcing feedback loop and balancing feedback loop. People refer to the former as “a loop of accelerating change.” In a reinforcing feedback loop, increased value in the first entity results in an increase in the second entity, which in turn further increases the value of the first entity. Conversely, in a balancing feedback loop, value increase in one entity results in a decreased value in the second entity, which consequently decreases the value of the first entity.

    Significance of Feedback Loops

    One of the major capabilities that differentiates and carves a unique niche for DevOps from the rest of its predecessor methodologies is its feasibility to incorporate feedback from customers as well as other systems in the pipeline, including the monitoring system. The feedback obtained helps to bridge the gap between the software’s function and customer expectations. It also provides valuable insights about what you can do to improve the build of the software and ultimately enhance its functionality and reliability.

    Feedback loops are particularly crucial when it comes to continuous testing. To carry out continuous testing effectively, building automated tests is only a job half done; the rest depends on the visibility of the test results and how you can utilize them to enhance the existing process. To do that, you need comprehensive feedback of your software’s performance across various stages of the software development life cycle (SDLC), starting right from development and continuing all the way to post-production. Implementing the right feedback loops will help you gather such detailed feedback.

    Thus, feedback loops serve as the difference between merely speculating how an end-user might consume a product and methodically improving the existing workflow to better suit the end-user requirements through feedback.

    — Kavin Elango

  • DevOps Chat: The CD Foundation, with CloudBees’ Kohsuke Kawaguchi and Tracy Miranda

    DevOps Chat: The CD Foundation, with CloudBees’ Kohsuke Kawaguchi and Tracy Miranda

    A new foundation, the CD Foundation, has been formed under the auspices of the Linux Foundation to serve as the home base for a vendor-neutral movement committed to making it easier to build and reuse DevOps pipelines across multiple continuous integration/continuous delivery (CI/CD) platforms.

    The first projects to be hosted under the auspices of CD Foundation includes Jenkins, the open source CI/CD system, and Jenkins X, an open source CI/CD solution on Kubernetes. Both were developed by CloudBees. Netflix and Google, meanwhile, are contributing Spinnaker, an open source multi-cloud CD solution. Google is also adding Tekton, an open source project and specification for creating CI/CD components.

    Founding members of the CD Foundation include Alauda, Alibaba, Anchore, Armory, Autodesk, Capital One, CircleCI, CloudBees, DeployHub, GitLab, Google, Huawei, JFrog, Netflix, Puppet, Red Hat, SAP and Snyk.

    In this DevOps Chat we speak with Kohsuke Kawaguchi (aka KK), founder of Jenkins and CTO of Cloudbees, and Tracy Miranda, open source community director for CloudBees. They give us the insight behind the CD Foundation and what the future may be for Jenkins, JenkinsX and CloudBees.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation. Enjoy!

    Transcript

    Alan Shimel: Hello everyone. It’s Alan Shimel, devopsy.kinsta.cloud. You’re listening to another DevOps Chat. Today’s DevOps Chat is about some big news that came out in the DevOps continuous integration/continuous deployment world, CI/CD. That is the founding of the CD Foundation and it’s part of the Linux Foundation. It was announced just this afternoon. I’m happy to be joined by a couple of folks from one of the founding members—really one of the pivotal founding members of the CD Foundation, CloudBees. From CloudBees, joining us, joining us today is Tracy Miranda, who is director of open source community at CloudBees, and none other than KK himself, founder of the Jenkins Project and community as well as founder and CTO of Cloudbees. Kohsuke, Tracy, welcome to DevOps Chat.

    Kohsuke Kawaguchi: Thanks for having us again.

    Shimel: Kohsuke, it’s always a pleasure to have you. Tracy, I know you’re new to this, but thank you for joining with us.

    Tracy Miranda: Yeah, [inaudible].

    Shimel: Guys, big news today. Linux Foundation starts the CD Foundation. CloudBees is one of the founding companies, and if I’m not mistaken, a board member of CD Foundation as well. Is that correct?

    Kawaguchi: Yeah, exactly. We are part of what – I think 20.

    Miranda: Yeah, I think we’re up to 23 members now.

    Shimel: I think it might have been 25.

    Miranda: Wow [chuckle] I think it’s continuously growing.

    Shimel: It’s funny. I spoke to a few companies today after the announcement. They mentioned that they’d like to join in it. That’s for them. I’d like to understand and maybe, KK, this is a question for you. What was the motivation to turn Jenkins and Jenkins X – we’re talking about two significant projects – over to this new foundation and to the Linux Foundation?

    Kawaguchi: From the Jenkins – I have multiple hats to wear in this, but from the Jenkins project perspective, a key part of it is we wanted to create this level playing field that creates – that accelerates the innovation in Jenkins family. The Jenkins family as you know, it’s been benefiting pretty much everyone in every organization, and there are a lot of companies with an eye on Jenkins. We want to drive them more into the project. We move the project forward together. In that way, I think we can do it faster. In case of Jenkins X, it’s been critically designed on a number of projects that’s coming out of the CNCF, which is a sister foundation that is also under the Linux Foundation. It made a lot of sense we’d be bringing these projects closer so that we can again collaborate faster. So those are, I think, very convincing obvious benefits for two projects.

    Now the CTO of CloudBees, CloudBees has been the biggest contributor of behind the Jenkins project. This move made a lot of sense for us as well, because one, our customers rely on a lot of open source projects and ecosystem. When those foundational resource projects that’s driving is strong, then obviously that benefits CloudBees and our customers. I think that’s a fairly obvious move for us as well. Also importantly, we – well, I, talk to a lot of people in the – our customer and prospect organizations and those technical people understand the need and the benefits that the objectivity really brings. I often see them struggling while the rest of the organization around them. Here, I talk about things like money, people, and time to invest in these efforts. It make the necessary business impact.

    I feel like a part of the role that I’m really looking forward for the CDF to play is to advocate the practice of continuously, elevate it beyond the technology practice. Again, like the CNCF for cloud-native, thereby we can make a bigger impact in the world. That’s the kind of thing a single vendor cannot very easily do because people automatically assume the intention behind these things, or a pitch behind them. Open Source foundation, there’s multiple vendors, and also the end user companies like _____ and _____ adds a lot of credibility and authenticity to that message. For all of these reasons, I think it’s kind of a no brainer, a win win win across the board. That’s how I think about it.

    Shimel: I think that’s a great way of looking at it. Tracy, let me ask you, you’re the liaison to the open source community. In this case, we’re really talking about the Jenkins community and Jenkins X. How is this announcement – it’s kind of new still, but what’s been the feedback from the community on this?

    Miranda: Yes, it’s been really, really positive. We’ve been talking to the Jenkins community, well, the Jenkins board members, Tyler and Kohsuke, the kickoff conversations over a year ago. Since I’ve come in as well, we’ve been keen to get everybody involved with how this would shape up. We’re hearing lots of exciting things. Some things we hadn’t even predicted at the beginning. So for example, Jenkins has a very emerging and very strong and enthusiastic Chinese community. They are super excited about using CDF as a way to really expand in China. I think we can look to that for new areas we want to expand into. So yeah, on the whole, very positive. Lots of enthusiasm which is fantastic.

    Shimel: Absolutely. As a matter of fact, I’m somewhat familiar with the Jenkins community in China. One of our partners held the Beijing Jenkins Day while I was out there last year. So I had a chance to meet a lot of those folks firsthand. It was a very enthusiastic group. They’re doing great things there in DevOps and CD as well. I’m sure it’s been used as well as Jenkins X. Actually, one of the founders of the Jenkins X project was presenting there that day. He was in from Australia. It was very, very well received. Back to this, both for Tracy and KK, one of the things I think people really admired about CloudBees was it wasn’t the usual vendor open source project manager relationship.

    So many of these open source projects that are managed by a single vendor tend to be run for the sole benefit of that vendor whether it be OpenCore or Razor-Razorblades kind of thing. Decisions are made for the benefit of the commercial vendor, oftentimes at the expense of the community. CloudBees was always above that. All these years there was an independent board. There was independent steering and governance of the Open Source community and project. With that already in place, KK, does that negate some of the reasons why you would need a separate foundation, or that just accelerates it?

    Kawaguchi: Yeah, so thanks for the nice words. We’re already proud of the way we participate in Jenkins project. I think there are two factors to that. One is number of people who founded CloudBees. They all come from open source background, the _____ folks, RedHat folks. We share a lot of – it’s like a common body. And that really played a critical part in recognizing, understanding how open source community works. The other important part to call out is Jenkins project is not owned by CloudBees. It is an independent project with its own decision-making process. This project, mind you, it came out of the drama with – well, a certain company. I won’t name names.

    Miranda: [laughter]

    Kawaguchi: So I think the people in the community felt really strongly about their independence. So we discussed it. So, in moving to CDF, many – well people are outside CloudBees, they also continue their _____ CloudBees. They’re also very anonymous desire and preference to come over to CDF because they see it as the next step of the governance to ensure this level playing field, which I touched on a little earlier. Then I think that’s been – that was seen as very valuable characteristic. I think the CloudBees has done good things over the past year _____ the project. I think it’s time to also codify and turn it into something more stable and fundamental. I think that’s what this move to CDF signifies.

    Shimel: I think that’s true. I think there’s another element too, KK. When we look at the history of Jenkins, when something is so successful, you don’t see many contenders. That was true with Jenkins. When Jenkins burst on the scene – we don’t have to go into the history of it, but Jenkins burst on the scene, and it quickly became the dominant CI and then CD tool of choice, not only for the DevOps community, but for the software developing and ops communities. Everyone used Jenkins. Recently, in the last year or two, three years, we’ve seen other contenders coming up, many of them commercial.

    Some of them have an open source, open core, kind of program business plan, but nevertheless, people had this, “Well, Jenkins is older, or it’s yesterday. This is tomorrow.” I think by putting it in a foundation like this, it guarantees a tomorrow. It’s the old, “So many eyes on the code, it has to be better” story. With so many hands in the pot, it has – it’ll accelerate faster. So that’s a good thing. I feel like we’re neglectful though. We should mention beyond Jenkins and Jenkins X, there was some other great software programs that were contributed to the CD foundation today, correct?

    Miranda: Yes, that’s right. Would you like to hear a little bit more about those?

    Shimel: Yeah, please.

    Miranda: We’re super excited. One of the other projects is Spinnaker, is what Netflix originally put together for continuous delivery. That is a well-known, a well-loved project in this space. We were really excited they bought into the vision and wanted to be part of it. Then the other project, last year Google announced the Knative serverless platform. As part of that project, there’s a build and pipeline side of it which has been renamed to Tekton. Tekton is the pipeline and build part of – behind Knative. What’s really exciting about this project is that we’re looking as part of the vision of CDF to really focus on integration and sharing concepts and making tools work together. Tekton will be the pipeline engine, set the definitions for which Jenkins X are going to evolve to. Spinnaker are going to look at it too, and for Jenkins’ next generation pipeline, there’s lots of exciting things in the work. Just really excited about all the possibilities coming up.

    Shimel: Sure. We should mention both Spinnaker and Tekton were contributed by Google.

    Miranda: Google and Netflix for –

    Shimel: And Netflix yes, for Spinnaker. But that’s not all. Once a technical oversight committee is formerly introduced, there may be other projects that are – come under the CDF management as well, correct?

    Miranda: Yes. Absolutely. I think one of the things we outlined in the charter is we want to focus on the entire software delivery life cycle. We want to have a whole set of complimentary projects. So we’re  looking – there’s some really exciting projects in the world of mobile and other areas as well. It’s not just cloud. So lots and lots of exciting projects out there. We look forward to welcoming and onboarding those as we develop, and as we get up and running properly.

    Shimel: We should also – I just wanted to mention that some of the other companies involved as founding members of this CDF are Alibaba, Autodesk, Capital One, CircleCI, CloudBees, obviously, DeployHub, GET Lab, Google, Huawei, JFrog, Netflix, Puppet, Red Hat, SAP. These are household names out there. This really is a heavyweight list of contributors, and I think guys, that that really talks to just how important CI/CD has become in the modern software factory.

    Kawaguchi: Like I said, it’s kind of on the cusp from growing from just the technology things to more bigger business impact. It took CloudBees all along, and so I’m really excited that this mission, this sense of passion is all shared by so many more companies. I think it makes the message a whole lot more critical.

    Shimel: Here’s another flip side. As this frees up CloudBees to really focus in on the commercial side of things. Though you weren’t – CloudBees didn’t manage Jenkins. The community and the Jenkins community managed itself. CloudBees played a – obviously it was a big role helping – making sure – KK, you’re wearing two hats with that. Now that CloudBees is freed from that, what do you think we’ll start seeing from CloudBees going forward?

    Kawaguchi: Actually, we are doing more than ever in open source. A number of key folks behind Nginx is at CloudBees _____. So just to make sure that this isn’t all a move by _____, trying to wash our hands away from it. In fact, it’s the opposite. At the last time also, CloudBees is getting bigger. We are working on a number of the products and services. So I don’t really see those things as mutually exclusive. I think everyone at CloudBees understands that those are both important parts of our business.

    Miranda: Yeah, and if I add to that, well, this one specific example, the DevOptics project, which focuses a lot on analytics, and these key measures you’re doing your software delivery. That’s one area where CloudBees is going to do a lot more and you’re going to see a lot more, and it’s pretty exciting going up to that next level.

    Shimel: Excellent. I would imagine you would. Tracy, if you don’t mind, this is going to be a change for you and how you deal with the open source community, or you think not really, not too much?

    Miranda: No, absolutely. I think it’s going to be quite different. We’ve got lots of different people working together. I think that’s what makes it really exciting, just having the different perspectives, the different ideas. I always say like with open source, what I’m most excited about is the unexpected things that you don’t really know what they are until they happen, the innovation, or the two people getting together and saying, “Hey, let’s take things in this direction.” It’s so full of possibilities that it’s really exciting.

    Shimel: I think it will be, and I think it’s going to be for sure. You know what, I haven’t read – who particularly from CloudBees is going to hold the board seat in the CD foundation? Has that been announced or decided?

    Miranda: So I’ll be representing CloudBees at Board of Directors, but also Kohsuke is heading up the technical oversight committee, which means he will also be at the board level. We’ve modelled a lot of the charter and foundation rules off the CNCF where it made sense.

    Shimel: Fantastic. That is really good. What about some of the other folks who are on the board of the Jenkins community board? People like Tyler and some of the other folks? Will they be – what will be their involvement going forward, if you know?

    Kawaguchi:  I think the charter will define the structure. It’s just got launched and we will, I think, in the coming days we will have people’s names and so on. I just don’t want to give the wrong information, but Tyler, for example, he’s been pretty instrumental in making this happen. So I expect him to stay involved in this going forward in some shape or form.

    Shimel: Absolutely. I realize it’s only announced today. You don’t have all the answers, but I thought I’d ask. So final question because we’re running low on time here is one of the – you called the CNCF a sister foundation, it would be hard to argue that hasn’t been a really successful effort by the Linus Foundation, and not just Kubernetes. Kubernetes seems to be the star pupil, or star favored child. There’s a lot of projects within CNCF that are really, really doing well. CNCFCon along with KubeCon is – to me, it was the most exciting event I’d been to certainly within the last six months, certainly since Jenkins World. Will there be a similar type of event now for CD Foundation? Or maybe you pitch one grand tent with CNCF for one big conference? Any ideas on that?

    Miranda: Well, first thing we have in place is that we are going to hold our very first continuous delivery summit. We’re going to co-locate that with the new KubeCon which is in Barcelona.

    Shimel: In May.

    Miranda: That’s right. Yeah, getting going quick. That will be our first opportunity to bring a lot of contributors together, a lot of the end user community as well. We just love the projects in CNCF. We want to continue collaborating closely with those folks. I think this will be a great first event. We did just hear today that there’s something like 12,000 people expected at that event in Barcelona. Big, big fans of CDCF.

    Shimel: Yes. I will be there – we’ll be there doing videos. Hopefully, we’ll get to, Tracy, meet in person, maybe do some video interviews there in person.

    Miranda: Yeah, that would be awesome.

    Shimel: Yeah, that would be great. Then, we’ll – I forgot when the next one. Well, after that, the Shanghai in July, and then in the fall is KubeCon back in North America. Will this be an ongoing collaboration or is that – after that, you might do your own?

    Kawaguchi: We don’t know that yet, I think.

    Shimel: Still to be decided. All right, well guys, congratulations first of all, on all of the new today. This sounds just so god darned exciting. It’s good stuff. Kohsuke, what can I say, man? You just keep doing it. Congratulations. Keep it up. Tracy, congratulations. I guess we’ll be hearing a lot more about the CD Foundation in the months and year ahead.

    Miranda: Yes, absolutely. Thanks so much, Alan. This has been wonderful.

    Shimel: You know what? I sound like Colombo. One last question. You’re probably too young to remember that, but anyway, why not CI/CD Foundation? Is CI becoming passé, out of style, we don’t refer to it? Or we’re just so focused on CD? Any thoughts on that?

    Kawaguchi: The practice that surround the continuous integration is still very much there. I just think of it as frontier moving on, and so therefore the concept is encompassing more and more. When we say continuous _____, so the _____ includes all of the things we talked about as CI. It just expanded into something bigger is how I think about it.

    Shimel: Yeah, all right, fair enough. Guys, again, thanks so much, Tracy, KK. Kohsuke from CloudBees on the announcement of the Linux Foundation setting up the CD Foundation and Jenkins, Jenkins X part of it. This is Alan Shimel for devopsy.kinsta.cloud, and you’ve just listened to a minute of DevOps Chat.

    — Alan Shimel

  • How CI/CD Can Fast-Track Next-Gen Media Rollouts

    How CI/CD Can Fast-Track Next-Gen Media Rollouts

    Using CI/CD can help digital service providers move as quickly as over-the-top content providers

    Online video consumption has been increasing tremendously with rapid change in consumer expectations to have seamless viewing experience across various digital devices. To capture this growing demand, it is critical for digital service providers (DSPs) to deliver fast-track rollout of next-gen media services. Moreover, to counter the growing influence of over-the-top content providers (OTTs), DSPs must deliver media services that are high quality, right the first time and consistent, along with creating competitive advantage through speed to market. However, unlike OTT players, DSPs are challenged by extremely long development and rollout timelines for new services and offerings, primarily because of vendor-specific hardware and software applications that do not support rapid changes.

    Figure 1: Rapidly changing landscape of consumer video consumption.

    Embracing an effective continuous integration and deployment (CI/CD) setup can enable DSPs to achieve the same service innovations and delivery agility that OTT providers are offering. An important characteristic of an effective CI/CD setup is to create a fully automated deployment that is easy to operate and visualize. The setup also should have remote accessibility to ease parallel deployment at multiple geographic locations and a robust monitoring toolchain to monitor each step in the deployment process.

    However, implementing a CI/CD setup with these characteristics for next-gen media landscape comes with its own challenges; it requires end-to-end CI/CD orchestration, which is operationally complex. This involves stitching together different types of video services such as live TV streaming, VoD, electronic program guide (EPG), 4K/HDR, user personalization and profiling. These services need to be rolled out in multi-geographic locations and multi-tenant environments, catering to multiple vendors. The service must offer a seamless viewing experience across all digital platforms including TV, mobile, tablet and desktop.

    In setting up such complex processes, cross-functional and cross-geographic teams own different parts of the process, which can create gaps in design, infrastructure setup, configurations and eventually deployment. The CI/CD pipeline created can be costly and ineffective.

    Setting Up Effective CI/CD

    One-click automated delivery pipeline tool for predictable rollouts/rollbacks

    Create a one-click automated delivery pipeline tool that can facilitate automated change requests (CRQ) to initiate CI/CD jobs. The tool triggers the entire deployment of all the app components in the respective tenants across multiple geographies.

    Key considerations in setting up a one-click automated delivery pipeline:

    • Track CRQ approvals over systems such as JIRA to trigger the automated delivery pipeline.
    • Build pipeline as a code with intermediate CI/CD stages using open source tools such as Jenkins.
    • Design a master job to accept component and configuration details. This performs multi-job configuration and triggers contextual deployment/rollback simultaneously across tenants.
    • Interconnect different aspects of networks, infrastructure, components and applications to run the entire process smoothly.

    Setting up an automated delivery pipeline provides greater accuracy and predictability by reducing the human error factor.

    Automated snapshots, pre- and post-validation scripts to ensure consistency

    Create a fully automated snapshot management process that includes automated snapshot creation, rollback and deletion across respective tenants. Automated snapshots before deploying the new version of application facilitates rollback to the previous consistent version with zero downtime, if any failure occurs.

    Ensure automated multiple validation checks at every deployment stage to ensure consistency. These checks include validation of network connectivity, memory usage, CPU utilization, running applications, error and debug log files, license files, binary file size and configuration files. These validations can be done via tools such as groovy, ansible, Jenkins, python or shell scripts with proper libraries that match industry standards.

    Auto-scaling tool to optimize resource usage

    Ensure the application always has the right amount of capacity to handle traffic demands. This requires integrating an auto-scaling tool with predictive machine learning (ML) models and change request process to dynamically scale up/down the infrastructure.

    Key considerations in setting up auto-scaling tool:

    • Identify external and internal influencing factors to analyze system throughput.
    • Leverage insights derived from a monitoring dashboard to manage changes in current payloads.
    • Leverage triggers from a ML model to plan future expected system throughput.
    • Create automated configurable scripts to support dynamic scaling based on the inputs received from change request and ML model.

    Creating such tool enables better infrastructure availability with efficient cost management.

    Tool-agnostic and on-demand environment creation framework built on open source

    Leveraging open source tools provide flexibility and control over the entire exchange-to-exchange deployment process. Using customized framework with a layered approach makes the delivery pipeline tool-agnostic. It gives flexibility to easily replace/plug in any tools in the DevOps chain without much dependency.

    This ensures increased reusability, as the same toolchain can be used for multiple environment setups. Further, a customized framework dynamically handles the resources and utilizes the infrastructure effectively.

    Service assurance dashboard for stability and monitoring end-to-end processes

    A service assurance dashboard can provide glimpse of end-to-end CI/CD performance in a single window. It provides real-time graphical insight into failures/blockages in the pipeline, highlighted in the context of the overall flow. The dashboard should dynamically fetch data from the deployment pipeline for respective components and monitor how the upgrade/rollouts are processing. It should provide insight into resource usage, which can trigger dynamic auto-scaling and give a visual comparison of previous firmware with the new one.

    Implementing such a service assurance dashboard can provide greater visibility and help in easy diagnosis and error prevention.

    Benefits

    • Accelerated time to market with nearly 60 percent reduction in waiting time with parallel deployment across multiple tenants.
    • Improved predictability with significantly reduced fallouts and downtime/failure rates.
    • Optimized cost with approximately 50 percent lower IT spends in the deployment process and resource usage.

    This article was co-authored by Sumit Thakur, manager, strategic insights at Prodapt; Vijay Ramachandran, senior project manager at Prodapt.

    — Lakshmiprasad A

  • 4 Steps to Improving Your Team’s Continuous Security

    4 Steps to Improving Your Team’s Continuous Security

    A recent npm survey revealed that 77 percent of developers were concerned about the quality and security of the open source libraries they use, while 52 percent said the tools currently available were inadequate. Given the trust issues in open source and the ongoing discovery of website and application vulnerabilities, security should be top of mind for developers at the start of every project—but also on an ongoing basis. I call this practice of constant re-assessment “continuous security.” Here’s how I like to describe it:

    Imagine you are standing at the bottom of an escalator that’s moving downward. Your goal is to reach the top. You might think, “Easy, I’ll just sprint my way to the top! Run as fast as possible and I’ll get there, and the problem is solved.”

    It’s not that easy, though. As you start your climb, you realize that you haven’t really trained for this and you’re slower than expected. The people traveling down toward you become obstacles to be evaded. Through all this, you have to maintain your upward momentum, or you’ll end up back where you started.

    Now, it might sound a little cliché, but that’s how I view security. Security is something you have to continually train for—a constant cycle of risk discovery, prioritization and skill-building to close the gap to get results.

    Continuous security is all about becoming more efficient at this entire life cycle and reducing the time to risk discovery and the overall cost of mitigation.

    Risk Discovery

    The first step to improving your security situation is to find something that needs improvement. These tend to be risks to the business and look different depending on your focus area. For a legal team, that might be license compliance; for a development team, it’s likely to be a security bug in a production service.

    When you start out, your abilities to discover risk are going to be pretty poor, but that’s okay. I like to apply this discovery process at the natural gates of the development life cycle. Major ones include: when the feature is designed, when code is first written and when someone makes a pull request, pushes code to staging and pushes it to production. The tooling and techniques you use in each of these stages will vary greatly by the needs of the business and change as your security program matures. There will be arguments continually about what is considered a risk and how it should be prioritized. This is part of the process of refining your risk discovery skills and tools.

    Policy and Standards

    There is rarely any change without pressure, and because we don’t always have an active incident to force improvement, that pressure sometimes has to come from within. Motivating your team through policies and standards can be the ideal way to influence improvement.

    Everyone in your organization will have their own perspective, so developing a strong set of policies and standards to measure yourself against will go a long way. Writing and adopting these standards should be collaborative, both with your security team and the organization as a whole.

    These policies help you determine what is a risk and how you prioritize it as an organization. At npm we have a standard rating scale that security assigns to risk (P0 = drop everything and fix critical, to P3 = fix within 180 days). This tells engineering how long each risk should take to mitigate, helping them prioritize their work accordingly.

    Mitigation and Improvement

    This is where the technical practitioner will have the most fun—fixing lapses in security is usually satisfying work. The important part is to use each incident as an opportunity to scale your efforts in risk identification.

    For development teams, that means pushing more automation in your CI/CD pipeline to arm developers to be able to find security issues earlier in the development process. I like to call this the CI/CD/CS pipeline: continuous integration, delivery and security. However, automation is far from perfect, and the best way to improve decision-making is to provide a human with the right amount of information in the right context whenever a decision needs to be made.

    An important thing to remember is that your solution does not have to be perfect. There are many situations in security where “better” is good enough to mitigate risk to an acceptable level. In the end, what you need to have is a constant feedback loop of risk discovery and mitigation, using technology and processes to manage and improve.

    Continuously Improving Continuous Security

    Setting goals and expectations is a part of continuous security, but you should keep in mind that “security” is not a set destination. The word “security” lies to us by suggesting that it’s possible to secure something, when in truth the mitigation of all risk is an impossibility.

    Instead, always try to maintain your upward momentum. Measuring all successes and failures honestly allows you to be effective at all times, whether in the middle of a security issue or a period of relative calm. A continuous, proactive approach to security is the only way to effectively ensure the safety of developers and code.

    — Adam Baldwin

  • The Linux Foundation Launches Continuous Delivery Foundation

    The Linux Foundation Launches Continuous Delivery Foundation

    The Linux Foundation announced it will provide the home base for a vendor-neutral Continuous Delivery Foundation (CDF) committed to making it easier to build and reuse DevOps pipelines across multiple continuous integration/continuous delivery (CI/CD) platforms.

    The first projects to be hosted under the auspices of CDF, which was launched at the Open Source Leadership Summit conference, includes Jenkins, the open source CI/CD system, and Jenkins X, an open source CI/CD solution on Kubernetes. Both were developed by CloudBees. Netflix and Google, meanwhile, are contributing Spinnaker, an open source multi-cloud CD solution, and Google is also adding Tekton, an open source project and specification for creating CI/CD components.

    Additional projects are expected to join CDF once a Technical Oversight Committee (TOC) is formally named.

    Founding members of the CDF include Alauda, Alibaba, Anchore, Armory, Autodesk, Capital One, CircleCI, CloudBees, DeployHub, GitLab, Google, Huawei, JFrog, Netflix, Puppet, Red Hat, SAP and Snyk.

    Chris Aniszczyk, vice president of developer relations for The Linux Foundation, said that while a lot of progress has been made over the years regarding adoption of CI, the challenges associated with mastering CD have proven more vexing. To address that issue, the CDF will evangelize CI/CD and DevOps methodologies, define and document best practices, provide guidelines and create training materials to better enable organizations to implement CI/CD best practices.

    Kim Lewandowski, a product manager for Google Cloud, said making it possible for organizations to reuse DevOps pipelines is a high priority. Building pipelines is a significant challenge for most DevOps teams; as such, they need to be able to reuse those pipelines across as many CI/CD platforms as possible.

    Kohsuke Kawaguchi, creator of Jenkins and CTO at CloudBees, noted the creation of interoperability standards between CI/CD platforms will go a long way toward achieving that goal.

    It’s not clear yet just how widely the CDF will be supported by the entire DevOps community. While CloudBees provides one of the most widely employed CI/CD platforms, other offerings have started gaining traction, especially among enterprise IT organizations that tend to not have a lot of internal DevOps expertise.

    On the plus side, the CDF should help spur adoption of DevOps best practices even further. Adoption of DevOps processes within most enterprise organizations tends to be uneven at best. One team may be employing DevOps extensively, while the rest of the organization continues to rely on less agile legacy processes.

    The CDF also should play a significant role in enabling organizations to embrace microservices-based applications built using containers. While Jenkins is the most widely relied-on CI/CD platform today, it is built on what is now considered in many quarters to be a legacy Java Virtual Machine (JVM) platform. In contrast, Jenkins X Background an open source CI/CD platform for building and deploying modern cloud-native applications on Kubernetes. The Linux Foundation also plays host the Cloud Native Computing Foundation (CNCF), which oversees the development of Kubernetes.

    It still may take years for IT organizations to fully make the transition to DevOps and cloud-native applications. But in the months ahead, it’s clear the CDF intends to function as a force multiplier in enabling organizations to achieve that goal.

    — Mike Vizard