devopsy.kinsta.cloud

Tag: Jenkins

  • Is it Time to Reconsider CI?

    Is it Time to Reconsider CI?

    DevOps is old, in technology terms, at this point. Many of you have never developed under a waterfall and have no idea what tools like Visual Test were, and that’s actually very cool. While knowing other processes and tools does help make informed decisions, being immersed in the tools that matter today is more important. We have entire DevOps stacks built up that allow for the development, building, testing and, increasingly, deployment of apps in an agile environment.

    As always happens, market leaders and those who compete with them, complement them or serve a subset of the market have grown up and become standards. Indeed, application release orchestration (ARO) was a phrase coined in the early days of DevOps that fell away completely; I heard it recently and thought, “Oh, are we bringing that name back?” Vendors have come and gone; some merged into other products, and some are just plain gone. Again, that is a good thing as the market, its needs and the customer base are all mature. While it is terribly painful to be the user of a product that just goes away one day, the prevalence of open source in the CI/CD landscape does buffer that pain a bit. And the market gets better as tools fall away and new ones take their place.

    But sometimes, the movement is slow, just simply due to inertia. Most of us – and by us, I mean IT shops, and by most I mean a lot – use Jenkins. The number seems to be on a steady downward trend, and the tool’s maintainers have definitely taken steps to improve the parts that make organizations consider other solutions, but the trend is there.

    And I, personally, think that is a good thing. Stagnation is what made waterfall such a weakness in enterprise IT. Everyone did it, so that makes it good. Same is true for the old phrases we’ve all heard at least one variant of – “No one ever got fired for buying IBM”, or “No one ever got fired for buying Oracle”, “…Cisco”, etc. The obvious answer to those statements then, as now, was, “Well, maybe they should have.”

    The same is true for Jenkins. It is a great system that can do everything. And that’s part of the problem. If you have small projects or are a team using it standalone with light requirements that all revolve around getting a quality product out the door quickly, Jenkins just might be too much for you. In the early days, Jenkins wasn’t, but the plug-in architecture and years of competition and changing market needs have made it huge. And while there is arguably better documentation than any other CI, and inarguably the largest user base to ask questions of, it’s still weighty to use in place of a SaaS that comes pre-configured and you can just hand it a repository with build files in it.

    There are a lot of enterprises that need what Jenkins has grown into–it did not expand in a vacuum. But there are many, particularly new, small, agile orgs or standalone teams that just don’t.

    There are a lot of options out there, and it is worth taking stock of what you are trying to do with your build/test process, and build a list of things required, then look seriously at what’s available. For some, lightweight is good.

    Of course, as much as one can be in our technological age, I am a minimalist. I like my D&D to be a rules-light variant, my dev tools to have enough functionality but not bury me in extraneous stuff, laugh at Excel for including abstract functions and then selling to consumers, etc.

    At my company that does development, we do indeed use Jenkins. But our projects range from simple knock-it-out one-offs to complex systems with multiple servers and clients, so we feel it is best for that range of variation. And like many others, the investment to move old projects would be questionable without a driving need. Not that I get to do much development these days myself, but I do stay in it just enough to be dangerous. But we have asked the question and determined the answer. For now.

    If you have issues with Jenkins, look around. There are options. They are not for everyone, but your org isn’t everyone. Find the best tool, not the most convenient, and keep kicking rear.

  • Jenkins and JenkinsX in 2024

    Jenkins and JenkinsX in 2024

    This article is a response to “The Future of Jenkins in 2024” article, published late last year.

    If you’re a software developer, it is practically impossiblee that you haven’t at least heard of Jenkins. 

    Developed initially by Sun Microsystems as Hudson, Jenkins is written in Java and has been the leader in software life cycle management for many years.

    When Jenkins was born, the world of software development was less complex and didn’t have concepts like CI/CD, DevOps and so on. Developers worked with less complex architectures: A monolithic application with a database and the infrastructure based on the VM.

    But after the introduction of containerized environments (or containerized applications), the world changed! The deployable assets changed. The software life cycle changed and Jenkins changed.

    The Legacy Approach

    The success of Jenkins is largely due to its plugin approach. It allows you to install many useful plugins to manage every stage of the software life cycle, including clogging your repository, configuring unit tests, configuring custom behavior and more.

    Jenkins has a lot of competitors. The new competitors (and Jenkins), new technologies and the contributions of the open source community have adopted a modern ‘as-code’ paradigm. Jenkins itself introduced the as-code (scripted and declarative) pipeline (using DSL in groovy a JVM language).

    The git server provider has implemented the webhook behavior.

    Has been introduced the CICD term useful to refer to continuous integration and delivery.

    The infrastructure has been “dematerialized” with Kubernetes.

    The infrastructure configuration and provisioning has become scriptable (CasC and IaC).

    Today, you can install Jenkins wherever you want; on a Windows server, Linux server, as a containerized application or on Kubernetes. Jenkins installation is very dynamic.

    Moreover, you can use a different Jenkins architecture: Single node (basically with one master node), master-slave (with persistent slave) approach, master-slave runtime approach (using containers) and master-slave runtime approach using pod containers in a Kubernetes cluster. The last example creates a very powerful “cloud native tool” – but we’ll see this architecture in the next sections).

    But what does Jenkins offer today? The Jenkins subsystem today supports both models: The legacy approach and the as-code approach using Jenkins pipeline (declarative or scripted). Though many might think, “But … we’re DevOps and we don’t care about the legacy model; we’re interested in the as-code model.”

    Wiith the as-code model working within the GitOps paradigm, we’re safer in disaster recovery and also we can implement a self-service approach useful to:

    • delivery where you want
    • delivery for what you want
    • delivery when you want    

    With Jenkins you can do this always and also using a high level approach since Jenkins uses a DSL based on groovy that is a JVM language.

    In my opinion, Jenkins has some advantages:

    • No vendor lock-in (Theoretically, I can provision a groovy sandbox, and inside it, I can run my pipeline and it works fine. This is not completely true because if you use a Jenkins plugin like withCredential(), you do not have it inside a sandbox and other mechanisms, but – theoretically – this is possible.)
    • Jenkins works with OO and scripted programming languages like groovy. We can use high-level data structures and objects to implement the pipelines.
    • “We do not learn the core concept” about a specific tool (I mean, the high-level program languages are familiar to software engineers). If you know Java, then you know groovy; if you are experienced with Python, then you can work with groovy. 
    • No declarative approach with serialization languages such as YAML
    • If you don’t know bash programming you can use a fully-groovy pipe.
    • Is open source
    • Is pluggable
    • The Jenkins infrastructure is completely under our control
    • Shared commons libraries on the system side and project side
    • You can completely separate the project source code from the pipeline project. Segregation of tasks (and when you work in a big company where the development department is isolated from the “delivery” department, this is very important)  

    Jenkins Common Architecture

    Suppose a business project is versioned into a GitLab instance. The Jenkins pipeline is useful to manage the life cycle of this project as follows:

    pipeline {
    agent {
    label ‘maven’
    }
    stages {
    stage(‘unmarshall payload’) {
    steps {
    script {
    // to do this you can enable “allowEnv: true” in BuildConfig
    payload = utils.unmarshall(env.GITLAB_WEBHOOK_PAYLOAD)
    }
    }
    }
    stage(‘check GitLab Event & Project Details’) {
    when {
    expression {
    script {
    gitlabUtils.isPushEvent(payload)
    }
    }
    }
    steps {
    script {
    gitlabUtils.projectRepositoryDetails(payload)
    }
    }
    }
    …

    Assume that:

    • The pipeline is triggered with webhook mechanism
    • The GitLab payload is stored into GITLAB_WEBHOOK_PAYLOAD env var.
    • The pipeline use the global Jenkins shared libraries (imported on top with: @Library(‘jenkins-shared-libraries’) _ )
    • The gitlabUtils is a groovy bunch function that contains the “action functions” as you can see from the code snippet
    • Jenkins works with a plugin called generic webhook plugin (which is a really powerful plugin useful to manage the webhook request) 

    Focus on isPushEvent(payload) function:

    def isPushEvent(payload) {

        /**

          * This function is used to check if the hook event is a push type event.

        **/

      return payload != null && payload.get(‘object_kind’) == ‘push’

    }

     

    As you can see, using this approach, you can do anything everywhere and when you want, using an OO language. For example:

    stage(“environment steps”) {

         steps {

           script {

    …

    ClosureParams params = FactoryClosureParams.factory(currentEnvironment, servicesWithLatestTag,

    servicesWithoutDockerImages,

    serviceWithPreviousSameTag,

    deployedServicesBySector,

    composeRepoBranch,

    hostsGroupToDeploy,

    latestCommitArray,

    sectorToDeploy)

    …

    }

          …

          }

    The above snippet is a piece of a generic stage in our pipeline, where:

    • ClosureParams is a class
    • FactoryClosureParams is a class

    These classes are stored into Jenkins shared libraries repo, into the path src/it/example/jenkins/*.

    Yes we do. Because with Jenkins we can also work with OO programming!

    Focus on  FactoryClosureParams class:

    package it.example.jenkins

    import it.example.jenkins.ClosureParams

    import it.example.jenkins.ProductionParams

    import it.example.jenkins.DevelopParams

    class FactoryClosureParams {

        private static final String developRegex = “(.*dev.*)|(.*DEV.*)”

        private static final String productionRegex = “(.*prod.*)|(.*PROD.*)”

        

        public static ClosureParams factory(String environment, List servicesWithLatestTag,

            List servicesWithoutDockerImages,

            Map serviceWithPreviousSameTag,

            Map deployedServicesBySector,

            String composeRepoBranch,

            List hostsGroupToDeploy,

            List latestCommitArray,

            List sectorToDeploy)

        {

            /**

            * Factory method.

            **/

            if (environment =~ developRegex) {

                List<Object> argsD = [environment, servicesWithLatestTag, servicesWithoutDockerImages,

                    serviceWithPreviousSameTag, deployedServicesBySector,

                    composeRepoBranch, hostsGroupToDeploy, latestCommitArray, sectorToDeploy]

                println(“INFO. Building Develop parameters “)

                return DevelopParams.newInstance(argsD)

            }

            if (environment =~ productionRegex) {

                println(“INFO. Building Production parameters “)

                List<Object> argsP = [hostsGroupToDeploy, latestCommitArray, sectorToDeploy]

                return ProductionParams.newInstance(argsP)

            }

            return null

        }

    }

    Assume that:

    • FactoryClosureParams is a class into Jenkins’ shared library src root path

    We have seen how Jenkins works with its pipeline using a classical jenkins architecture.

    But … if I need to build a project from different repos, with different technologies inside – for example – in a docker container, in a single pipeline? How can I design this scenario?

     

    Suppose you have an application composed by:

    • A go (golang) agent with some routines that manipulates some files
    • A Python backend that call some external service, read and write files, and anything else
    • A Java web application that expose the results to localhost:8080

    We have seen that the Jenkins pipeline uses a global agent defined on top of the pipeline, but Jenkins also offers a mechanism for defining a stage-side agent.

    This mechanism becomes really powerful when Jenkins is provisioned into Kubernetes” because we can use this mechanism using the Kubernetes pods and obviously its containers.

    Before seeing how we can use this Jenkins setup, I want to explain to you why I said: “provisioned into Kubernetes.”

    You do not have the constraints to provision the Jenkins master on the Kubernetes cluster; you can manage this setup using a Jenkins plugin called Kubernetes Jenkins Plugin.   

    Jenkins as a Native CI/CD Tool in a Kubernetes Cluster

    With the Kubernetes Jenkins plugin, you can use the Kubernetes cluster API to start up some agents (previously configured: UX, CasC) at runtime like Jenkins slaves. 

     

    Let me do a recap of some points:

    • With the Jenkins pipelines we can define – by design – a pipeline-side agent or the stage-side agents
    • Using the Kubernetes Jenkins Plugin, we have the ability to set up a master-slave Jenkins architecture at runtime
    • Using the Kubernetes Jenkins Plugin, we can define more than one “agent type” and more then one container inside every agent
    • With the Kubernetes Jenkins Plugin you have a special step called container() which is useful to run part of your pipeline inside a specific container (in a specific pod)

    And then your pipeline becomes something like this:

    stage(“Testing project”) {

                steps {

                    script {

                        container(‘dotnet5’) {

    …

    …

    …

    …

    }

    stage(“Containerization. Auth, build and push”) {

                steps {

                    script {

                        prjId = pipelineConfig.get(‘project’).get(‘id’)

                        container(“aws-cli”) {

          aws …

    …

    …

    …

         container(‘docker’) {

               docker login …

                                }

    }

     

    As you can see, you can use the container() step what and when you want – inside the pipeline.

    This mechanism permits you:

    • To use a runtime agent (based on k8s pod lifecycle)
    • To use indistinctly all containers inside a pod agent
    • To use a multi agent (with multiply containers inside it) strategy – stage side
    • To build a project (single repo) based on different technology “one shot” (thus into a single – job –  pipeline run)
    • To build multi projects (multi repos) based on different technology, different versions of the same technology and so on “one shot” (thus into a single job)
    • To enforce running the pipeline inside a specific agent
    • To start a pod with complex container useful to run a complex analysis inside the pipeline (for example, DataOps or MLOps)
    • To use the Kubernetes resource, because your pipeline agents are Kubernetes pods
    • To use jenkins as CI/CD tool

    JenkinsX

    The JenkinsX project was born before some of the other modern and consolidated technology and tools, and some modern technologies developed faster than JenkinsX.

    For this reason, the JenkinsX project has become an open source architecture project.

    But… what do I mean when I say “open source architecture project”?

    I mean that JenkinsX is a set of open source tools useful to create a CI/CD stack into a cloud-native environment (Kubernetes).

    And yes, of course, JenkinsX also has custom resources like Preview.

    In fact as you can see, the below image represents the architecture of JenkinsX.

     

    Image: From the JenkinsX official site.

    A modular architecture that use the extensibility of Kubernetes and powerful tools like Tekton (for pipeline), Lighthouse (for ChatOps – in our case, this is useful to trigger the pipeline), Kubernetes secrets (stored in an external vault and linked by external secrets operator), interfaces (JX CLI, dashboard, octan), etc.

    This architecture is really powerful because we have all we need:

    • Pipeline engine (Tekton)
    • Trigger mechanism (Lighthouse)
    • Command CLI (JX CLI)
    • UI (octant)
    • Kubernetes resources and JX custom resources

    But we don’t always need all this.

    The latest observation will be the springboard to Tekton.

    Tekton

    As I said, some other technology has become stable and soon became a cloud-native standard (in Kubernetes). One of these is Tekton.

    Tekton is a pipeline engine. And offers all you need! It is a really big Kubernetes-centric project and adds a lot of Kubernetes CR to work.

    In summary, Tekton is formed by:

    • Tekton pipeline
    • Tekton triggers
    • Tekton CLI (tkn)
    • Tekton dashboard
    • Tekton catalog
    • Tekton hub
    • Tekton operator
    • Tekton chain   

    Tekton also exposes its APIs (available only for Pipeline and Task).

    The main Tekton resources useful to implement a pipeline are: Pipeline and Task (with their executors PipelineRun and TaskRun).

    Ok, but this article isn’t a documentation guide about Tekton or Jenkins, but if you want, you can learn more about this pipeline engine at the Tekton official site.

    The real point is: What is the relationship between Tekton and Jenkins?

    As I said before, Tekton is the pipeline engine beside the JenkinsX project, but also has another relationship with Jenkins (without “X”).

    Before explaining this relationship, I want to tell you how Tekton works with the Task resource. Basically, a Task is a collection of Steps (that is a common concept in Jenkins), and it runs as a K8ss pod.

    Look at the following snippet:

    apiVersion: tekton.dev/v1 # or tekton.dev/v1beta1

    kind: Task

    metadata:

      name: example-task-name

    spec:

      params:

    – name: pathToDockerFile

       type: string

       description: The path to the dockerfile to build

       default: /workspace/workspace/Dockerfile

    – name: builtImageUrl

       type: string

       description: location to push the built image to

      steps:

    – name: ubuntu-example

       image: ubuntu

       args: [“ubuntu-build-example”, “SECRETS-example.md”]

    – image: gcr.io/example-builders/build-example

       command: [“echo”]

       args: [“$(params.pathToDockerFile)”]

    – name: dockerfile-pushexample

       – image: docker.io/library/golang:latest

          name: ignore-unit-test-failure

          onError: continue

          script: |

               go test .

       volumeMounts:

         – name: docker-socket-example

           mountPath: /var/run/docker.sock

      volumes:

    – name: example-volume

       emptyDir: {}

        

    As you can see, every step is a minimal container (inside a pod) with its features and its commands or script to run. This is the same scenario that you can do with a classical Jenkins instance using Kubernetes Jenkins Plugin (with the container() step). 

    What do I want to prove with this observation?

    Probably nothing… but we have seen the past, the present and the future of one of the most important CI/CD tools in the history.

    Or… I want to say that, if you work in a cloud-native context with Kubernetes clusters and you have marked Jenkins as “obsolete tool” (for your context), you probably don’t know the deep powerful truth about Jenkins, and probably don’t know the Tekton project and the relationship with its “grandpa” Jenkins.

    In other words, I think that Jenkins (in some ways) will be a main character for the future of the DevOps chains!           

     

  • Using LLMs to Automate Pipeline Conversions From Legacy to Tekton

    Using LLMs to Automate Pipeline Conversions From Legacy to Tekton

    There have been many technological advancements in the market today, but the main hindrance is the cumbersome process of switching from the existing technology/framework to the new one. This is where teams and organizations are left behind in the tech curve. In the DevOps landscape, CI/CD pipelines are the backbone of DevOps, and how you work with them largely dictates your DevOps efficiency.

    CI/CD pipelines, too, like other major technologies and frameworks in the past, have seen various evolutions; the latest among them is being made reusable in an attempt to standardize CI/CD. However, here too, there are several cons:

    1. The pipelines are not proprietary to end-user enterprises and belong to the DevOps platforms which are not open-source
    2. Migrating to these platforms and pipelines is a huge task
    3. They may not be as efficient as depicted

    Speaking of new advancements in technology, most might be familiar with Tekton for CI/CD pipelines: It is Google’s open source framework that is now a graduated CDF project. It redefines how pipelines are built (through reusable tasks) and these tasks are open source and community-vetted. They allow for pipeline authorship, and enterprises can re-use them across platforms that are compatible with Tekton, thus preventing vendor lock-in.

    While this mitigates the first challenge of pipeline ownership and migrations, however, the same problem plagues DevOps teams – transitioning from existing legacy technologies/frameworks to newer, much more capable frameworks like Tekton.

    However, what if there was a way to seamlessly automate your transition from legacy pipelines to cutting-edge, reusable Tekton pipelines without toil? This is precisely what the article aims to explore – automating the conversion of Yaml scripts of existing pipelines to Tekton Yaml scripts.

    Let us consider the three most popular CI/CD platforms for this use case – Jenkins, GitLab and Azure DevOps. Before we proceed to the proposed approach to achieve this (which is detailed in the second half of this article), let us first look at the limitations of Jenkins, GitLab and Azure Pipelines and why one should convert.

    Common Limitations of Jenkins, Azure and GitLab Pipelines

    Resource Intensive: Legacy pipelines can be resource-intensive, especially for larger organizations or projects. Managing server resources, optimizing configurations and ensuring sufficient hardware can be a challenge.
    Complex Setup: Setting up and configuring Jenkins, GitLab and Azure DevOps can be complex, particularly for self-hosted instances. This complexity may require experienced administrators and substantial time investment.
    Plugin and Integration Compatibility: All three platforms may face compatibility issues with certain plugins or integrations.
    Scaling Challenges: Scaling in organizations, especially when dealing with multiple microservices, necessitates a shift from individualized scaling to a template-based approach. Failing to do so leads to a chaotic increase of pipelines, undermining standards and organizational processes.
    Costs: While both Jenkins and GitLab offer free and open-source versions, some advanced features or capabilities may only be available in higher-tier paid plans. This can lead to cost considerations at scale.
    Customization Complexity: These platforms can be complex to customize workflows and integrate custom scripts as they require advanced scripting knowledge.
    User Interface Complexity: Though they have feature-rich interfaces, it can sometimes be overwhelming for new users to navigate the platforms and find specific settings or options.
    Limited Native Integration & Plugins: The native integrations of Jenkins, GitLab and Azure may not cover the full spectrum of tools and services a team may need. Additional integration efforts or third-party tools may be required to bridge gaps in the development workflow.

    As can be seen in the above limitations, many are common including scalability challenges, being resource intensive, limited integrations etc. All the challenges boil down to one common denominator: Complex pipeline management at scale.

    This is where a framework like Tekton can help tackle all the challenges at once, by just easing pipeline management: with its K8s-native pipelines that are reusable, open source and tasks that are community-vetted.

    Tekton Pipelines: Advancing CI/CD with Kubernetes-Native Automation

    Tekton, an open-source project, is rapidly gaining traction as a game-changer in the world of CI/CD pipelines. Its Kubernetes-native design is a standout feature, seamlessly integrating with Kubernetes clusters to harness its scalability and resource efficiency.

    Furthermore, Tekton’s cloud-native roots ensure it can execute tasks in lightweight, ephemeral containers, optimizing resource utilization. The ability to declare CI/CD pipelines as code, coupled with GitOps practices, enhances automation and reproducibility.

    How are Tekton Pipelines Different?

    Kubernetes-Native Automation: Tekton pipelines are deeply integrated with Kubernetes, utilizing Custom Resources and Operators to define and execute CI/CD workflows.

    Declarative and Modular: These pipelines are defined declaratively using YAML manifests, promoting version control, collaboration and code reviews. The modular design of tasks and pipelines enables teams to create reusable components for building, testing and deploying applications.

    Containerization and Isolation: Each task in a Tekton pipeline runs within a dedicated container, ensuring isolation and reproducibility of the build and deployment environment.

    Event-Driven Architecture: Tekton pipelines can be triggered by various events, such as code commits, pull requests, or external events, providing a responsive and event-driven CI/CD process.

    Scalability and Efficiency: Tekton pipelines leverage Kubernetes’ inherent scalability, enabling efficient execution of tasks across a dynamic pool of worker nodes.

    Visibility and Observability: They offer built-in monitoring and logging, allowing teams to gain insights into pipeline execution, track progress and diagnose issues.

    Integration and Extensibility: Tekton pipelines can be easily extended with custom tasks and integrations, enabling teams to connect with various tools, services and cloud platforms.

    Cloud-Native Best Practices: By embracing Tekton pipelines, organizations align with cloud-native best practices, promoting the use of Kubernetes and containerization for streamlined application delivery.

    Why Transition From Jenkins, GitLab, and Azure Pipelines to Tekton Pipelines?

    Migrating from Jenkins, GitLab, or Azure pipelines to Tekton pipelines can be a strategic choice, especially for organizations deeply entrenched in Kubernetes and cloud-native methodologies.

    Optimize Kubernetes-Centric Workflows: Projects tightly integrated with Kubernetes may find Tekton pipelines more aligned with their ecosystem.

    Accelerate Cloud-Native DevOps: Organizations aiming to embrace a cloud-native approach and leverage Kubernetes infrastructure.

    Performance and Scalability: Tekton pipelines run on clusters that can dynamically be created and destroyed according to demand, thus taking resource optimizations to the next level.

    Cloud-Native Best Practices: Best practices, such as containerization, declarative configuration and infrastructure-as-code are a natural fit due to their Kubernetes-native design.

    Proposed Workflow for Automating Pipeline Conversions at Scale

    The overarching objective of this workflow is to facilitate a seamless conversion process using large language models (LLM) without delving into the intricacies of fine-tuning but rather using the benefits of prompt engineering. Central to this approach is the use of the chat context and the strategic application of roles for LLM models to establish context. Along with this, sample pipelines are leveraged to train and fine-tune the model to define how the corresponding YAMLs of all these platforms look for any given pipeline.

    The use of prompts and instructions tailored to fit the use case of generating contextual pipeline YAML and conversion of Jenkins/Gitlab pipelines to Tekton pipelines makes this workflow possible. We maintain a context and a contextual mapping of real-world examples and the most appropriate prompt designs to produce the most accurate responses. The role of users in this context adds weight and further refines the response received.

    However, before proceeding into details and examples, let’s set the base for what exactly generative AI means and its context in our use case:

    Generative AI and LLM for Automating YAML Script Conversions

    Generative AI, such as LLMs, create content like text, benefiting from patterns in large datasets. Large language models (LLMs), with billions of parameters, excel in generating human-like text by understanding language structures. They have broad applications, including natural language understanding, summarization, translation and content generation, promising automation and human-like interactions with AI.

    Adapting the Model to Solve Real-world Use Cases

    Fine-tuning and prompt engineering are two common methods used to train large language models (LLMs) like GPT-3, BERT and others. These methods help adapt pre-trained models for specific tasks or generate contextually relevant responses. Both fine-tuning and prompt engineering play crucial roles. Fine-tuning helps the model comprehend complex use cases, while prompt engineering provides detailed requirements, extracting the best results from the model.

    Why Prompt Engineering?

    Prompt engineering stands out as a highly advantageous approach in specific use cases where precision, control and customization over AI model behavior are paramount. This method empowers developers and users to meticulously design input prompts, effectively guiding AI models to produce contextually relevant and desired responses.

    Prompt engineering is both an AI engineering technique for refining large language models (LLMs) with specific prompts and recommended outputs and the term for the process of refining input to various generative AI services to generate text or images.

    The process of prompt engineering to fit our use case includes :

    Defining the goal: The first step in the process of AI prompt engineering involves setting a clear objective.

    Crafting the initial prompt: With the goal in mind, it’s time to draft an initial prompt. This could take the form of a question, a command or even a scenario, depending on the goal.

    Testing the prompt: The initial prompt is then input into the language model, and the response is analyzed.

    Analyzing the response.

    Refining the prompt: With the insights gathered from testing and analysis, it’s time to revise the prompt. This could involve making it more specific, adding more context or changing the phrasing.

    Iterating the process: The testing, analyzing and refining steps are repeated until we are satisfied that the prompt consistently guides the model toward generating the desired response

    To conclude, here’s an example of typical YAMLs from Jenkins, GitLab and Azure DevOps, and how the same YAML looks when converted to a Tekton YAML. This goes a long way in facilitating teams to finally make a jump from their existing pipelines to cutting-edge reusable pipeline frameworks:

    Jenkins YAML

    GitLab YAML

    Azure YAML

    Converted Tekton YAML

  • The Future of Jenkins in 2024

    The Future of Jenkins in 2024

    As software development practices evolve, the tools that support them must also adapt to stay relevant. Jenkins, the venerable automation server used by many DevOps teams, is at such a crossroads as it faces the changing landscape of cloud-native applications and Kubernetes adoption. 

    This article examines Jenkins’ current utility, particularly its features that have made it a staple in CI/CD practices, and addresses the question of whether it will maintain its relevance in 2024 and beyond.

    How Jenkins is Used by DevOps Teams

    CI and CD

    Jenkins has become an integral part of many DevOps teams due to its CI/CD capabilities. Continuous integration (CI) is a software development practice where developers regularly merge their code changes into a central repository. Continuous delivery (CD) is the practice of delivering the software in short cycles to ensure that it can be reliably released at any time.

    Jenkins supports the CI/CD approach in a number of ways. First, it allows developers to automate code integration. This means that whenever code changes are pushed to the repository, Jenkins can automatically build and test the project. Second, Jenkins allows for the continuous delivery of software by automating simple deployment processes (although it lacks full deployment orchestration capabilities).

    Build Pipeline Visualization

    Another key feature of Jenkins is its build pipeline visualization. This feature allows DevOps teams to visually represent the process of software delivery from code integration to deployment. The pipeline visualization provides a clear and intuitive representation of the software delivery process, making it easier for teams to understand and manage the process.

    Build pipeline visualization in Jenkins is customizable, allowing teams to tailor it to their specific needs. For example, it can be customized to show the status of each stage of the delivery process, from code integration to testing to deployment. This allows teams to quickly identify any issues or bottlenecks in the process and address them.

    Automated Testing

    Automated testing is another area where Jenkins shines. With Jenkins, DevOps teams can automate the process of testing their code by running scripts and integrating with automated testing tools as part of the build process.

    Jenkins supports a wide range of testing frameworks and tools, allowing teams to choose the ones that best fit their needs. Moreover, Jenkins can be configured to automatically run tests whenever changes are made to the code. This ensures that any issues or defects in the code are detected early and can be addressed promptly.

    Workflow Scripting

    Workflow scripting is another key feature of Jenkins in DevOps teams. With Jenkins, teams can script their workflows, automating the process of software delivery. This allows teams to standardize their workflows, ensuring that each delivery process is consistent and predictable.

    Jenkins supports the Groovy scripting language for workflow scripting. This means that teams can use the power and flexibility of Groovy to script their workflows. Moreover, Jenkins provides a user-friendly interface for scripting workflows, making the process easy even for those who are not familiar with Groovy.

    Current Usage and Adoption Statistics

    As of 2023, Jenkins remains a leader in the CI/CD space. With its extensive plugin architecture and large user base, it is used by thousands of organizations worldwide. According to the 2022 DevOps Survey by the Cloud Native Computing Foundation (CNCF), Jenkins was used by over 50% of respondents for automation and CI/CD, maintaining a significant presence in the market.

    According to the Jenkins website, there are over 1 million active Jenkins users and more than 200,000 active installations worldwide, with the platform handling 1 million+ jobs per day. It had about 44% of the global market share for CI/CD in 2023.

    The growth trajectory of Jenkins, however, has shown signs of plateauing when compared to newer, more cloud-native-focused CI/CD tools. This observation is backed by Google Trends data, reflecting a steady but not sharply increasing interest in Jenkins over recent years.

    Despite facing competition from emerging tools designed for cloud-native development, such as GitHub Actions, GitLab CI and CircleCI, Jenkins has responded by evolving its ecosystem. The creation of Jenkins X is a strategic move to cater to Kubernetes users and address some of the criticisms related to cloud-native development. However, as we’ll discuss, Jenkins X has not been readily adopted by the industry as of yet.

    Limitations of Jenkins in Cloud-Native Environments

    While Jenkins is a powerful tool for DevOps teams, it has serious limitations in cloud-native environments. These limitations stem from the fact that Jenkins was originally designed for traditional, monolithic applications, not for the dynamic and distributed nature of cloud-native applications.

    Statefulness

    One of the critical limitations of Jenkins in cloud-native environments is its inherent statefulness. Cloud-native architectures typically favor stateless applications, which are easier to scale and manage in dynamic environments like Kubernetes. 

    Jenkins, however, relies on maintaining state—it stores configuration, build history and user data, making it challenging to scale and manage efficiently in a cloud-native setup. This statefulness requires persistent storage, which complicates deployments in distributed systems where resiliency and horizontal scaling are key. Moreover, the risk of data loss or inconsistency increases in cloud-native environments, particularly during scaling operations or when recovering from failures.

    Configuration Complexity

    Jenkins is often criticized for its complex configuration requirements. Setting up Jenkins in a cloud environment involves intricate configurations that are not only time-consuming but also prone to errors. This complexity is exacerbated when integrating with modern cloud-native tools and services, requiring extensive customization and maintenance of plugins. 

    In addition, Jenkins’ traditional master-slave architecture was not designed for dynamic cloud environments, where resources need to be provisioned and deprovisioned rapidly. This leads to a significant operational overhead for DevOps teams.

    Immutable Infrastructure

    Adopting an immutable infrastructure approach is a cornerstone of cloud-native environments, promoting the idea of replacing or rebuilding servers rather than modifying them. 

    However, Jenkins’ traditional model does not align well with this principle. It typically requires modifications to its runtime environment for updates or changes, contrasting with the cloud-native approach of immutable deployments. This can make it challenging to maintain consistency and reliability in cloud-native systems where automation and minimal manual intervention are paramount.

    Microservices Limitations

    Jenkins was originally designed for monolithic architectures and doesn’t naturally fit into microservice-based applications, which are prevalent in cloud-native environments. In microservices architectures, applications are broken down into smaller, independently deployable services. 

    Jenkins, however, tends to treat applications as monolithic units, making it less efficient for handling multiple, independent microservices. This limitation affects its ability to provide fine-grained control over individual services’ build, test and deployment processes. Consequently, Jenkins may struggle with the high degree of automation, flexibility and scalability required in microservices environments.

    What is Jenkins X?

    Jenkins X is an open source project that provides pipeline automation, built-in GitOps, and preview environments to help teams collaborate and accelerate their software delivery at any scale. It is an improved version of the original project, designed specifically for Kubernetes, a popular container orchestration platform.

    Jenkins X streamlines the technical aspects of building, testing, and deploying software. It automates the installation of popular tools like Helm and Draft, making it easier for developers to get started. It also provides pre-configured pipelines so teams can start delivering software right out of the box.

    Jenkins X also introduces a new Jenkinsfile format, Jenkinsfile.yaml, which provides a more straightforward and human-readable way to describe pipelines. This new format, combined with the automation provided by Jenkins X, helps reduce the complexity of pipeline configuration and management in Kubernetes.

    Limitations and Criticisms

    Jenkins X has not been widely adopted by the software industry and has not achieved the respect and prominence of its predecessor, Jenkins. Let’s review the main limitations and criticisms of the tool.

    Lack of Maturity

    Jenkins X’s relatively recent entry into the market means it has not had the time to mature like its predecessor. This lack of maturity can manifest in several ways, including fewer features compared to established tools, fewer community-tested plugins, and potentially more bugs or unresolved issues. Due to the fast-paced evolution of cloud-native technologies, tools need time to adapt and stabilize; Jenkins X is still in this phase of its lifecycle.

    Moreover, a mature tool typically has extensive documentation and a large community of users who can offer support and share best practices. While Jenkins X has seen growth in its user base, it still lags behind more mature CI/CD solutions in these areas. This can lead to challenges when teams are looking to solve complex problems or when they require advanced support.

    Limited to Git Projects

    Another criticism of Jenkins X is that it’s limited to Git projects. While Git is the most popular version control system, it’s not the only one. Some teams may use other systems like Mercurial or SVN, and these teams would be unable to use Jenkins X.

    Even within the world of Git, Jenkins X has limitations. For example, it doesn’t support all Git hosting platforms. While it works with popular platforms like GitHub and Bitbucket, it doesn’t support others like GitLab. This can be a significant limitation for teams that use unsupported platforms.

    Community Support

    Finally, there are concerns about community support for Jenkins X. While the project is open source and theoretically has the potential for wide community involvement, some critics argue that it’s primary source of support is CloudBees, the company behind Jenkins.

    These critics worry that if CloudBees were to withdraw its support, the project could falter. They also argue that the project’s focus on certain tools and platforms (like Kubernetes and Git) reflects CloudBees’ business interests more than the needs of the broader DevOps community.

    Conclusion

    In conclusion, Jenkins’ role in the CI/CD landscape has been pivotal, and its continued evolution with projects like Jenkins X demonstrates an effort to adapt to the needs of modern, cloud-native environments. While Jenkins X has yet to reach the widespread adoption and maturity of its predecessor, its specialized focus on Kubernetes and GitOps reflects the shifting priorities in software development.

    Jenkins, despite showing signs of plateauing in growth, still possesses a strong user base and an established presence in the market, supported by a vast plugin ecosystem and a robust community. Its limitations in cloud-native contexts—such as managing statefulness, configuration complexity, and supporting immutable infrastructure—pose challenges that Jenkins X seeks to address.

    As we approach 2024, the relevance of Jenkins will likely hinge on its ability to integrate more seamlessly with cloud-native technologies and the willingness of the community to evolve its ecosystem. The breadth of its adoption and the depth of its integration in existing development pipelines mean Jenkins will continue to be a significant player in the DevOps toolchain. However, its long-term relevance will depend on the project’s ability to innovate and remain adaptable in a landscape that is rapidly embracing cloud-native solutions.

  • CloudBees Unfurls Dual CI/CD Strategy at DevOps World Event

    CloudBees Unfurls Dual CI/CD Strategy at DevOps World Event

    During a DevOps World event, CloudBees took the wraps off the first major update to the open source Jenkins continuous integration/continuous delivery (CI/CD) platform made in the past several years.

    At the same time, the company also unveiled a proprietary DevSecOps platform based on Kubernetes that is optimized for building and deploying cloud-native applications based on portable Tekton pipelines.

    Shawn Ahmed, chief product officer for CloudBees, said the updates to Jenkins make it possible to horizontally scale Jenkins controllers while at the same time ensuring high availability. The latest update also provides a workspace caching mechanism to simplify sharing them across multiple jobs in a way that significantly reduces the amount of compute and storage resources consumed, he added.

    CloudBees also added a Baseline capability to the Jenkins repository to improve startup times without causing build storms.
    Collectively, those capabilities will increase the number of jobs that can be simultaneously executed to improve overall productivity, said Ahmed.

    CloudBees has also added a Pipeline Explorer tool in the cloud-based edition of Jenkins that enables DevOps teams to visually debug and troubleshoot pipeline jobs. That tool will also provide the foundations for self-healing and artificial intelligence (AI) capabilities that CloudBees plans to add to the Jenkins platform later this year and into 2024, added Ahmed. For example, AI tools will surface suggestions on how to fix a problem in a pipeline, he noted.

    The DevSecOps platform, meanwhile, leverages Tekton pipelines to provide DevOps teams with more granular control over how pipelines can be orchestrated, said Ahmed. It also provides access to GitHub Actions-style domain-specific language (DSL), support for feature flagging, security, compliance, pipeline orchestration capabilities, analytics and value stream management (VSM) tools that can be deployed by an internal IT team or accessed via a single or multi-tenant managed cloud service.

    Those built-in capabilities will make it simpler for platform engineering teams to centrally manage building and deployment of cloud-native applications by enabling developers to self-service their own engineering requirements via an internal developer platform (IDP), said Ahmed. The overall goal is to provide a more integrated approach that breaks down software engineering silos that exist in many existing DevOps workflows, said Ahmed.

    Each DevOps team will need to decide for themselves to what degree to move forward using Jenkins to build and deploy applications versus opting for a more modern alternative in the form of the DevSecOps platform. CloudBees, however, is committed to providing plug-ins and integrations across both platforms to enable DevOps teams to protect their investment in existing tooling, noted Ahmed.

    The status of Jenkins X, an open source CI/CD platform based on Kubernetes that CloudBees previously donated to the Cloud Delivery Foundation (CDF), is not immediately clear. Regardless of the approach, however, there is a clear opportunity to revamp DevOps workflows by upgrading Jenkins or switching to a more modern platform such as DevSecOps. The challenge, as always, is determining the pace at which any such transition might be made.

  • CDF Report: Steady Growth in Number of Jenkins Jobs Run

    CDF Report: Steady Growth in Number of Jenkins Jobs Run

    The Jenkins project that operates under the auspices of The Continuous Delivery Foundation (CDF) reported this week that over a two-year period, usage of pipelines grew 79%, with total workloads defined in the continuous integration/continuous delivery (CI/CD) platform growing 45% over the same period.

    Based on analysis conducted from June 2021 to June 2023, the report found the number of jobs defined increased from 27,105,176 jobs to 48,625,398 jobs per month. The number of all jobs defined on Jenkins increased from 50,785,205 to 73,746,418 jobs per month (45%).

    Sacha Labourey, a member of the governing board for the CDF and chief strategy officer at CloudBees, a provider of commercially supported instances of the open source Jenkins CI/CD platform, said it’s clear that more applications are being constructed by more organizations than ever using DevOps workflows.

    At present, the CDF reported there are now more than 600 active contributors to the Jenkins project. That level of support has made it possible for Jenkins to become a standard CI/CD platform. DevOps teams know there is a substantial open source community they can count on for support, said Labourey.

    The challenge now is how to centrally observe DevOps workflows to accelerate application development while at the same time giving developers more freedom to innovate. Platform engineering principles represent an opportunity to achieve that goal, as long as developers still have an opportunity to innovate and collaboratively address bottlenecks in DevOps workflows as they occur.

    The platform engineering team should be observing processes and proactively making recommendations and define a set of guardrails that ensure DevSecOps best practices are followed, he added.

    Otherwise, application developers are likely to view platform engineering as another effort by centralized IT teams to take power away from the developer, noted Labourey.

    Striking that balance is especially critical as the number of cloud-native applications being built and deployed alongside classic monolithic applications has significantly increased, he added. DevOps teams need to be able to centralize observability of what are two distinct styles of application development that tend to occur at a significantly different pace from one another, Labourey said.

    Hopefully, advances in artificial intelligence (AI) will make it easier for DevOps teams to strike the right balance between management and innovation. In the meantime, however, as the pace of application development continues to accelerate, each DevOps team will need to determine how best to evolve their existing workflows. That’s especially critical at a time when economic headwinds are requiring organizations to find ways to increase existing developer productivity rather than hire or contract additional developers.

    In the meantime, it’s apparent that DevOps, as a methodology for building and deploying applications faster, is here to stay. The next major challenge is finding a way to achieve that goal at levels of scale that require automation everywhere from the network edge to the cloud.