devopsy.kinsta.cloud

Tag: CI/CD pipelines

Infrastructure as Code and Six Key Automation Concepts

This summer, I walked into an Infrastructure as Code (IaC) flame war because I was blissfully unaware of just how broadly IaC had been reinterpreted. Reviewing the comments, it’s clear that thinking has evolved since Kief Morris’ 2016 Infrastructure as Code tomb. So, after strapping on my asbestos suit, I decided to dig into what is creating such a wide range of opinions and see if I can share some 2019 insights.

Why does IaC provoke such passion? The reason is simple: As an industry, we aspire to something that delivers on our collective vision of fully-automated, self-governing infrastructure. In the 00s we called it autonomic computing, then software defined data center (SDDC) and, now, IaC. Sadly, IaC is not it—or, more specifically, it’s not it alone.

The actual definition of IaC is way smaller than what people want it to be.

My take-away from IaC discussions is they are built around six key automation themes. Let’s take a few minutes to examine each of the themes. Then we’ll have a basis for answering “If that’s not IaC, then what is it?”

Programmatic Configuration (IaC)

Programmatic configuration captures the idea that we can define our infrastructure in a machine readable format. For example, you describe a planned cluster build in YAML to execution by a tool such as Terraform that requests the resources from your dynamic infrastructure and, hopefully, tracks that resource descriptors you get back from that request. Basically, it’s just your IaaS input format.

Dynamic Infrastructure (or IaaS)

It describes a system that allows operators to change the resource allocations via an API. It’s a critical part of the discussion because there must be a dependable way to pragmatically change the infrastructure. Some would consider this cloud; however, we should not assumes either managed services or VMs because dynamic infrastructure is much broader.

Desired State (or State Reconciler)

Desired state actively compares dynamic infrastructure to programmatic manifest and makes automatic adjustments to keep them synchronized. Typically, the idea of maintaining state is separated from configuring systems. Kubernetes and other Autoscaling cluster managers perform this role.

Deployment from Source Control (or Immutability)

It implies that all of the configuration components of the system can be stored as versioned artifacts and deployed from source. A Dockerfile that builds a container is a good example of an immutable process and the generated artifact. The repeatability offered by deploying from source control is transformative compared to past generations of operations that relied on either manual or scripted changes to production systems. Immutability is not IaC: We are focused on an generated deployment artifact rather than managing the infrastructure.

Composable (or Modular) Automation

It allows us to manage complex structures through layers of abstraction. In concept, composability is as simple as using platform plugin to script against an API.

CI/CD Pipeline

CI/CD pipeline is the glue that pulls all these pieces together to create a resilient environment by continuously adjusting the system components based on the operators’ IaC descriptions. The pipeline relies on having dynamic infrastructure managed to desired state as a target; however, it adds the critical concepts of staged changes over time to the system. Since we need to create systems that can adapt and improve over time, the CI/CD pipeline provides mechanism for controlled implementation change over time.

Together, these six concepts reflect an integrated new way to think about the infrastructure that is beyond IaC. They allow us to maintain and control our systems over time in a way that is resilient and automatic. More importantly, they bring complex infrastructure problems back into terms that people can comprehend by creating more deterministic and predictable chains of change. That allows us to keep our hands out of the infrastructure and our minds focused on what we are delivering: a continuously integrated data center.

Since all six concepts are interdependent, continuously integrated data center (CI DC) completely captures our aspirations for IaC. The term intuitively builds from source code into a fully operational infrastructure. More importantly, CI DC demands that we acknowledge the ongoing maintenance required. For that reason, I’ve embraced CI DC to replace IaC.

Does the term work for you? How do you see IaC or CI DC being delivered in 2020? I’d like to hear your thoughts.

— Rob Hirschfeld

December 12, 2019
Extending Automation to Your First Code Commit

I am a big advocate for software reuse, and an even bigger advocate if that reuse comes in the form of software automation. My team and I rely on reusable software processes that accept simple inputs to produce predictable results. We have tight deadlines with requirements that often seem impossible. Automation makes us more efficient and our work more predictable.

Once upon a time, we used glue and bubble gum to cobble together means by which to build, then test and then eventually deploy application versions. It worked until it didn’t, and industry opportunities arose for mechanization. Enter continuous integration (CI), continuous deployment (CD) and DevOps, in part.

You likely know this and have already taken advantage of CI/CD platforms such as Jenkins, CircleCI and others. The simplicity of the as-code model allows us to turn easy-to-create declarations into the otherwise complex provisioning of servers, databases, caches and file systems, along with the execution of code to leverage what was provisioned.

Life as a developer has been made better using these as-code tools in the tool chain. Spinning up a Kubernetes cluster on Google to (re-)deploy an image built, tested and pushed to DockerHub from a CloudBees/CodeShip pipeline is something I rely upon each time I commit to my Git host.

In this article, I am going to talk about Project-as-Code to extend automation to the first code commit with a working app running through a functional CI/CD pipeline and deployed to a production-like environment.

Barriers Keeping Enterprise IT Departments from Coming Along for the DevOps Ride

Unfortunately, basic usage of all these great tools comes with the added complexity and cognitive load cost. Understanding Docker image creation, HashiCorp Terraform syntax for infrastructure-as-code, Kubernetes cluster deployment and more are just a few of the competencies developers and operators need to have.

Learning a new set of tools every few years comes with the territory, but there’s no reason to ignore new potential avenues for automation. I’ve wondered if there is a better way to abstract and automate the setup of these common pieces in the modern software production pipeline.

Another common issue I’ve noticed—these pipeline components often get set up sequentially, with the first meaningful code commit being the blocker to setting up all the other pieces of the CI/CD pipeline. Without this commit, these tools sit relatively idle while waiting to provide value.

For those of us who work on new projects regularly, getting things started has always had its issues, especially for consulting scenarios where requirements and team dynamics differ between clients. We are disciplined enough to first reach in our toolbox to reuse as much as possible. Still, DevOps has yet to be adopted wholesale in a standard way by many enterprises.

For these situations, I find the benefits of DevOps are easy to explain and understand. Developers like to reuse and fewer barriers, the business likes savings and an enterprise likes productivity and innovation. However, to inject those benefits is still a very manual, time consuming process, making the overall sell of DevOps harder.

The purpose of DevOps automation is to simplify tedious error prone manual processes. I see another opportunity for automation: entire DevOps projects using the same as-code model we have become accustomed to.

Generating Your First Meaningful Commit “As-Code”

Why not use a straightforward declarative model-driven means of not only generating tech-stack specific application source code, but all the config files that lead to DevOps being automated. With this, the toolchain learning curve is minimized, and idle tools now are immediately called to action. The result is a self-sufficient operational DevOps environment working through an application code base that comprises a fully deployable executable first code commit. Developers then get their reusable assets and environment(s), the business sees cost and time savings, and the enterprise gets the productivity and innovation necessary to stay competitive.

In an article written by Pete Hodge titled “Hello production,” he goes into great detail about the value of quickly standing up a flowing CI/CD pipeline leading to a functioning app deployed to production. He contends deploying even a useless application into production, as soon as you can, is the right way to start a new project because it pulls unknown risk forward, opens up parallel streams of work and establishes good habits. Project-as-Code standardizes this thinking with model-driven automation.

The next generation of DevOps is unfolding with automation remaining a key component to expand scale, velocity and productivity. As automation has worked its way up the toolchain, automating the automation behind DevOps will shift the paradigm to improve how projects are created, how DevOps is implemented and the quality and features of the first commit into a Git repository. What now takes weeks to months could instead take hours to days, falling in line with our expectations now met by other successfully mechanized aspects of DevOps.

Hopefully this article will serve as a starting point for your next DevOps project and how Project-as-Code automation can be a part of getting you started easier and faster.

— Steven Randolph

November 1, 2019
How to Implement an Effective CI/CD Pipeline
DevOps bridges the gap between development, operations and IT services teams efficiently. To foster a DevOps culture, implementing the right DevOps tools with the right DevOps process is essential. Continuous integration/continuous delivery/continuous deployment (CI/CD/CD) help developers and testers ship the software faster and safer in a structured environment.

Unlike the traditional software life cycle, the CI/CD implementation process gives a weekly or daily update instead of monthly or quarterly. The fun part is customers won’t even realize the update is in their applications, as they happen on the fly.

A Brief Insight About CI/CD/CD Terminology
- Continuous Integration: CI is a process in which developers integrate new codes into a shared repository several times a day. This is a better approach than the traditional one, where the developer will build new codes in isolation and then integrate into the main repository at the end of the project lifecycle. The primary goal is to detect any integration bugs at the initial stage so it can be rectified quickly. It triggers a new build whenever a new code merges with the existing main repository. Test runs are performed against these new builds to check for any breakage.
- Continuous Delivery: At the end of the CI, CD comes in. CD ensures to automate the software delivery process and commits to deliver the integrated code into the production stage without any bugs or delay. CD in the DevOps implementation process, helps developers merge the new code with the main branch consistently so they can build an instant software-ready product. It is responsible for checking the quality of the code and performs tests to check whether it can release the functional build to the production environment.
- Continuous Deployment: The most critical part of the automation occurs at this stage of the delivery pipeline. Whenever there is an important change in the code, the corresponding build and deployment occur simultaneously. It is achieved through continuous deployment process where it allows to perform live deployment changes for every code modification that passes through the CI stage. During this stage, there is no manual intervention right from the initial code until the code is in the production stage.
Benefits for Implementing CI/CD

The ultimate aim for any software application is to reach its potential customers faster than ever before. This is exactly what the CI/CD pipeline offers to any business model that leverages a software process which reduces the risks in each build and helps the end product to reach the customer quicker.
- Achieve Faster Feedback Through CI Tools: For every committed code, the corresponding tests are run simultaneously to avoid any link breakage at later stages. Faster feedback helps to check the quality and impact of the code.
- Greater Visibility: With CI/CD pipeline setup, the whole process of new builds, test results and any issues with new builds can be analyzed. This transparency allows the developers to know which changes in the build led to the brokerage and to avoid them in the future.
- Early Bug detects: Conducting various types of automated testing allows to identify any bug issues at an early stage without any last minute surprises. These automated tests along with few manual test runs, help to fix any issues that may arise.
The Different Stages of CI/CD Pipeline Implementation

To get the full benefits of the modern DevOps environment, embarking on a scalable CI/CD DevOps implementation is important. It lays a strong foundation for building an agile DevOps for your business.

Tools for the CI/CD Process

Tools are the magic wand that help automate a successful CI/CD pipeline and ensure a faster software delivery.
- Open-Source Tools or Proprietary: Depending upon the budget and the client’s requirement, business can opt for open-source tools such as Jenkins for automating the CI/CD process.
- Cloud or Self-Hosting Services: To self-host your CI/CD application, tools such as Jenkins and Cruise Control help to setup, configure and monitor the application effectively. Otherwise, you can get SaaS providers to host on cloud services.
- Build Status: Choose a tool which embraces the transparency and visibility of the entire process through appropriate notifications, through email or other type of communication.
Possible Challenges of CI
- You might need a discipline strategy to address an issue by the CI/CD pipeline, otherwise the whole fast feedback is in vain.
- Slower builds eventually take place as newer builds keep adding to the product.
- Failure to train the resources to adapt to new tools and patterns.
Wrapping Up

In this era of faster software shipping and building quick releases on the fly, adopting a more robust project cycle that accelerates the delivery pipeline is crucial. Implementing DevOps services is the optimal way to get the effective collaboration between cross-functional teams through the CI/CD process.

— Jay Elango
October 1, 2019
Bringing 5G-Enabled Digital Services to Life Calls for Cloud-Native Operations

Spotify, Netflix, IoT connectivity and cloud storage are a few examples of the digital services providing new options for fun and function to consumers and businesses alike, while representing a growing boon for communications service providers (CSPs).

Gartner predicts by 2020, nearly one quarter (23%) of CSPs’ large enterprise business communication revenues will be driven by digital services—up a staggering 10 percentage points from 13% of revenues in 2017. This gain should come as no shock to developers who favor building in the cloud or using tools such as Microsoft Teams to collaborate, or who stream video from HBO Go or skim stories in Apple News during their commutes. Today’s customers (consumers or businesses) expect value-driven, digitally enabled services that inspire personalized and meaningful experiences with their brands of choice.

The good news is companies are eager to meet demand: CSPs want to deploy services in the cloud to address the challenges of web-scale companies. In fact, 85% of companies surveyed say launching new digital services is critical to their business strategy.

What’s the catch? To do it, they’ll need to be agile. Although telecom is not an industry known for its agility, and is often thought of as the opposite, it has been transforming—like many industries—through adopting more agile approaches in business strategy. CSPs are striving to maximize resources and reduce inefficiencies, while at the same time sparking innovation. Nothing has been more of a stimulus for agility than the cloud, especially for the web-scale companies whose services pose the greatest threat to the telco industry.

A key requirement will be cloud-native applications that improve the agility of network functions. However, being cloud-native involves a lot more than just shoehorning code into a container previously ported from an appliance to a virtual machine.

Rather, at Oracle, cloud-native means faster development and deployment through DevOps and continuous integration and continuous delivery (CI/CD) pipeline. CSPs can only achieve an agile cloud with an environment enabling developers to innovate true cloud-native solutions if they implement an operational framework that embraces a key set of principles. Specifically, we at Oracle have identified the principles for successful cloud-native operations.

System Immutability

Everything, including software and configuration, is code. All changes are made through CI/CD, where they are deployed as immutable artifacts. No manual configurations or customizations are allowed. This makes it easier to implement the principle of least privilege as there is no need to run scripts in the production environment. Also, any changes not coming though the delivery pipeline can be considered malicious.

Automate Everything

All aspects of build, test, verification and deployment are automated. This includes activities such as backup, recovery, password/key rotation, etc. Fully automating the DevOps pipeline, including verification and testing, removes much of the potential for human error. This automation also allows changes to be applied to the environment with confidence and provides for rapid repair.

Disposability

All services are transient and treated as short-lived. Instead of focusing on never failing, services are designed to go up and down quickly without service interruption. Regular repaving (re-deployment) of the environment ensures failed or failing services are removed and new ones deployed.

Externalized Configuration

Configuration (including passwords, credentials, location of backing stores, etc.) is decoupled from the software image, and like software, can be treated as a build artifact in a controlled and versioned manner. Versioned configuration enables development and production parity as an artifact and can eliminate costly operational errors.

Logs as Event Streams and Constant Telemetry

Everything needed to debug or diagnose any functional, operational or security issue is found in logs, traces or metrics data. These are treated as a stream of time-ordered events and stored in a centralized collector outside the system, enabling better threat monitoring, forensics and diagnostics through event correlation or analyzing the aggregated and holistic view. Constant telemetry is fundamentally enabled through logs as event streams and, thus, the two should always be considered together.

Delegated Governance

Some shared aspects of the environment are centrally managed like networking, identity management or infrastructure, but in a true DevOps fashion, teams delivering a service are responsible for operating the service. This is allowed only with strict governance, enforced through checks in the continuous delivery pipelines, giving greater control over rate of change. Business agility is improved as applications have better visibility into the operations of their service with tighter feedback loops that ultimately improve quality.

Independent Lifecycle

Independently upgrading, scaling and deploying each microservice is paramount for supporting other cloud-native principles as well as minimizing the amount of change in the system at a given time. Furthermore, such decoupling makes other principles, such as repaving, easier and promotes easier isolation of issues.

As 5G introduces increased connectivity and speed, along with the need for more security and reliability, service providers will need the agility, flexibility and security of the cloud and cloud-native principles. These principles are enablers for better cloud operations. Together, they provide the ideal context in which to implement a broader view of the Cloud Native Computing Foundation (CNCF) landscape, giving developers a robust Platform-as-a-Service of backing services beyond just containers.

As 5G moves from a fabled future to a reality, the market for digital services that reap its benefits is likely to continue to grow. By leveraging these principles, along with DevOps and the CI/CD pipeline, organizations will be able to evolve and prepare for 5G’s promise for innovative business models and new revenue streams—be they digital services or beyond.

— Shirin Esfandiari

August 20, 2019
Doing DevOps in an Interconnected World
Doing DevOps would be simple if you had total control over the tools, code and infrastructure that your company used. But the fact is, you don’t. Instead, we live in a world where most CI/CD chains connect in some way to third-party resources or processes.

In other words, doing DevOps effectively means integrating with entities that are external to your organization. Here’s a look at what this entails in practice and why it’s so important.

The Interconnected World of DevOps

Conversations about DevOps often assume that all DevOps processes happen internally. In reality, many software delivery pipelines depend on third-party resources and services. Consider the following examples of ways companies could make their software delivery chains dependent in part upon third-party agendas:
- They integrate open source code into their codebases and therefore depend on an open source community to write some of their code.
- They maintain relationships with partners and need to be sure their release schedules and toolchains are compatible with their partners’ needs.
- They need to make sure the applications they deliver work on different operating systems, browsers and hardware devices that are controlled by third parties.
- They are subject to regulatory rules handed down by third parties and must ensure their delivery chains meet those regulatory requirements.
- Their applications depend on APIs that are designed and updated by third parties.
In these ways and more, the CI/CD pipelines organizations build to do DevOps need to be able to integrate with, and adapt to, decisions made by third parties. If they don’t, the organizations risk problems such as broken code, the loss of partners and regulatory compliance issues.

Succeeding at Interconnected DevOps

This is why it’s critical to design CI/CD pipelines and other DevOps processes in ways that make them mesh with third-party resources and services. In practice, this means:
- Adopting tools that are flexible and have broad coverage. You don’t want to tell a partner you can no longer work with their code because your CI server doesn’t support their new programming language.
- Use branching in the CI/CD process. Multiple branches give you more flexibility to support different target environments. They also help you react to sudden changes made by third parties without having to overhaul your entire CI/CD process.
- Make sure your collaboration and communication channels integrate with those of third parties on whom you depend. If you use open source code in your application, for example, add the open source project’s security announcements to your Slack channel so you’ll be notified right away when a vulnerability arises.
- Choose your integrations wisely. Just because you can integrate a third-party API into your application, for example, doesn’t mean you should. Too many dependencies put the efficiency of your CI/CD pipeline at risk.
No matter the details of your CI/CD chain, the most important thing is simply to recognize that we live in a very interconnected world and most organizations don’t have total control over their software delivery processes. That’s just a fact of modern DevOps. But with the right planning and strategies, you can ensure that third-party dependencies add value to your software delivery pipeline rather than creating unnecessary risk and overhead.

— Chris Tozzi
July 10, 2019
DevOps and Security: Be Ready to Shield Your Application
All of us have heard of continuous improvement/continuous delivery (CI/CD). There are many benefits to implementing CI/CD, as it helps seamless integration from end to end for development and deployment processes. CI/CD helps in rapid improvement, shorter release cycles and more, but it also helps with the challenge of handling security effectively at DevOps speed.

Tools for Securing DevOps

There are thousands of DevOps security tools available in the market that help us at HCL Technologies secure our software development at various stages of the CI/CD pipeline. There are new vulnerabilities rolling out daily, and so we must be alert and keep our applications safe and future-ready.

These are the DevOps security tools we use (most tools are open source with an option of enterprise edition):
1. OWASP SonarQube
2. OWASP ZAP
3. OWASP Dependency-Check
4. OWASP Glue
5. Anchore
6. Clair-CoreOS
7. FOSSA
8. Cyberark Conjur
Our DevOps Toolchain Diagram

Securing CI/CD

Let’s consider a job in which we check out the code from GitLab/GitHub with auto-commit (a Jenkins job will be triggered as soon as the developer makes any changes in the code). You can refer to this link for more information about Jenkins GitLab automation.

Now, in Jenkins, we can run a security test and present it in the form of reports and dashboard to meet our needs. To get started, we can use OWASP SonarQube for static analysis. OWASP SonarQube includes found bugs, PMD and several security plugins bundled out of the box. Next, for dynamic analysis and penetration testing, we can use OWASP ZAP, which presents reports in XML/HTML format that can be used to create Jira issues.

In the above scenario, we can also use ZAP through custom tools integration. In our case, we have installed ZAP on our local machine. We can also use OWASP ZAP installed in a remote machine.

Steps for Remote OWASP ZAP Connection

Windows Command: C:Program FilesOWASPZed Attack Proxy>zap.bat -config api.disablekey=true -config api.addrs.addr.name=.* -config api.addrs.addr.regex=true

Linux Command: Zed Attack Proxy>zap.sh -config api.disablekey=true -config api.addrs.addr.name=.* -config api.addrs.addr.regex=true

OWASP Configuration in Jenkins

The reports will be generated in .xml and .html format. Now, with the reports generated by ZAP, we developed a Java program to create Jira issues for every vulnerability detected and assign it to the concerned person.

Next, we have OWASP Dependency-Check analysis on our project. To configure for Jenkins, follow these steps:

First, we need the OWASP dependency-check plug-in (which I have already installed). This process may take several minutes, as it downloads from NVD server.

Here are the next steps:

Alternatively, we can use the OWASP Glue plug-in, which runs OWASP ZAP, OWASP Dependency-Check and other security automated tests.

We have also used FOSSA in our projects for license audits and vulnerability management for OS dependencies. We can use FOSSA as on-premises, but it is available only for use on Ubuntu machines.

At HCL, we are currently working on container security with Anchore and Clair-CoreOS. We have also integrated identity access management (IAM) to our CI/CD jobs. As for IAM, we are working on CyberArk Conjur and HashiCorp Vault.

CyberArk Conjur

The Conjur Jenkins credentials plug-in:

HashiCorp Vault

Conclusion

I have worked on various tools for security with DevOps. Below is a quick comparison of the tools that can be used for application security.

Now that you have an idea of which tools to use, embrace security instead of worrying about it. DevSecOps can help improve and increase the outcome and effectiveness of any organization.

— Debarghya Pandit
April 16, 2019