devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

  • Splunk Survey Surfaces Gains in Observability

    Splunk Survey Surfaces Gains in Observability

    A global survey of 1,250 observability practitioners, managers and other experts published today by Splunk found that sophisticated observability practitioners are able to cut downtime costs by 90%. That figure is based on an estimated cost of $23.8 million annually for comparative newcomers to $2.5 million. However, only 9% of respondents are advanced enough to be considered observability leaders compared to 59% that are still beginners.

    Those leaders reported they are seeing a 69% better mean-time-to-resolution (MTTR) for unplanned downtime or performance degradation thanks to investment in observability. The survey also noted those same organizations launched 60% more products or revenue streams from application development initiatives in the last year.

    Observability has always been a core DevOps tenet, but achieving and maintaining is a challenge. Most DevOps teams today aspire to be able to maintain some level of continuous monitoring. However, as it becomes easier and less costly to instrument applications, interest is rising in observability platforms that make it simpler to launch queries and investigate anomalies.

    Spiros Xanthos, senior vice president and general manager for observability at Splunk, said observability platforms are clearly driving down the cost of downtime. There is also a clear correlation between organizations that have embraced observability and the number of microservices-based cloud-native applications that have been deployed, he noted. In effect, increased complexity is driving organizations to turn to observability platforms to manage IT environments in the age of the cloud, said Xanthos.

    The survey found 75% of respondents have multiple cloud-native applications that run in multiple environments. Just over one-third (34%) of internally developed applications are based on microservices constructed using containers, with 28% of respondents reporting they exclusively run cloud-native applications on public cloud infrastructure.

    Overall, the survey also found that two-thirds of leaders reported their visibility into application performance is excellent compared to 44% of beginners, while 64% of leaders reported that visibility into their security posture is excellent compared to 42% for beginners.

    The survey also found 59% of leaders can push code to production on demand for most internally developed applications compared to 28% of beginners. A total of 41% of leaders said they can detect problems associated with internally developed applications within minutes compared to 20% for beginners.

    It’s still early days as far as adoption of observability platforms is concerned but, theoretically, IT should become easier to manage even as the number of platforms employed continues to expand. Advances in analytics and artificial intelligence (AI) are improving the signal-to-noise ratio so that not only are more issues being surfaced sooner but that more trusted recommendations are being made, said Xanthos.

    Observability is, of course, a journey. Most organizations are not even quite sure what type of questions they should formulate to get the most value from investment in an observability platform. Organizations should start small, with a finite project to gain an understanding of how to leverage observability more broadly as they gain more maturity, said Xanthos.

    Regardless of the approach to observability, the one thing that is certain is that traditional IT monitoring is no longer sufficient in an era where IT environments get more complex by the day.

  • Apple Allows 50% Fee Rise | @ElonMusk Fans: 70% Fake | Microsoft Salaries up by 100%?

    Apple Allows 50% Fee Rise | @ElonMusk Fans: 70% Fake | Microsoft Salaries up by 100%?

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Apple lets devs increase subscriptions “without consent,” Twitter has more than 19% spam accounts, and we dig into the claim that Microsoft is about to double salaries. (more…)

  • Increasing Use of SLOs to Enable Observability

    Increasing Use of SLOs to Enable Observability

    Observability is a growing discipline among most IT and operations departments. To release stable software faster, operators need continuous visibility into metrics like performance, uptime and availability. As a result, engineers are increasing their use of service-level objectives (SLOs) across the board—a recent study found that 82% of companies are increasing their use of SLOs.

    SLOs enable deep visibility into specific applications’ performance and are often used by site reliability engineers (SREs) to ensure quality and avoid service interruptions. Not only that, but various teams are correlating SLOs to aspects of the business to reduce cost and help direct decision-making. But while many environments have visibility, there are still gaps present.

    Nobl9 recently released a global survey of IT professionals and executives that tracked the state of service-level objectives in 2022. Below, we’ll look at the key findings from the report and consider what they mean for the state of SLOs and observability in general.

    State of Observability and SRE

    In general, site reliability engineering is still maturing across organizations. Though only 31% of companies have adopted SRE, it’s slated for much future growth, as 46% said they plan to embrace SRE in the future.

    These operators are now faced with many cloud-native observability tools, which are producing a sea of data in the form of metrics, logs and traces. A full 39% of companies use anywhere from six to 10 observability and monitoring tools and 35% use more than 11.

    With so many tools now deployed, who is using this observability and monitoring data? Namely, at 74% of companies, observability data support operational requirements. Operations teams are the ones most likely to use SLOs to monitor uptime, performance and overall efficiency. After operations teams, security teams also use this observability data (71%), which makes sense as SLOs can inform incident response. Other areas that follow are customer support, compliance and capacity planning.

    One interesting finding is that only 42% of companies use SLOs for service-level agreement (SLA) adherence. This indicates that SLOs are most often applied to internal optimizations and decision-making.

    Hybrid Environments Complicate Visibility

    Companies are mostly tracking SLOs to increase their visibility into networks (83%), databases (76%) and applications (75%). Other top areas include private cloud environments and legacy computing arrangements. But although observability is trending, organizations still lack complete visibility across the entire stack.

    Of note is that nearly half (46%) of respondents said their monitoring and observability tools don’t provide full visibility into all of their company’s IT assets. For example, only 45% of companies have visibility into their containers and just 35% have visibility into their microservices architecture.

    A lack of full-stack visibility may be complicated due to rising hybrid and multi-cloud conditions, as 78% say a hybrid cloud environment makes monitoring infrastructure more difficult.

    Benefits of Tracking SLOs

    The report defined SLO as “a performance and availability target set for a given system, application [or] service over time.” And the benefits are readily apparent for organizations following such targets—they can help increase performances, direct decision-making and help avoid outages.

    As a result, 70% of companies are currently using SLOs in some fashion. Here are some benefits of tracking SLOs:

    • Increase microservice performance: 87% said using SLOs for microservices architecture would improve service performance.
    • Enable full-stack observability: 58% said some of their company SLOs are mapped to business operations.
    • Improve business decision-making: 91% agreed using SLOs can help drive better business decisions.
    • Prevent service interruption: 67% said their company prevented business interruptions thanks to SLO thresholds alerts.
    • Reduce expenses: 90% indicated SLOs saved their company money.

    Final Thoughts

    More teams are tracking service-level objectives than ever before. And most companies (71%) that aren’t using SLOs now plan on adopting them soon. The data demonstrated that the observability market is an evolving area with room to grow. And, the same can be said about the SRE role.

    Yet, it’s good to point out that not all companies will adopt precisely the same roles or monitoring procedures. Therefore, there will likely continue to be variance in who is using SLOs and how they are applied. Regardless, the study suggested that investigating SLOs has the potential to benefit the software life cycle in many ways.

    The study, “Adoption of SLOs Grow to Increase Visibility and Drive Business Improvements” surveyed 309 participants around the globe in various sectors. The study was sponsored by Nobl9 and conducted by Dimensional Research. For more information, you can download the report here.

  • Why Over-Permissive CI/CD Pipelines are an Unnecessary Evil

    Why Over-Permissive CI/CD Pipelines are an Unnecessary Evil

    Balancing the speed of innovation with security is an age-old problem in software development. This is no different in modern DevOps. Development teams want a frictionless path to deploy new code but often struggle with access permissions that are linked to services. The CI/CD pipeline makes it easier for developers to quickly introduce new features and updates, in an agile way. As application changes move through the CI/CD pipeline, automation can allow changes to happen very quickly, without creating downtime or delays on the customer side.

    Permissions of CI/CD Servers

    When it comes to the permissions of CI/CD servers, DevOps often don’t want to deny access. After all, the wide access permissions are part of allowing the speed of innovation, and development teams are regularly—and often understandably—cautious about putting least-privilege or other access rules in place.

    But there are some security risks to consider. Any developer being granted admin permissions to deploy new code is open to vulnerabilities, weaknesses and misconfigurations. Cloud sprawl has created a lack of visibility into whether or not the infrastructure is properly set up and connected with logging tools. How does the development team know if things are getting properly patched?

    The balance is extremely sensitive. DevOps need the speed of change and the ability to innovate freely, but security teams can’t handle the growing risk. Something has to change that can allow security to shift left and get involved earlier in the CI/CD pipelines rather than getting in the way.

    Reducing the Risk Without Impacting Innovation

    Here are some suggestions for accelerating the speed of innovation while maintaining a healthy security posture:

    • Map threats and secure connections. Conduct a threat modeling exercise to map threats to the pipeline. Remember that every connection to the CI/CD pipeline is a potential point of compromise. Regularly patch and scan all services that connect to the pipeline and block the devices if they fail to meet security policy requirements.
    • Learn to love logs. When a security issue arises in production, developers, IT engineers and security engineers need to react quickly and efficiently to solve the problem. Logging is critical for enabling fast and coordinated responses. Waiting on manual sharing of sensitive data can slow down reaction times. But when every stakeholder can get the data they need from logs, access to information is no longer the weakest link in the security incident remediation process.
    • Tighten access control and enforce strong admin permissions. Be specific about who has access to which data. Clear and enforced separation of duties should ensure no one person or account can access more than necessary in the continuous software delivery pipeline. Improperly credentialed users or bad actors can’t establish control over the entire pipeline and mess with the code or processes. Moreover, they won’t easily be able to penetrate deeper into other corporate systems and data.
    • Leverage flags to manage risks. Developers often use feature flags (or feature toggles) to add new features to applications while mitigating the risk that those features will introduce performance or stability issues. Although most developers likely don’t think of feature flags as a security tool, they can be used for that purpose, too. New features are prime vectors for security vulnerabilities, especially if developers haven’t thoroughly tested them yet.

    Many security teams do not realize that just because their network is private, they’re still sharing pipelines with third parties, including anyone from maintenance to the team deploying your mobile app. By embracing these tools, developers can shift left, shoring up defenses early in the process, without adding a blocker to the pace of innovation coming from DevOps CI/CD pipelines.

  • Why Data Lineage Matters and Why it’s so Challenging

    Why Data Lineage Matters and Why it’s so Challenging

    Change introduces risk. It’s one of those foundational principles of software development that most of us learned very early in our careers. Nevertheless, it always seems to keep cropping up in spite of those repeated life lessons. Our inability to foresee the impact of changes, even small ones, often leads to negative outcomes.

    As the complexity of interconnected IT systems has increased, this problem has grown worse. It’s harder and harder to predict how even a small modification might impact systems upstream or downstream of the change. Data lineage accomplishes that, but it’s extraordinarily difficult to do data lineage well.

    Why Data Lineage Matters

    In the old world, data resided in silos with relatively few connection points between them. Today, data flows multi-directionally among multiple systems. Many of those systems share data with external organizations, which means that information subsequently flows to internal systems there, as well. Consequently, it’s easier than ever to overlook the impact of a single change on multiple other systems.

    Imagine, for example, that your company provides a mobile app that enables customers to quickly and easily request a new service call. In the past, your customers could specify which product variant they owned. Your design team has determined that since you already have that information in your CRM database, you no longer need to collect it from users of the mobile app.

    You’ve been tasked with simplifying the app, so you remove that input field and replace it with a query that pulls the customers’ product information from your CRM. Theoretically, the resulting work tickets should include all of the information that your field service personnel need to get the job done.

    Unfortunately, there’s a problem; some of the customers’ product records in your CRM system contain null values. Your mobile app is querying that information, but it’s delivering invalid results to your ticketing system. Your field service personnel no longer know how to prepare for their next service call.

    That might sound like a relatively simple problem of data quality—one that might have been foreseen if someone had been more thorough about vetting the CRM data. Even that might not have solved the problem, though, if null values in the CRM database were not introduced until later. If customer data from a newly acquired company is merged into your CRM system, for example, you might not have product information tied to those new records.

    Sometimes these issues are more difficult to predict, and they may not be apparent right away. Imagine what happens when your marketing team decides to redefine its customer segmentation criteria. A new data field is added to designate the customer category; then it’s populated with values for all customers and the old field is deprecated.

    Unbeknownst to the marketing department, though, the company’s top executives are still looking at the old data. They’re using dashboards that rely on data from the deprecated field. New customers are being added every day, but no one is entering any values in the deprecated field. As a result, the executive team is looking at distorted data. Suddenly, the C-suite is tracking very different KPIs than the rest of the organization. What’s worse, no one even notices that there’s an issue until weeks or months after the fact.

    Next, imagine a similar scenario, but instead of populating an executive dashboard, the source system is feeding data to the AI algorithms that drive customer recommendations on your e-commerce site. When AI is trained on flawed data—or if the data changes materially in some way—it can dramatically impact the effectiveness of that technology. In this case, it distorts your customer recommendation engine.

    Tackling the Challenge With Data Lineage

    Data lineage tools provide a systematic means of understanding the impact of changes by presenting a complete roadmap of potential implications, both upstream and downstream.

    This amounts to risk reduction and cost control. If we can predict how changes might impact other systems, we can avoid creating problems like the ones we described earlier. That leads to less wasted effort, faster time-to-market and lower costs. We’re all familiar with the principle that says it’s faster and cheaper to solve a problem when you catch it early in the development process; data lineage provides a critical missing link, potentially ferreting out problems before they even reach QA.

    There are benefits to data governance and regulatory compliance as well. When the European Union passed the General Data Protection Regulation (GDPR) back in 2016, it set off a chain of new requirements for systems that house personally identifiable information (PII). If a customer asks you to delete their data, you’re legally obligated to comply with that request. But what happens upstream and downstream of that change? Will it break anything? Or will some of the customer’s PII remain elsewhere in your systems, rendering you non-compliant with the law?

    Data lineage addresses these questions by automating the process of detecting upstream and downstream effects. It provides clear auditability that can help determine where changes originated and how the data arrived at its current state.

    Column-level data lineage is especially difficult because it requires that SQL queries be parsed to discern exactly what changed and how. This means tapping into database logs, but because there are so many database vendors—each with their own unique SQL dialect—parsing that information for various databases can be particularly challenging. As each of those SQL dialects evolves over time, data lineage tools must constantly be refined to accommodate those changes.

    An example column-level lineage view that shows all the tables and columns that are involved in computing the sales table used for reporting in an imaginary e-commerce business selling craft beers.

    As interconnected systems extend their reach even further and the flow of information accelerates, column-level data lineage will become more important. Today, it’s an extraordinarily useful tool that saves time, effort and money. Soon, data lineage will be indispensable.

  • Nobl9 Shares SLO-as-Code Methodology

    Nobl9 Shares SLO-as-Code Methodology

    Nobl9 has released version 1.0 of an open specification for defining service level objectives, dubbed OpenSLO, and, in addition, has defined a repeatable SLO methodology.

    Kit Merker, Nobl9 COO, said the Service Level Objective Development Lifecycle (SLODC) methodology represents an effort to create a set of best practices for achieving and maintaining SLOs that are implemented as code within an application environment.

    A survey of more than 300 IT managers and executives conducted by Dimensional Research on behalf of Nobl9 found only 29% of respondents had no plans to implement SLOs. A full 94% of respondents that have or plan to implement SLOs intend to map them directly to business operations, with 91% reporting they expect that effort to improve decision-making. More than 80% also said their organizations are planning to increase the use of SLOs, with 87% indicating SLOs should improve overall microservices performance.

    The challenge is that most organizations have limited visibility into their IT environments. The survey, for example, found less than half of respondents (46%) had visibility into all their IT environments. Only 45% and 35% claim to have visibility into containers and microservices, respectively. More than three-quarters (78%) said hybrid clouds make observability more difficult.

    Ironically, 45% of companies reported they already employed 11 or more observability and monitoring tools. On the plus side, 31% have hired site reliability engineers (SREs), while nearly half (46%) planned to create that role.

    Nobl9 is trying to spur greater adoption of SLOs by making available an open source SLO specification that defines a common interface for constructing SLOs across a Git-based workflow. Since OpenSLO was initially launched, more capabilities have been added including a DataSource object that makes it easier to reuse connection details and which makes creating SLOs less verbose and three alerts have been defined: AlertCondition, AlertPolicy and AlertNotifcationTarget.

    Nobl9 is also releasing an OpenSLO-to-Nobl9 converter to transition OpenSLO YAML files into Nobl9 YAML when required.

    SLOs, of course, are not a new idea. They have been employed as a metric to track the performance of IT services for decades. However, as more microservices-based applications are built and deployed, it’s becoming more challenging to maintain SLOs across applications that have many more dependencies than legacy monolithic applications. Ultimately, each IT team needs to provide some sort of objective benchmark that assesses their overall effectiveness at delivering application services. SLO-as-code is intended to make it simpler to gather the metrics that confirm whether service levels are being achieved. Those SLOs are then tracked and integrated across everything from financial operations to supply chains, noted Mercer.

    It’s not clear to whether reliance on SLOs has waned over the years or if application environments simply became too complex to track meaningful metrics. However, as applications are increasingly viewed as services, it’s now only a matter of time before SLOs become more widely adopted across a modern application environment. The challenge, of course, is that defining an SLO is a lot easier than maintaining it—especially within today’s highly dynamic application environments where services tend to come, change and eventually go unexpectedly.

  • Progress Expands Scope of Compliance-as-Code Capabilities

    Progress Expands Scope of Compliance-as-Code Capabilities

    Progress this week extended its DevSecOps portfolio—built atop the Chef automation framework it acquired in 2020—to now include the ability to programmatically address compliance mandates.

    At the same time, Progress has updated the Progress Chef InSpec framework for automating the discovery of compliance issues to add support for SAP ASE, IBM DB2, Mongo, Cassandra, Oracle, MS SQL platforms along with RHEL, CentOS, Ubuntu and macOS Monterey.

    Progress has also updated the Progress Chef Enterprise Automation Stack (EAS) to add the ability to combine infrastructure configuration processing with compliance audits within a single consolidated policy definition along with a high availability capability that makes certain the platform is always accessible.

    Prashanth Nanjundappa, vice president of product management for Progress, said Progress Chef Cloud Security enables DevOps teams to take advantage of a single policy-as-code platform to consistently implement compliance policies alongside security controls across both multiple public clouds and on-premises IT environments.

    Progress Chef Cloud Security leverages certified industry benchmarks, such as the Center for Internet Security (CIS) benchmark, to continually scan and automatically detect security configuration issues at multiple stages within a DevOps pipeline. Progress has now expanded CIS benchmark profile coverage for Amazon Web Services (AWS), Microsoft Azure and Google Cloud Platform (GCP) to include more than 100 customizable pre-built service and resource templates. DevOps teams can take advantage of automated creation of code, test and documentation artifacts for those resources. Finally, visibility into the status and expected completion time for compliance scans has been added along with advanced Shell command support for control and testing without file output.

    In general, the ability to address compliance requirements within a DevSecOps workflow has become crucial because the number of misconfigurations that occur in the cloud has exploded. Developers that often have little to no security or compliance expertise are programmatically provisioning cloud infrastructure in a way that, for example, leaves ports wide open. As IT organizations embrace more cloud computing platforms, that security issue only compounds, noted Nanjundappa.

    More challenging still, cloud computing environments are becoming more complex with the rise of cloud-native applications based on platforms such as Kubernetes. The odds a developer is going to make a mistake only increase with each microservice deployed.

    Progress is betting that, as cloud computing evolves, responsibility for security and compliance will increasingly shift left toward DevOps teams; they’ll be responsible for implementing polices defined by cybersecurity and compliance specialists. The goal is to provide a set of guardrails that make configuration mistakes less likely because policies are being tested within the context of a DevSecOps workflow, said Nanjundappa.

    It’s only a matter of time before software supply chain security reviews that are occurring in the wake of a series of high-profile breaches force the DevSecOps issue within more organizations. The goal, of course, should be to eliminate the need to identify DevSecOps as separate from a DevOps workflow. Regardless of the approach, however, the one good thing about all this attention is that application environments will inevitably become a lot more secure.

  • How Waterfall Methodologies Stifle Enterprise Agility

    How Waterfall Methodologies Stifle Enterprise Agility

    In this age of sweeping digital transformation and accelerated change, organizations are recognizing that depending solely upon the drawn-out process of waterfall methodology is no longer sustainable if they are to remain competitive and relevant.

    Enterprise infrastructure must incrementally embrace evolution from monolithic architectures in response to the escalating pace of technological innovation, ever-changing consumer needs and demands and the failure of a long-running single project focus.

    EA Must Continuously Adapt 

    Traditional enterprise architecture (EA) frameworks struggle to manage the impact of disruptive technologies and systems that are agile and flexible. An architecture design approach that embraces continual evaluation and iteration is better equipped to navigate change and capitalize on emerging trends and innovations. Adaptability, in this context, may be defined as the fluid ability to modify, restructure and recreate the architecture in relation to changes within and external to the enterprise. It involves the early identification and rapid response to new product concepts, important trends, events and consumer demands; developing a “from the outside in” perspective.

    This adaptability, or ‘evolvability,’ quotient enables the flexibility and agility to deliver services and products reliably in this new development norm. Enterprise leaders are quickly acknowledging the need to inject a more open and agile format into their enterprise’s processes, at least in some measure, to succeed in today’s unpredictable environment.

    The Waterfall Model Constrains Enterprise Agility

    Traditional organizational architecture can impose limitations on an enterprise’s ability to successfully reach its digital transformation goals. The up-front model, with a focus on one long-range project, can slow productivity and choke creativity. While planning is needed as agility scales, the detailed technology life cycles with large timeline projections are no longer effective or profitable in meeting the business mandates that drive enterprises forward. Enterprise leaders are increasingly abandoning the five-year architectural plan for one that is designed to evolve with the ever-changing software development environment.

    Enterprise architects must now develop and promote adaptive methods that support agility in order to appropriate the value of new technologies like AI, machine learning, big data, IoT and intuitive tools that enable advanced analytics and enterprise-wide collaboration. A less intentional architecture, decomposed into smaller units, can be managed by autonomous cross-functional teams that are accountable to peers and managers with shared strategic objectives, bringing all fields into a coherent whole.

    A continuous architecture design with no end state, guided by iterative changes, will fluidly address the changing needs of the enterprise. A shift from a project-oriented to a product-oriented management mindset leads to faster learning cycles, shorter time-to-market, higher quality and increased revenues.

    The Attributes of a Flexible and Adaptive EA

    An open architecture designed to evolve in an ongoing manner and enable agility will include many beneficial characteristics and principles. Flexible and adaptive EA:

    • Provides product-centricity that connects software development to business goals;
    • Explores multiple solutions simultaneously with shorter analysis, testing, experimentation cycles and progression to eliminate weak solutions;
    • Empowers autonomous, cross-functional teams with clear accountability roles and alignment to a shared mission statement and effective information sharing;
    • Enables flexible and efficient resource allocation driven by changing demands or activity levels;
    • Strengthens an evolving roadmap that shows vision and targets, open to the whole enterprise, to allow input and questions for collaboration, cohesion and transparency;
    • Facilitates incremental changes to software architecture in a continuous manner;
    • Provides a continuous integration and delivery model with automatic testing for ‘good enough’ MVPs to keep the business moving forward.

    An adaptive EA allows for timely decision-making as agility scales, ensuring that the right products are delivered at the right time in alignment with business initiatives. When design accountability is distributed across the enterprise, consensus on complex decisions involving diverse stakeholders is more easily attained.

    The business rewards of transitioning to an adaptive architecture should be an easy sell by examining and tracking the monetary, time and customer success gains. The evolution toward adaptive enterprise architecture can reap numerable benefits. In addition to fostering an enterprise culture of continuous improvement, it accelerates and improves development cycles, journeys and outcomes for stakeholders and customers.

    Adaptive EA addresses the inherent uncertainties and complexities associated with dynamic business changes. It can deliver the rapid incremental changes, capabilities and benefits that a waterfall model inhibits. The agile and open methodology, empowered to continually evaluate and modify, provides visibility into project requirements and feasibility to reduce risks typically experienced with a traditional design and waterfall methodology.

    Making the Transition to an Agile EA

    Transforming enterprise architecture to a more adaptive and agile model will require planning and investment. A necessary first step is to assess what the enterprise goals and objectives are in making the transition. To what degree is the organization ready and what needs to be in place in order to move forward?

    Investing in an enterprise-wide training program of agile technologies, principles and methodologies will develop a cohesive and cooperative mindset that overcomes resistance to change and optimizes transition success.

    Of utmost importance is the development of a culture of communication, from the C-suite down across all stakeholders. A shared vision must be communicated and encouraged within a collaborative framework to cultivate enterprise-wide buy-in and acceptance of changes from waterfall methodologies to current roles and practices. In the migration to agile, there can be many obstacles, but by adopting an incremental approach with frequent reassessments an enterprise can ease the transition.

  • How to Secure CI/CD Pipelines With DevSecOps

    How to Secure CI/CD Pipelines With DevSecOps

    Many companies are adopting a DevOps approach in their workflows as IT moves toward a more automated and cloud-native world—but for some industries, this migration isn’t easy. Many of these companies—in finance, health care, government—are obligated to meet compliance requirements. For these organizations, DevSecOps adds necessary security focus to the DevOps methodology.

    Compliance means adhering to certain standards and controls to ensure that applications are secure. But compliance only goes so far. How many applications contain third-party dependencies inside? How expensive would it be for a company in the financial industry, for example, to fail to meet compliance standards?

    Initially, containerizing an application while complying with requirements of whichever framework or standards apply to your organization could seem close to impossible. Within the cloud-native landscape, however, technologies are also rapidly evolving to enable healthy migration and ensure that applications can meet compliance standards without being rewritten from scratch.

    First, let’s look at the standards organizations must adhere to in terms of compliance.

    CIS Benchmarks for Kubernetes and Docker

    The Center for Internet Security (CIS) benchmarks offer guidelines for the correct use of containers in applications. The CIS benchmarks specifically discuss Docker and Kubernetes.

    NIST SP 800-190

    The U.S. government’s National Institute of Standards and Technology (NIST) defines containers as a form of virtualized operating systems within its application container security guide.

    PCI

    The payment card industry (PCI) standards are a framework that helps protect data in the e-commerce and retail areas.

    HIPAA

    The Health Insurance Portability and Accountability Act (HIPAA) sets the compliance standard for patient data protection in health care.

    Securing Infrastructure via CI/CD Pipelines

    All these standards guide organizations to keep their infrastructure secure and compliant with their specific industry best practices, but security doesn’t end (or even start) there.

    Secure systems start from the design stage and from the code itself; you should aim to have security controls and enforcement built in as early as possible in the development life cycle. The practice of moving security checks earlier in the development pipeline (and, thus, nearer the source code) is known as shift left.

    You likely already have CI/CD pipelines in place for your application(s), and this is a great place to shift left your security checks. There are some great tools in the CNCF portfolio to help you achieve this.

    Enter Containers

    In the cloud-native world, containers are first-class citizens. From the DevSecOps point of view, containers represent a new attack surface to consider, starting with how the container image is defined and what ends up inside the container (the Dockerfile) to how the container evolves over time in response to new vulnerabilities that might be found.

    The first thing we can do is to convert our written policies to something that we can automate (such as code); for example, using a tool like OPA/conftest and a Dockerfile linter like hadolint, we can validate that the Dockerfile is compliant with our security policies.

    For instance, we could have a policy in place to limit the base images used to build new containers (limiting the FROM directive in the Dockerfile, for example).

    The second thing to consider is that, depending on how it has been built, a container image can have not only our application but also additional software included, like dependencies and operating system binaries.

    This additional software could have new vulnerabilities discovered in the future, so we should monitor the images that we have already built and pushed to our registry by periodically scanning them to find if they are open to any vulnerabilities that weren’t known at the time of building the container image.

    Just as before, we could sign our application artifacts. Now, our deliverable artifact is a container image, so it is preferable to be able to sign container images. In this way, we can trust that the container image we run comes from us and that it has been through our security process. There are projects like Notary, Grafeas and sigstore/cosign that can help with the container signature process.

    Running Containers in Kubernetes

    If we are running containers, the next logical step is to consider how we run them. Kubernetes use is on the rise as a solution to orchestrating containers across several machines. Introducing Kubernetes into the equation requires considering yet another attack surface in our DevSecOps workflow.

    Manifests

    Alongside our application’s source code, we will have the Kubernetes manifests that define how to deploy our application into a Kubernetes cluster.

    We can take the same approach as the container definition: Define some policies-as-code to enforce early in the process so our manifests are aligned with our security policies (not running untagged images, running images only from our trusted sources, ingress must have TLS, not using node port services, for example) and lint the YAML files to fail early if there are syntax problems, for example.

    We could also enforce most of the same policies and some additional ones at runtime when the manifests get applied to the cluster via a Kubernetes Admission Controller.

    Some tools that can help us with these enforcements are OPA/conftest, kubeval or kubelinter for linting and validation and, for the admission controls, we can use OPA/Gatekeeper or Kyverno, in addition to the API server admission plugins.

    Secrets

    Secrets require special attention since they contain sensitive information.

    If you’re using a GitOps approach, application secrets are often stored in repositories. Here are some resources that discuss sealed secrets in GitOps and Kubernetes secrets. 

    RBAC

    Do not use default service accounts: Define explicit service accounts associated with their permissions so you know exactly who can do what.

    Avoid the use of wildcards in roles and cluster roles; with RBAC you can define fine-grained permissions for users and for service accounts, as well.

    Know Your Dependencies

    Another attack surface that we need to consider is the application’s dependencies. It is important to first know what they are and if they have any known vulnerabilities. To help us with that, we can create a software bill of materials for the application and run an analysis on it.

    Finally, just like we periodically check for the images stored in the registry, we can continuously analyze what’s running in our Kubernetes cluster. This will ensure that there is no suspicious activity like abnormal network requests, images with critical CVEs running, etc.

    Conclusion

    By now, you should have a better idea of what it takes to secure your pipeline. Keep in mind, however, that this is mostly about process and cultural change.

    First of all, you need to know your attack surface—that is not only a technological choice, it’s also about what you think is important.

    These concepts are hard to apply from the beginning, so a better approach is to introduce changes, get used to them and think about how to improve the existing process step by step.

    Join us for KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain (and virtual) from May 16-20—the first in-person European event in three years!

  • Red Hat Adds Managed Ansible Service on Azure

    Red Hat Adds Managed Ansible Service on Azure

    Red Hat today announced general availability of a managed instance of the Red Hat Ansible Automation Platform on the Microsoft Azure cloud.

    Announced at the Red Hat Summit 2022 event, this offering makes it possible to deploy the control plane for Ansible on the Microsoft Azure cloud as an alternative to existing support for Amazon Web Services (AWS) or an on-premises IT environment.

    In addition, in version 2.2 of the Ansible Automation platform, Red Hat is previewing support for content signing technology that automatically validates content to improve overall security.

    Thomas Anderson, vice president of Ansible for Red Hat, said this managed service enables IT teams to take advantage of integrated Microsoft billing in addition to the latest features and content provided for the platform and other capabilities that have been added to optimize Microsoft Azure environments.

    The open source Ansible platform has been gaining traction as an automation framework that enables IT teams to declaratively automate tasks. It requires less programming expertise than rival automation frameworks, which Anderson said is the primary reason so many third-party vendors have adopted Ansible to automate a range of IT tasks.

    The challenge many organizations encounter today is managing multiple islands of automation across multiple clouds and on-premises IT environments. Ansible presents IT teams with the ability to begin unifying all those islands under a single framework, said Anderson.

    It’s not clear to what degree DevOps teams that have embraced rival automation frameworks might be willing to replace them with Ansible. However, as more IT teams embrace automation to make up for shortages of IT personnel, it’s apparent that Ansible is being used across a wide range of IT disciplines spanning everything from DevOps to network management.

    Regardless of the approach, the need to automate as many tasks as possible is becoming critical as IT environments grow increasingly more complex. Many of the scripts that IT teams created on their own simply don’t scale. More challenging still, they are not often documented, so every time the IT professional that created that script leaves the organization it becomes difficult to maintain that scripting code.

    In the longer term, of course, automation frameworks will be increasingly infused with machine learning algorithms capable of learning the unique nuances of individual IT environments. It’s not likely those algorithms will replace the need for IT professionals any time soon. However, rather than resisting automation, more IT professionals have concluded that the tasks required are simply too challenging to manually perform. Many will conclude they simply prefer to work for organizations that have a consistent approach to IT automation that makes it feasible for them to manage a wide range of IT tasks.

    One way or another, IT automation is becoming a standard capability that now goes well beyond the scope of a few isolated DevOps workflows. The issue is finding a way for IT professionals with varying skill levels to collaborate across a common framework that is both easily accessible and highly extensible.