devopsy.kinsta.cloud

Category: Blogs

  • Cloud Drift Detection With Policy-as-Code

    Cloud Drift Detection With Policy-as-Code

    Cloud configurations can change and change often. Introducing new technologies, releasing new features and supporting new business requirements entail a constant flow of configuration changes in web application development.

    However, drift occurs regardless of how well-designed your IaC implementation is. The term “drift” is used to denote a state in which the actual state of your infrastructure deviates from the configuration.

    This article examines cloud drift detection, why it occurs and how to remediate it.

    Understanding the Problem

    Cloud configurations are prone to change and can change frequently. Businesses often use infrastructure-as-code (IaC) to manage cloud provisioning changes. IaC makes it easier and more reliable for organizations to manage and facilitate changes to the cloud deployment process.

    However, inconsistencies in your IaC deployment process can lead to uncertainty about how and where your resources are provisioned, controlled and protected, resulting in lower productivity. No matter how robust your IaC implementation, drift can creep in.

    Changes to infrastructure that are made independently of the code responsible for provisioning can lead to considerable drift, and if not adequately monitored, can lead to significant security concerns.

    What is Drift?

    In the context of infrastructure and application management, drift refers to the system state in which the actual configuration and state of the system deviate from the intended or expected configuration and state.

    Drift can be caused by software updates, manual changes and incorrect configurations. Drift management is an essential aspect of infrastructure and application management. By detecting and correcting drift, organizations can ensure that their systems remain stable, secure and compliant with their policies and industry standards.

    Configuration Drift: Definition, Causes and Examples

    Configuration drift can creep in over time despite consistently building and configuring your servers. It refers to the state of the system when configuration changes are not in sync with the value previously set.

    For example, configuration drift can occur when you are yet to document the changes made to the production environment. This results in the production and staging environments getting out of sync.

    Some of the other causes of configuration drift are configuration changes because of applying patches and updating network equipment, the addition of new resources to the network and lack of clarity about the desired state of the system.

    You can leverage configuration drift management tools such as Netwrix and Aqua to detect configuration drift.

    Infrastructure Drift: Definition, Causes and Examples

    Infrastructure drift refers to a phenomenon in which the actual state of the infrastructure differs from its desired state because the defined configuration, settings or properties of the infrastructure and the actual state of the provisioned resources differ.

    Infrastructure drift may result from manual changes, updates outside of configuration management processes, conflicting IaC code, changes in network configurations, changes due to security settings and configuration differences during system upgrades and maintenance.

    To detect infrastructure drift, you can take advantage of infrastructure drift detection tools such as Terraform, CloudQuery and driftctl.

    Infrastructure Drift and Configuration Drift: How do they compare?

    Both configuration and infrastructure drift can lead to security vulnerabilities, compliance issues and operational challenges. Although infrastructure drift and configuration drift are related, they aren’t the same.

    Infrastructure drift refers to inconsistencies or discrepancies between the intended or desired state of the infrastructure and the actual state of the resources provided. Configuration drift refers to inconsistencies or discrepancies between the current configuration of a component or system and the intended or desired configuration.

    Infrastructure drift encompasses everything from physical components to virtual components to networks, storage and computing resources. Configuration drift, on the other hand, focuses on individual components or systems within the infrastructure and their configuration settings.

    What is Drift Detection?

    Drift detection in the context of cloud infrastructures typically compares the actual state of resources deployed in the cloud environment with the defined state described in IaC templates.

    Typically, this is automated by tools, scripts or services that analyze and compare configurations, resource states or event logs. These tools are adept at issuing notifications or alerts when discrepancies are detected.

    By detecting discrepancies, organizations can identify and resolve inconsistencies, security vulnerabilities and compliance violations. Consequently, the risks related to security breaches are reduced and remediation actions become easier to implement.

    Why Policy-as-Code for Cloud Drift Detection?

    Policy-as-code provides several benefits when used for cloud drift detection:

    • Early detection of drift: Policy-as-code helps detect drift early detection by comparing the current state of the cloud environment with the desired state defined in the policies. This helps identify problems and solve issues before they escalate to a catastrophic level.
    • Automated enforcement: Policy-as-code enables automated enforcement of policies, ensuring that the cloud environment complies with the defined rules and requirements. This helps reduce the potential risks associated with human errors.
    • Faster remediation: Policy-as-code enables faster remediation by automatically detecting drift and taking the necessary remediation actions such as rolling back changes, updating configurations or notifying relevant teams. This facilitates faster response times and reduces the risks associated with security breaches and downtime.
    • Enforce best practices, policies and guidelines: With policy-as-code, best practices, policies and security guidelines can be implemented automatically, reducing security and compliance risks.

    Steps for Drift Detection Using Policy-as-Code

    Here are the steps that you need to follow to detect and fix drift in your infrastructure:

    1. Define your policies: You should first define the policies that should be used to describe the desired state or the baseline of your cloud environment and cover security and compliance aspects of your cloud environment.
    2. Codify and deploy policies: Then these policies should be codified and stored in the version control system. The next step should be to deploy the policies to your cloud environment using a policy engine.
    3. Identify configuration changes: Once your policies have been deployed, the next step should be to identify any new resources that may introduce risks and compare them to the baseline you’ve defined.
    4. Evaluate and remediate: You should evaluate the state of the environment against the defined policies to detect any drift. As soon as any drift is detected, you should take corrective actions to remediate the issues by rolling back changes, updating configurations or notifying the relevant team members.
    5. Review policies: To ensure that the policies are up-to-date and relevant, it is imperative that you review them often and make changes to policies as needed.
    6. Update and deploy: The last step should be to update the code files and then deploy the solution yet again.

    Build a Successful Drift Detection Strategy

    Maintaining your infrastructure at the desired state, ensuring compliance and mitigating risks associated with configuration drifts requires a successful drift detection strategy. Here are key strategies you can adopt to build a successful drift detection strategy.

    Understand When Drift Becomes a Risk

    Drift can profoundly impact a system’s stability, security and compliance. When your server configuration deviates from its intended configuration, it might stop working or become vulnerable to security threats.

    Most importantly, this might be a significant threat to security if you don’t have proper monitoring in place. To prevent infrastructure drift and update your security tools, it is important to evaluate and communicate your adoption of IaC.

    Detecting Drift: Detecting drift Between IaC and the Cloud

    To mitigate the risks associated with drift, configurations that have been changed in real-time must be identified and reset. You can achieve this programmatically and continuously by comparing configuration changes between IaaS and IaC. Programmatically identifying the configuration changes, i.e., detecting the changes to the configuration using code, is the best strategy to reduce drift-related risks.

    From Drift Detection to Drift Remediation: Responding to drift

    By detecting drift, you’ve won half the battle–you should now devise strategies to remediate it. By keeping an eye on configuration drift and responding to it promptly, you ensure the consistency and reliability of your infrastructure, minimize security risks and keep your system in a consistent state.

    Figure 1: Detect and Remediate Drift in the Cloud

    It’s important to have effective configuration drift response procedures in place to ensure the stability and security of your environment and also keep it effective for change management and control. You can respond to drift in the following ways:

    • Develop mechanisms to detect and report configuration drift
    • Understand the underlying causes of drift
    • Documenting the drift, including affected resources, configuration deviations and impacts
    • Resolve the configuration drift and return the system to the desired state
    • Following change management processes and obtaining approvals prior to implementing changes
    • Monitor the system to ensure drift has been resolved and the configuration remains stable

    Key Strategies for Detecting Drifts Between IaC and the Cloud

    There are a few strategies that can be adopted for detecting drift.

    Detect Configuration Changes That may Introduce Risks

    To identify configuration changes that can introduce risks, you should monitor and analyze changes proactively. Here are some of the key strategies to do so:

    • Create a baseline configuration for your system.
    • Review the system logs regularly for any suspicious or unexpected configuration changes.
    • To capture detailed information about configuration changes, comprehensive logging is helpful.
    • Implement a change management process for managing configuration changes.
    • Take advantage of automated configuration management tools such as Chef, Puppet, Ansible, etc.
    • Leverage threat intelligence to ensure that you are always up to speed on the most recent vulnerabilities, security threats and attack trends.
    • Conduct training to increase awareness about configuration changes’ potential consequences and risks.

    Establish Baselines and Safely Auto-Remediate Drift

    Establishing baselines and safely auto-remediating drift involves setting a reference point for the desired state of your infrastructure and automatically adjusting any detected drift back into compliance with the baseline.

    You can maintain the desired state of your infrastructure, reduce manual effort and quickly resolve configuration drift by establishing baselines and implementing security processes for automatic remediation.

    This minimizes the impact of drift on your systems while improving your environment’s security, compliance and stability.

    Track all Configuration Changes in Your Cloud Environment

    Tracking all configuration changes to your cloud environment is critical to maintaining visibility, accountability and security. It can help you improve visibility, detect unauthorized or unintended changes and maintain compliance and accountability.

    This way, you can proactively manage your cloud infrastructure, identify potential security risks and ensure its integrity and stability.

    Below are the key strategies for tracking cloud configuration changes at a glance:

    • Ensure detailed logging for all cloud components, including infrastructure, applications and security
    • Track API calls and activities within your cloud environment with cloud monitoring and auditing
    • Set up alerts and notifications for critical deviations from established baselines
    • Implement configuration changes through formal change management processes
    • Manage cloud configuration with tools such as AWS Config and Azure Automation

    Managing Drift in Production

    To effectively manage drift in production, it is crucial to consistently observe the configuration and state of production systems, keeping them in line with the policies that outline the desired state.

    Here are a few guidelines to help you reduce production drift:

    • Specify the required state: Define the intended state of the production environment as a set of policies using declarative compliance tools.
    • Monitor continuously: To determine any deviations, use a monitoring tool to monitor the production environment or compare the system’s present state to the desired state to identify any deviations.
    • Identify the root cause: If drift is detected, it is crucial to identify its root cause. You can do this by examining the logs, analyzing system performance or observing the configuration alterations over time.
    • Rectify the drift: After determining the issue’s root, you should address it to align its compliance with the intended state. To do this, you may need to examine logs, examine configuration changes, and analyze system performance.
    • Automate the remediation process: To ensure that drift is automatically and quickly rectified, you should automate the remediation process. To achieve this, you can leverage a declarative compliance tool that can help you automatically implement the changes to bring the system back into conformity with the intended state.
    • Review and update policies: You should review and revise your policies to ensure they are still relevant and useful and stay updated. To achieve this, you may need to adjust the policies to reflect new compliance requirements or shift your business requirements.

    Summary

    By identifying drift in cloud environments, organizations can use policy-as-code to ensure that their cloud infrastructure is safe, compliant and secure. In this way, they can ensure compliance with industry best practices, regulatory standards and internal policies. This reduces the risks associated with configuration deviations and enables enterprises to maintain a secure and reliable cloud environment.

  • Checkmarx Brings Generative AI to SAST and IaC Security Tools

    Checkmarx Brings Generative AI to SAST and IaC Security Tools

    Under an early access program, Checkmarx today made available query builder and guided automation tools that take advantage of OpenAI’s generative artificial intelligence (AI) technologies to make it simpler for developers to resolve application security issues.

    AI Guided Remediation surfaces actionable remediation recommendations for vulnerability issues such as misconfigurations directly from within integrated development environments (IDEs).

    Meanwhile, AI Query Builder makes it possible to use natural language text to create a query for both the Checkmarx static application security testing (SAST) and the infrastructure-as-code (IaC) security tool that creates rules for scanning code. Those rules can be easily fine-tuned or modified and queries for other use cases can easily be added.

    In addition to reducing the amount of time it takes to create a query by 65%, that approach also dramatically reduces the number of false positive alerts that arise based on rules created by a security administrator.

    Checkmarx CEO Sandeep Johri said these additions to the Checkmarx One Application Security Platform are aimed at improving the application security experience for developers. Most developers don’t want to be inundated by alerts that lack any real context, nor do they want to be bothered with remediation details.

    It’s not likely developers will be interested in how AI could help them write more secure code from the start, but the faster a reliable fix is surfaced, the sooner developers can return to writing code, noted Johri.

    In the longer term, Checkmarx will add support for multiple large language models (LLMs) beyond those provided by OpenAI to provide other AI capabilities that are based on more domain security knowledge, said Johri.

    However, despite these advances, vulnerability remediation will not become fully automated using AI any time soon, he added. Instead, it will become much simpler to identify code that has either inadvertently or deliberately introduced a vulnerability, said Johri.

    In fact, generative AI tools such as GitHub Copilot can themselves introduce vulnerabilities into code. As a general-purpose AI platform, the recommendations surfaced are based on a mix of instances of clean and flawed code, Johri noted. There will also be instances where cybercriminals attempt to subvert an LLM that creates code by injecting snippets loaded with malware into the samples used to train a generative AI model.

    On the plus side, however, generative AI tools should narrow the divide that currently exists between application developers and cybersecurity teams as more issues are discovered and remediated before applications are deployed in a production environment. The challenge has aways been surfacing application security issues at the time developers are writing code rather than sending them a list of vulnerabilities to address weeks (sometimes even months) after a developer has moved on to another project.

    Naturally, there is a lot of trepidation when it comes to all things generative AI; one area of certainty is that the benefits far outweigh the risks—especially when it comes to developing secure-by-default applications.

  • Linux Foundation Europe to Host RISE Open Source Project

    Linux Foundation Europe to Host RISE Open Source Project

    The Linux Foundation Europe, in collaboration with RISC-V International, announced today it will host a RISC-V Software Ecosystem (RISE) Project. Through the project, multiple organizations will contribute engineering and financial resources to harden critical open source software for use in commercial applications and enterprise IT environments.

    Founding members of the project include Andes, Google, Intel, Imagination Technologies, MediaTek, Nvidia, Qualcomm Technologies, Red Hat, Rivos, Samsung, SiFive, T-Head and Ventana.

    RISC-V International is defining a set of application processors that includes software development tools, virtualization support, language runtimes, Linux distribution integration and system firmware that is intended to be used primarily in edge computing environments.

    Amber Huffman, chair of the RISE Project, said at a minimum, each of the founding members contributing €80,000 and additional engineering resources. The RISE Project deliverables will be prioritized by the RISE Technical Steering Committee (TSC) as the project looks to address performance, security and reliability issues involving open source software, she added.

    The TSC will also be able to shift skills from one initiative to another as priorities continue to evolve, Huffman added.

    The overall goal is to use a more holistic approach to eliminate duplicate work that each member of the RISC-V community would otherwise do separately to create an application processor, noted Huffman.

    The Linux Foundation Europe is an arm of the Linux Foundation that has been created to host projects that are of particular interest to organizations investing in open source software based in Europe. The RISE project is being launched at a time when the European Union debates a proposed Cyber Resilience Act that requires organizations that sell hardware platforms that connect to the internet to ensure that the devices their software runs on comply with cybersecurity best practices.

    The Linux Foundation also provides the funding for  the Open Source Security Foundation (OpenSSF) that is marshaling resources to enable maintainers of open source software to address vulnerabilities and adopt DevSecOps best practices to better ensure the security of downstream applications that incorporate that code. In addition, the OpenSSF is building out an incident management capability that will assist maintainers in the event a zero-day vulnerability is discovered.

    It’s not clear when all these investments might eventually lead to higher-quality open source software. However, it’s clear that vendors that benefit from open source software are rallying to address a range of quality assurance issues. Less clear is the degree to which enterprise IT organizations that benefit from open source software are making similar commitments.

    In the meantime, it’s not likely any of these issues will have a material impact on organizations’ reliance on open source software. The benefits of reduced costs and increasingly faster pace of innovation enabled by open source software development far outweigh the risks. However, as cybercriminals become more adept at embedding malware within software supply chains, there’s a clear need for increased vigilance.

  • I Guess This is Growing Up: Devs and CISA’s Secure-by-Design Guidelines

    I Guess This is Growing Up: Devs and CISA’s Secure-by-Design Guidelines

    With the downward pressure of a global recession, inflation and general post-pandemic turbulence underpinning disruption to multiple facets of life, it seems only fair that we in the IT, software and security industry would eventually feel the winds of change, too. We have rolled with the punches of escalating cybercrime and data breaches; the collective, fragmented sheriffs of a digital Wild West, and it’s high time we updated our strategy for safeguarding software against bad actors who seek to do widespread harm.

    The recently released National Cybersecurity Strategy signals the need for a seismic cultural shift for most companies and their developers and DevOps teams. The most glaring recommendation comes in the form of security accountability that falls primarily on software vendors. This is a positive step, though it is sure to cause teething problems, especially as many organizations struggle to accurately assess their security maturity across the board, particularly among the development cohort.

    Since CISA Director Jen Easterly’s trailblazing speech on secure design principles, we have eagerly awaited formal guidelines on how, exactly, organizations can approach this significant restructuring of software development as we know it. Thanks to a collaborative effort between the cybersecurity authorities of seven countries, we have a rather comprehensive overview of how to drive down vulnerabilities in software.

    This should be cause for celebration, but understandably, there is hesitation about how this will impact innovation, feature delivery and, ultimately, the bottom line as the pressure of security stewardship mounts for vendors. We’ve grown accustomed to dealing with cumbersome tooling that is ill-fitted with developer workflows, overwhelming them with a list of fixes in an information avalanche. Add SCA and SAST scanning, and you’ve got thousands of identified issues in a codebase that are just not practical to solve. 

    Despite this, in terms of our collective attitude towards code-level security, is it time we grew up and faced the music?

    Security Commitment and Transparency Boost Our Potential

    Back in March, the White House announced that one of its top priorities in the National Cybersecurity Strategy was to shake things up and start holding entities liable for sloppy software security measures:

    “Companies that make software must have the freedom to innovate, but they must also be held liable when they fail to live up to the duty of care they owe consumers, businesses or critical infrastructure providers,” the White House noted in the strategy.

    In the era of “speed at all costs” software development, my opinion may be unpopular, but I truly stand behind the idea that coding with security front-of-mind from the beginning is the only way we safely create our digital future. We were already facing insurmountable challenges with data breaches, nation-state attacks and infrastructure disruption, and that was before code-capable AI tools hit the mainstream. Until every person touching code cares more about the security implications of their actions, we will continue to see cybersecurity issues surge at a rapid rate.

    I know this is easier said than done. I have a lot of empathy for those standing at the bottom of this steep mountain trying to change the conversation around security accountability and enable developers in their organizations to share responsibility. To adequately fulfill key secure-by-design principles in software development, it is going to require precision training that establishes a basic skillset before deep-diving into more advanced security architecture methods. Developer-led threat modeling is a golden opportunity for the AppSec and development teams to work together toward a common goal and explore access control and insecure design pitfalls that plague less mature organizations.

    Still, the core issue has always revolved around a lack of ownership for security (until something goes wrong—then it’s the AppSec team left holding the bag) coupled with a low appetite for best practices to be established and executed across every part of the SDLC. The truth is, it was never acceptable for developers to start coding with so little security expertise, nor for us to focus on speed and functionality at the expense of safety and quality. Yet, here we are, and this is our crucible moment to accept shorter-term pain for longer-term ongoing success.

    We should want to create the most secure, robust software possible, and the “radical transparency” called for by Jen Easterly is a boon to vendors and clients alike. It is definitely a mark of value to showcase the company’s security prowess; brand trust and reputation are everything in this hyper-competitive landscape.

    Developers Need Guidance and Support to Thrive in a Secure-by-Default Environment

    The CISA guidelines pay particular attention to open-source and third-party components, both of which are widely used in most software development projects. It is imperative that developers are sufficiently security-aware to navigate potential inherent bugs with any preexisting code they use. However, in addition, they must be able to apply working knowledge of security fundamentals in the languages and frameworks they actively write in every day. Annual compliance videos played at 2x speed won’t cut it; they need training and a tech stack that is hyper-relevant, easy to access without context-switching from their workspace and that ultimately drives them to apply secure coding best practices as second nature.

    They cannot do this alone, and they can’t be expected to upskill in their own time. The cultural shift required to make the development team a fundamental cog in a thriving security program takes courage, time and commitment. But the reward is cohesion between AppSec and developers that would have been virtually impossible a decade ago and software that stands above those who are, in one way or another, happy to let security slide despite the risks.

    I guess this is what growing up looks like for all of us in the software industry, and it’s crucial if we want to stop the same old vulnerabilities that have plagued us for so long that they are old enough to drink.

  • Forget Change, Embrace Stability

    Forget Change, Embrace Stability

    DevOps is about change. It was introduced to accelerate change, and it has opened up a whole new world of tools and possibilities that are all based on the premise of changing IT. Starting with Agile, we now have DevOps variants and subprocesses that run the entire software development life cycle (SDLC) process.

    Want to make more frequent updates that can regularly show demonstrable progress to business owners? Agile and CI! Need to get the app settled in its inevitable home with automation? You need CD! Want testing and zero-day protections? Start with DevSecOps built into the CI/CD process and add in application and API security! Want to fully automate deployment in the whay builds are automated? Grab a GitOps tool and let’s goooo! The list goes on and on. Literally. We’ve almost got more tools than we have names for spaces at this point.

    And all of those tools introduce change. Most of them introduce a massive amount of change. You don’t get to GitOps without breaking some of your existing processes and reworking them. We have fallen in love with change in IT.

    But there is a funny thing that’s happened along the way; I’ve mentioned it before, but never from this perspective. All great changes require time for consolidation. Be it stock market swings or revolutions, afterward, there is a period of consolidation. In IT, containers are a good example. When they became capable of actually running workloads, IT experienced a period of time when making all the various machines run in VMs was the thing. People weren’t (generally) looking for the new big change in the process, they were just glad that those one-off machines that had custom hardware could now be moved to something more manageable and that spinning up servers could be done in days or weeks instead of the months it used to take when you had to order hardware. But that consolidation created organizational standards and stability that the organization enjoyed for years – even after containers rose to challenge VMs and some organizations even today.

    So every once in a while, take a breather, consolidate gains, create standards that have meaning for your organization, then continue on with whatever the shiny new change is. Make sure that once you have identified, for example, the need for code scanners that can work within or that are complementary to the integrated development environment (IDE), that you have one scanner—or one scanner per programming language—to maintain and monitor instead of one tool per project. That saves time and creates stability, both of which will serve the organization moving forward.

    Change is here to stay; it’s the only constant. IT is caught in a cycle of change followed by change, with the next wave driven by generative AI. But in times of change, seek stability and, in times of stability, seek changes that can have a positive impact. So go standardize and organize before the next wave hits.

    Mostly, the changes will serve the organization by freeing up your time and letting you spend more of your time rocking the IT world. Those systems stay up because of you. More time is more innovation, and more stability means less burnout. All of which is good for the organization.

  • Finding Your Passion

    Finding Your Passion

     

  • What Is a Cloud Operations Engineer?

    What Is a Cloud Operations Engineer?

    CloudOps, short for cloud operations, refers to the processes, tools and strategies employed to manage, monitor and optimize the performance, security and availability of cloud-based infrastructure, applications and services. It encompasses a set of best practices and methodologies that ensure the smooth functioning of cloud environments, including resource provisioning, configuration management, automation, deployment, monitoring and incident response. The primary goal of CloudOps is to maintain reliable, scalable and cost-effective cloud solutions that meet the needs of businesses and their end-users.

    What is a Cloud Operations Engineer?

    A cloud operations engineer is a professional responsible for managing, monitoring and optimizing cloud-based infrastructure, applications and services within an organization. They play a crucial role in ensuring the smooth functioning of cloud environments by handling tasks such as resource provisioning, configuration management, deployment, automation and incident response. 

    Additionally, they collaborate with development and IT teams to design and implement scalable, secure and cost-effective cloud solutions that align with business objectives and end-user requirements. Cloud operations engineers must possess a deep understanding of cloud computing technologies, platforms and best practices, as well as strong problem-solving, communication and collaboration skills.

    Responsibilities and Roles of a CloudOps Engineer

    A CloudOps engineer plays a vital role in managing and maintaining cloud infrastructure and services. Their responsibilities and roles might include:

    • Cloud infrastructure management: Design, deploy and maintain cloud infrastructure, ensuring optimal performance, security and scalability.
    • Cloud migration: Responsible for planning, executing and managing cloud migration of applications, workloads and infrastructure from on-premises or other cloud environments to a target cloud platform.
    • Resource provisioning: Allocate and manage cloud resources such as compute, storage and networking to meet the needs of applications and services.
    • Configuration management: Define and implement infrastructure configurations and manage changes to maintain consistency across environments.
    • Automation: Develop and implement automation scripts and tools to streamline and simplify repetitive tasks, improving efficiency and reducing human error.
    • Monitoring: Set up and maintain monitoring tools to track the performance, availability and security of cloud services, proactively identifying and resolving issues.
    • Incident response: Troubleshoot and resolve incidents, collaborating with development and IT teams to minimize downtime and maintain service quality.
    • Security and compliance: Implement and enforce security policies, best practices and compliance requirements to protect sensitive data and maintain cloud infrastructure integrity.
    • Cost optimization: Monitor and analyze cloud resource usage to identify opportunities for cost savings, recommending and implementing cost optimizations.
    • Backup and disaster recovery: Develop and maintain backup and disaster recovery plans to ensure data and application availability during unforeseen events.
    • Collaboration and communication: Work closely with development, IT and business teams to ensure alignment with project goals and provide guidance on cloud best practices.
    • Continuous improvement: Stay up-to-date with emerging cloud technologies, platforms and trends, continuously improving cloud operations and adapting to changing requirements.
    • Documentation: Create and maintain comprehensive documentation of cloud infrastructure, configurations, processes and procedures to ensure knowledge sharing and team collaboration.

    How to Become a CloudOps Engineer

    Becoming a CloudOps engineer typically requires a combination of education, relevant experience and certifications. The following steps can help you pursue a career as a CloudOps engineer:

    • Gain foundational knowledge: To become a CloudOps engineer, you will need a solid foundation in IT infrastructure, networking and cloud technologies. You can gain this knowledge through a degree in computer science or information technology or through certifications such as AWS Certified Solutions Architect or Microsoft Certified: Azure Solutions Architect.
    • Develop cloud skills: CloudOps engineers need to have expertise in cloud platforms such as AWS, Azure or Google Cloud. You can gain experience by working with cloud technologies in your current job or by creating personal projects on cloud platforms.
    • Learn automation tools: CloudOps engineers need to automate infrastructure and deployments using tools like Ansible, Puppet, Chef or Terraform. You can learn these tools through online courses or tutorials.
    • Develop soft skills: CloudOps engineers need strong communication, collaboration and problem-solving skills. You can develop these skills by working in team-based environments, attending networking events or conferences and taking courses or workshops in communication and collaboration.
    • Gain experience: To become a CloudOps engineer, you need hands-on experience in deploying, managing and monitoring cloud infrastructure. You can gain this experience by working on cloud projects, participating in hackathons or contributing to open source cloud projects.
    • Consider certification: CloudOps engineers can demonstrate their expertise through certification programs such as AWS Certified DevOps Engineer or Microsoft Certified: Azure DevOps Engineer. Certification can help you stand out in a competitive job market and demonstrate your proficiency in cloud technologies.

    Conclusion

    In conclusion, a cloud operations engineer is an essential professional in today’s cloud-centric IT landscape. They play a critical role in managing, monitoring, and optimizing cloud infrastructure, ensuring that applications and services run efficiently, securely and cost-effectively. 

    With a deep understanding of cloud technologies, platforms and best practices, they collaborate with various teams to align cloud solutions with business objectives and end-user needs. As cloud adoption continues to grow across industries, the demand for skilled cloud operations engineers is expected to rise, making it a promising career path for those seeking opportunities in the ever-evolving world of cloud computing.

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    Indeed.com

    AT&T
    Dallas, Texas
    Software Engineer
    $158,766 to $210,100

    LinkedIn

    Northwestern Mutual
    Syracuse, New York
    Senior Systems Software Engineer – DevOps
    $97,300 to $139,900

    CareerBuilder.com

    United Healthcare
    Remote
    Senior DevOps Engineer
    $85,000 to $167,300

    SimplyHired

    GEICO
    Remote
    Azure DevOps Engineer
    $82,000 to $185,000

    Dice.com

    Integrated Practice Solutions
    San Diego, California
    DevOps Engineer
    $80,000 to $140,000

  • CDF Marries Emporous Repository to Ortelius Management Platform

    CDF Marries Emporous Repository to Ortelius Management Platform

    The Continuous Delivery Foundation (CDF) has added the open source Emporous repository for software artifacts to an Ortelius platform for managing the software artifacts that make up a microservice.

    Ortelius was originally created by DeployHub and subsequently contributed to the CDF, which also oversees the development of the open source Jenkins continuous integration/continuous delivery (CI/CD) platform.

    Emporous is based on a Universal Object Reference (UOR) conceptual model that enables dynamic schema registration and decentralized content-addressable reference of a wide spectrum of software artifacts. Originally developed by Red Hat, Emporous will now be managed as a sub-project within the larger Ortelius incubation-level project by the CDF.

    DeployHub CTO Steve Taylor said this integration would make it simpler to store, organize and search metadata related to software artifacts, along with the artifacts themselves, within the context of the Ortelius platform for managing microservices.

    Ortelius provides insights into the relationships and dependencies between different software components and applications. It does this by collecting and aggregating pipeline data from multiple sources within a software supply chain. Emporous will be incorporated within Ortelius to enhance metadata search capabilities and provide a single repository to store any type of artifact, from containers to jar files. It will now also be able to store metadata and also the artifacts themselves within the same Emporous catalog.

    Those capabilities will significantly improve security and compliance reporting across a software supply chain, added Taylor. DevOps and cybersecurity teams will be able to automatically apply and enforce supply chain policies and procedures, he said.

    DevOps teams need a way to aggregate all the data from the various tools they use and the data must be easily searchable. Armed with those insights, it should become simpler to surface insights into, for example, which components of an application contain known vulnerabilities.

    In the longer term, IT organizations will be able to apply machine learning algorithms and other forms of artificial intelligence (AI) to automatically surface insights that will enable DevOps teams to automate workflows, said Taylor.

    Microservices-based applications are considerably harder to build and manage than monolithic applications, but they are also easier to maintain because individual microservices can be ripped and replaced.

    Most microservices today are built using containers, but there are also microservices that are written in Java, for example. In some cases, developers view monolithic applications as being a large, less flexible microservice that they invoke via an application programming interface (API).

    There’s no doubt that the rise of microservices made application environments a lot more complex. The tools needed to simplify the management of the next generation of applications are finally starting to become more widely accessible. In theory, the number of organizations that can successfully build and deploy these types of applications should steadily increase in the months and years ahead.

    In the meantime, there’s clearly still plenty of work that needs to be done to improve the overall state of software delivery. A recent CDF survey found only 22% of respondents were using continuous integration and continuous delivery (CD) to automate the building, testing and deployment of code in a production environment. As CD becomes more distinct from CI, however, the number of organizations that can achieve that goal when deploying microservices-based applications should steadily increase.

  • Is Your Monitoring Strategy Scalable?

    Is Your Monitoring Strategy Scalable?

    Every company is trying to get better insights into its operational effectiveness, and they are running into the same problem: Scale. So what does a scalable monitoring strategy look like and how can you safeguard against the most significant issue in observability?

    What is a Scalable Monitoring Strategy?

    We’ll begin by identifying the two things most impacted by scale: Cost and performance. Cost can be broken down into storage and computation. It is obvious that to hold more data, more storage is needed, but what about compute power? This is important because queries need to search through more data—and thus, use more compute power—to return a result.

    This creates a trade-off between performance and cost-effectiveness. It becomes more and more challenging to have queries that resolve instantly as the dataset increases, so engineers will often simply tolerate a performance decline. This further impacts usability and the general usefulness of the system. If it takes 10 seconds to resolve every query, you’ve just cut down your data discovery efficiency by a factor of 10.

    Scaling Means Performance and Cost-Effectiveness

    To scale a monitoring strategy, a wise architect needs to break this standoff between performance and cost and approach the problem like a data scientist. A scalable monitoring strategy starts with a simple question: What is the use case for this data?

    Some data is only ever ingested because it might be needed. It spends its entire lifetime undisturbed and is eventually compressed or deleted. On the other hand, some data is queried every minute of every day and is integral. Knowing how data is used means an engineer can decide the value of that data.

    In the battle to create a scalable monitoring strategy, there’s a three-step approach.

    Step One: Track Data Usage

    Which data is ingested and never queried? Which data never spends any time at rest? Build a map of how often data is consumed so that any strategic choices are informed by the use case.

    It is most common to group data into three different use cases:

    Frequent Access – Data that is constantly queried and needs to be available at a moment’s notice.

    Monitoring – Data that drives dashboards or trains machine learning models but which is largely useless after it has been processed.

    Compliance – Data that is held in case it is needed, but it is only queried sometimes. For example, audit logs.

    Step Two: Optimize Storage and Consumption

    Once the use case of data—whether it’s logs, metrics or traces—is understood, the next stage is to optimize. This means creating different storage and query solutions for the different usages we’ve seen above.

    Frequent Access – Rapid queries that can be optimized and tuned. OpenSearch is a good option, although management overhead can be painful, especially at scale.

    Monitoring – This is largely about transforming data. For example, ingesting logs, converting them into metrics and deleting the original log. This is very powerful because metrics take up far less space than logs and can be stored much more cost-effectively.

    Compliance – Low-cost storage like Amazon S3 is a good option, but this data must still be accessible, even if simply by reindexing.

    How Easy is it to Build These Capabilities?

    It is trivial to create an OpenSearch cluster, but it’s challenging to manage it at scale. Likewise, it’s simple to convert logs to metrics, but doing so in a performant way at scale is complex. Holding onto compliance logs as they scale near infinitely can be difficult, and reindexing this data is a non-trivial operation when dealing with large data volumes.

    However, the key takeaway from this analysis should be that these are capabilities that any organization should have if they intend to scale their monitoring solutions. If they can be attained through a SaaS vendor, then this is a serious option to consider because it enables companies to immediately take advantage of this capability without the upfront, unpredictable and often ongoing cost of in-house engineering.