devopsy.kinsta.cloud

Category: Cloud Management

  • Survey Finds Cloud Management Strategies Shifting

    Survey Finds Cloud Management Strategies Shifting

    Even though organizations continue to consume cloud resources inefficiently, they continue to increase the amount of money allocated. These were among the findings of a global survey of 1,000 technology practitioners and decision-makers conducted by Forrester Consulting on behalf of HashiCorp.

    The survey found a full 94% of respondents reported that their organization struggled with cloud waste. Top factors contributing to avoidable cloud spending included overprovisioning resources (50%), idle or underused resources (50%) and a lack of needed skills (43%). Nevertheless, more than half of respondents (56%) said they boosted their cloud spending in the last year compared to 22% which reduced it.

    At the same time, however, 92% of respondents said they are moving toward adopting, standardizing or scaling platform teams as part of an effort to centralize the management of cloud services, but only 39% are scaling the use of platform teams throughout the organization. The survey also found, however, that organizations are addressing skills shortages by adopting a common operating model/shared automated workflows (58%), reskilling/upskilling (56%) and working with partners (43%).

    The survey also found many organizations are also committed to multi-cloud computing, with nearly three-quarters (73%) reporting this strategy is helping them reach their business goals. Another 19% said they expected this strategy to help them achieve their business goals in the next year.

    The most-cited reasons for adopting a multi-cloud strategy were reliability (51%), cost reduction (48%), security and governance (45%), digital transformation (45%) and scalability (45%). Only 10% had no choice but to go multi-cloud due to acquisitions or other issues.

    Security (88%), availability (84%) and scalability (84%) were identified most often as being important/very important multi-cloud success factors, followed closely by regulatory/compliance requirements (82%), staffing and skill levels (82%), budget (80%), visibility into cloud infrastructure (78%), automated tooling (77%) and platform teams (75%).

    The top cloud initiatives were data protection (77%), followed by secrets management and access control (both at 75%), the survey finds. The top security concerns were password/credentials/secrets leakage (50%) followed by data theft (49%), phishing (46%) and ransomware (42%).

    Overall, respondents said the biggest benefits of having a multi-cloud computing strategy were a stronger security posture (74%) and helping attract, motivate and retain skilled staffers (68%).

    Chris Van Wesep, senior director of product marketing for HashiCorp, said it’s clear organizations are purposely using multiple clouds to reduce costs by pitting one cloud service provider against another. The challenge organizations face when using that approach is they need to define a consistent set of development processes that can be used across multiple clouds to keep total costs under control.

    Organizations that embrace multi-cloud computing are also more likely to implement security best practices because of their need to consistently manage multiple cloud computing environments, he added.

    Of course, there is always going to be a great disparity between high- and low-performing organizations. The survey found that about 25% of survey respondents are high performing, with an equal number falling at the other end of the spectrum. The rest fall somewhere in between those two ends of the spectrum. The issue is that the gulf between the high and low ends of that spectrum appears to be widening as more organizations fail to muster the level of skills and expertise needed to succeed in the cloud era.

  • VMware Streamlines IT Management via Cloud Foundation Update

    VMware Streamlines IT Management via Cloud Foundation Update

    VMware this week updated its Cloud Foundation platform and rolled up recent updates to the company’s virtualization portfolio all at once.

    Version 5.0 of VMware Cloud Foundation (VCF) encompasses the latest versions of VMware vSphere 8.0u1 for workload management, VMware vSAN 8.0u1 for scale-out storage, VMware NSX 4.1 for networking and VMware vRealize LifeCycle Manager 8.10 (Aria) for cloud management.

    In addition, VMware has added a single sign-on (SSO) capability, dubbed Isolated SSO Workload Domains, that enables administrators to configure new workload domains using SSO instances from different providers or platforms via NSX.

    Rick Walsworth, director of product marketing for the Cloud Infrastructure Business Group at VMware, said that as the management of IT continues to evolve in VMware environments, more organizations are opting to update their entire VMware software stack simultaneously rather than attempting to update components individually.

    The primary reason for that shift is that the company ensures that all the latest VCF component releases have been thoroughly stress-tested from an integration perspective, he noted.

    During uncertain economic times, VMware is seeing an increased focus on simplifying IT environments as IT teams are asked to do more with less, said Walsworth. As such, more organizations are looking to standardize on a stack of infrastructure software that can be managed more consistently, he added.

    More organizations are also looking to reduce the total cost of IT by employing the same stack of infrastructure software across hybrid cloud computing environments where workloads are deployed both in on-premises IT environments and public cloud, said Walsworth.

    The company is also seeing increased adoption of DevOps best practices to programmatically manage IT environments that need to respond to rapidly changing requirements, he noted. IT teams are looking to work more collaboratively rather than relying solely on specialists to manage specific functions, said Walsworth.

    Some of that shift is also being driven by Tanzu, a distribution of Kubernetes that the company makes available as part of VCF, he noted. Most application developers don’t have a preference for a specific Kubernetes distribution, so many organizations view Tanzu as an opportunity to extend their existing management framework to a de facto standard for building the next generation of cloud-native applications, said Walsworth.

    VMware is in the midst of being acquired by Broadcom. Nevertheless, VMware this week reported revenues for the first quarter 2023 of $3.28 billion, a 6% year-over-year increase. Subscription and software-as-a-service (SaaS) revenue for the first quarter was $1.22 billion, an increase of 35% year-over-year. That increase in revenue suggests any concerns about the future direction of the company are not impacting the willingness of IT organizations to continue to rely on VMware.

    It may be a while before all the dust settles around Broadcom’s acquisition of VMware. In the meantime, however, it’s clear that the way IT is managed in the cloud era is continuing to rapidly evolve regardless of who owns VMware.

  • Why You Need a Multi-Cloud and Multi-Region Deployment Strategy

    Why You Need a Multi-Cloud and Multi-Region Deployment Strategy

    As I began writing this article, I encountered some technical challenges in my attempt to provision three cloud providers in three separate regions to simulate multi-cloud and multi-region deployments.

    I couldn’t get one of the regions to behave as I expected when provisioning the environment. As it turned out, the region (and by association, cloud) was down due to a cut undersea cable, and it would take several weeks until the outage was mitigated.

    Fortunately, there are plenty of clouds in the sky to choose from, but this experience emphasized the importance of multi-cloud redundancy, especially in a production environment.

    This article examines the benefits of adopting a multi-region and multi-cloud deployment strategy for better resiliency, regulatory compliance and optimal user experience.

    Resilient Reasons to Choose Multi-Cloud and Multi-Region Architectures

    The first reason to choose a multi-cloud or multi-region environment is for data resilience. When our data is in a single region or on a single cluster, there is a good chance that you will experience data failure. You need a mitigation strategy if you want to keep the data lights on 24/7. These strategies fall into two buckets: Multi-region and multi-cloud.

    We first need to define the differences between multi-cloud and multi-region. The architectures are simply the implementation details of how we plan for one versus the other. However, depending on your database provider, those details can represent a significant time investment, which we’ll look at later.

    Multi-Region

    With multi-region, we typically deploy our applications to globally distributed regions. However, regional deploys are subjected to physical faults. Inclement weather, political unrest, acts of foreign or domestic terrorism and other physical variables have all led to regional outages in the past.

    Regardless of the cloud, the regions are exposed to similar physical faults. Instead of losing your data, we can switch to another region to handle the workload. Even though much as our industry hates latency, it is always preferred over loss.

    Deploying to multiple regions allows for failover handling. If the Sydney region goes down, Singapore may still be online. It would be a globally bad day if the same physical faults simultaneously impacted Frankfurt, Sydney and Seattle, for example.

    Multi-Cloud

    Physical faults are only one of the problems that public and private clouds encounter. User error, poorly configured networks and even banal errors can have substantial consequences that impact entire clouds, either from the provider or the user.

    In these circumstances, regional failover won’t help you. An expired certificate, accidental deletion or disabling the wrong port will take you offline, regardless of whether you’re in Seattle or Sydney. If that happens, someone somewhere is most definitely awake and is likely very unhappy.

    Choosing a multi-cloud configuration protects you from these types of faults so that if Google, for example, has interruptions, AWS is still online—or IBM, Deutsche Telekom, OVH or Hetzner.

    The Correct Answer: Deploy Both!

    Multi-region is a go-to method to mitigate against mostly physical faults. Multi-node clusters in each of those regions handle general data corruption or failures on a more granular level.

    Multi-cloud is a go-to method to mitigate configuration or software-level faults, which are not that uncommon.

    The correct answer is to blend the two, working within operational cost constraints and general survivable requirements from regulatory and business objective use cases. For example, I do want access to my heart-rate data from two years ago, but the order of browsing across Amazon shopping is less important.

    Regulatory Reasons for Multi-Region Architecture

    Apart from data resiliency, most companies serving a global market are subject to a number of geopolitical influences, as well. Most users will be familiar with the GDPR regulations ensuring European data stays on European soil. Further regulations require the ability for government oversight to request data at will from service providers, which requires its own set of reasonable isolation from those not in those jurisdictions.

    Using a multi-region database provider allows companies to ship single binaries of software without creating isolated applications for different countries—a practice that was the norm not too many years ago.

    Performant Reasons for Multi-Region Architecture

    Co-locating data and access provides the optimal pathway for getting a user’s data to the end destination. It goes without saying, but the less copper your data has to traverse, the faster the experience will be for your users. Retention and regulation aside, in many cases, you need to mitigate latency caused by region separation.

    It’s not just the individual user’s data. Imagine a sales enablement platform where an account executive needs to manage all of APAC or EMEA. If these accounts are optimally stored for reads on geo boundaries, you can remove seconds of delay between page navigations for your account manager. We all know how much account executives like to click!

    Choosing a DB Provider With Multi-Region and Multi-Cloud

    At this point, you should be convinced of the benefits of multi-region and multi-cloud deployments. But at what cost? Creating replication, failover, logging and synchronization mechanisms is a big, expensive ask. Even with deployment tooling, coordinating all the pieces for multi-region deploys is still a significant operational lift, let alone multi-cloud, where variables in APIs come into play.

    Choosing a single provider that deploys across regions and clouds means you get to benefit from their collective experience while reducing a significant burden.

    Having a database provider offer one-click multi-region deploys is the way to go, but you still need to handle the data access layer. The role-based access, federation and other settings that enable your developers still need to be distributed along with the underlying data sources.

    You need a provider that makes it simple to deploy the same configuration for horizontally scaled instances, leaving you to pick your geo router of choice to distribute the request to the nearest point of presence.

    Closing

    Multi-cloud and multi-region deployments provide significant redundancy, scalability and flexibility. Not only does this enable faster access times and improved reliability, but also provides the flexibility required to adapt to disruptions or unpredictable increases in traffic.

    Deploying multi-cloud and multi-region is no small feat–but the peace of mind and security it provides make it worth the effort.

  • Cloud Drift Detection With Policy-as-Code

    Cloud Drift Detection With Policy-as-Code

    Cloud configurations can change and change often. Introducing new technologies, releasing new features and supporting new business requirements entail a constant flow of configuration changes in web application development.

    However, drift occurs regardless of how well-designed your IaC implementation is. The term “drift” is used to denote a state in which the actual state of your infrastructure deviates from the configuration.

    This article examines cloud drift detection, why it occurs and how to remediate it.

    Understanding the Problem

    Cloud configurations are prone to change and can change frequently. Businesses often use infrastructure-as-code (IaC) to manage cloud provisioning changes. IaC makes it easier and more reliable for organizations to manage and facilitate changes to the cloud deployment process.

    However, inconsistencies in your IaC deployment process can lead to uncertainty about how and where your resources are provisioned, controlled and protected, resulting in lower productivity. No matter how robust your IaC implementation, drift can creep in.

    Changes to infrastructure that are made independently of the code responsible for provisioning can lead to considerable drift, and if not adequately monitored, can lead to significant security concerns.

    What is Drift?

    In the context of infrastructure and application management, drift refers to the system state in which the actual configuration and state of the system deviate from the intended or expected configuration and state.

    Drift can be caused by software updates, manual changes and incorrect configurations. Drift management is an essential aspect of infrastructure and application management. By detecting and correcting drift, organizations can ensure that their systems remain stable, secure and compliant with their policies and industry standards.

    Configuration Drift: Definition, Causes and Examples

    Configuration drift can creep in over time despite consistently building and configuring your servers. It refers to the state of the system when configuration changes are not in sync with the value previously set.

    For example, configuration drift can occur when you are yet to document the changes made to the production environment. This results in the production and staging environments getting out of sync.

    Some of the other causes of configuration drift are configuration changes because of applying patches and updating network equipment, the addition of new resources to the network and lack of clarity about the desired state of the system.

    You can leverage configuration drift management tools such as Netwrix and Aqua to detect configuration drift.

    Infrastructure Drift: Definition, Causes and Examples

    Infrastructure drift refers to a phenomenon in which the actual state of the infrastructure differs from its desired state because the defined configuration, settings or properties of the infrastructure and the actual state of the provisioned resources differ.

    Infrastructure drift may result from manual changes, updates outside of configuration management processes, conflicting IaC code, changes in network configurations, changes due to security settings and configuration differences during system upgrades and maintenance.

    To detect infrastructure drift, you can take advantage of infrastructure drift detection tools such as Terraform, CloudQuery and driftctl.

    Infrastructure Drift and Configuration Drift: How do they compare?

    Both configuration and infrastructure drift can lead to security vulnerabilities, compliance issues and operational challenges. Although infrastructure drift and configuration drift are related, they aren’t the same.

    Infrastructure drift refers to inconsistencies or discrepancies between the intended or desired state of the infrastructure and the actual state of the resources provided. Configuration drift refers to inconsistencies or discrepancies between the current configuration of a component or system and the intended or desired configuration.

    Infrastructure drift encompasses everything from physical components to virtual components to networks, storage and computing resources. Configuration drift, on the other hand, focuses on individual components or systems within the infrastructure and their configuration settings.

    What is Drift Detection?

    Drift detection in the context of cloud infrastructures typically compares the actual state of resources deployed in the cloud environment with the defined state described in IaC templates.

    Typically, this is automated by tools, scripts or services that analyze and compare configurations, resource states or event logs. These tools are adept at issuing notifications or alerts when discrepancies are detected.

    By detecting discrepancies, organizations can identify and resolve inconsistencies, security vulnerabilities and compliance violations. Consequently, the risks related to security breaches are reduced and remediation actions become easier to implement.

    Why Policy-as-Code for Cloud Drift Detection?

    Policy-as-code provides several benefits when used for cloud drift detection:

    • Early detection of drift: Policy-as-code helps detect drift early detection by comparing the current state of the cloud environment with the desired state defined in the policies. This helps identify problems and solve issues before they escalate to a catastrophic level.
    • Automated enforcement: Policy-as-code enables automated enforcement of policies, ensuring that the cloud environment complies with the defined rules and requirements. This helps reduce the potential risks associated with human errors.
    • Faster remediation: Policy-as-code enables faster remediation by automatically detecting drift and taking the necessary remediation actions such as rolling back changes, updating configurations or notifying relevant teams. This facilitates faster response times and reduces the risks associated with security breaches and downtime.
    • Enforce best practices, policies and guidelines: With policy-as-code, best practices, policies and security guidelines can be implemented automatically, reducing security and compliance risks.

    Steps for Drift Detection Using Policy-as-Code

    Here are the steps that you need to follow to detect and fix drift in your infrastructure:

    1. Define your policies: You should first define the policies that should be used to describe the desired state or the baseline of your cloud environment and cover security and compliance aspects of your cloud environment.
    2. Codify and deploy policies: Then these policies should be codified and stored in the version control system. The next step should be to deploy the policies to your cloud environment using a policy engine.
    3. Identify configuration changes: Once your policies have been deployed, the next step should be to identify any new resources that may introduce risks and compare them to the baseline you’ve defined.
    4. Evaluate and remediate: You should evaluate the state of the environment against the defined policies to detect any drift. As soon as any drift is detected, you should take corrective actions to remediate the issues by rolling back changes, updating configurations or notifying the relevant team members.
    5. Review policies: To ensure that the policies are up-to-date and relevant, it is imperative that you review them often and make changes to policies as needed.
    6. Update and deploy: The last step should be to update the code files and then deploy the solution yet again.

    Build a Successful Drift Detection Strategy

    Maintaining your infrastructure at the desired state, ensuring compliance and mitigating risks associated with configuration drifts requires a successful drift detection strategy. Here are key strategies you can adopt to build a successful drift detection strategy.

    Understand When Drift Becomes a Risk

    Drift can profoundly impact a system’s stability, security and compliance. When your server configuration deviates from its intended configuration, it might stop working or become vulnerable to security threats.

    Most importantly, this might be a significant threat to security if you don’t have proper monitoring in place. To prevent infrastructure drift and update your security tools, it is important to evaluate and communicate your adoption of IaC.

    Detecting Drift: Detecting drift Between IaC and the Cloud

    To mitigate the risks associated with drift, configurations that have been changed in real-time must be identified and reset. You can achieve this programmatically and continuously by comparing configuration changes between IaaS and IaC. Programmatically identifying the configuration changes, i.e., detecting the changes to the configuration using code, is the best strategy to reduce drift-related risks.

    From Drift Detection to Drift Remediation: Responding to drift

    By detecting drift, you’ve won half the battle–you should now devise strategies to remediate it. By keeping an eye on configuration drift and responding to it promptly, you ensure the consistency and reliability of your infrastructure, minimize security risks and keep your system in a consistent state.

    Figure 1: Detect and Remediate Drift in the Cloud

    It’s important to have effective configuration drift response procedures in place to ensure the stability and security of your environment and also keep it effective for change management and control. You can respond to drift in the following ways:

    • Develop mechanisms to detect and report configuration drift
    • Understand the underlying causes of drift
    • Documenting the drift, including affected resources, configuration deviations and impacts
    • Resolve the configuration drift and return the system to the desired state
    • Following change management processes and obtaining approvals prior to implementing changes
    • Monitor the system to ensure drift has been resolved and the configuration remains stable

    Key Strategies for Detecting Drifts Between IaC and the Cloud

    There are a few strategies that can be adopted for detecting drift.

    Detect Configuration Changes That may Introduce Risks

    To identify configuration changes that can introduce risks, you should monitor and analyze changes proactively. Here are some of the key strategies to do so:

    • Create a baseline configuration for your system.
    • Review the system logs regularly for any suspicious or unexpected configuration changes.
    • To capture detailed information about configuration changes, comprehensive logging is helpful.
    • Implement a change management process for managing configuration changes.
    • Take advantage of automated configuration management tools such as Chef, Puppet, Ansible, etc.
    • Leverage threat intelligence to ensure that you are always up to speed on the most recent vulnerabilities, security threats and attack trends.
    • Conduct training to increase awareness about configuration changes’ potential consequences and risks.

    Establish Baselines and Safely Auto-Remediate Drift

    Establishing baselines and safely auto-remediating drift involves setting a reference point for the desired state of your infrastructure and automatically adjusting any detected drift back into compliance with the baseline.

    You can maintain the desired state of your infrastructure, reduce manual effort and quickly resolve configuration drift by establishing baselines and implementing security processes for automatic remediation.

    This minimizes the impact of drift on your systems while improving your environment’s security, compliance and stability.

    Track all Configuration Changes in Your Cloud Environment

    Tracking all configuration changes to your cloud environment is critical to maintaining visibility, accountability and security. It can help you improve visibility, detect unauthorized or unintended changes and maintain compliance and accountability.

    This way, you can proactively manage your cloud infrastructure, identify potential security risks and ensure its integrity and stability.

    Below are the key strategies for tracking cloud configuration changes at a glance:

    • Ensure detailed logging for all cloud components, including infrastructure, applications and security
    • Track API calls and activities within your cloud environment with cloud monitoring and auditing
    • Set up alerts and notifications for critical deviations from established baselines
    • Implement configuration changes through formal change management processes
    • Manage cloud configuration with tools such as AWS Config and Azure Automation

    Managing Drift in Production

    To effectively manage drift in production, it is crucial to consistently observe the configuration and state of production systems, keeping them in line with the policies that outline the desired state.

    Here are a few guidelines to help you reduce production drift:

    • Specify the required state: Define the intended state of the production environment as a set of policies using declarative compliance tools.
    • Monitor continuously: To determine any deviations, use a monitoring tool to monitor the production environment or compare the system’s present state to the desired state to identify any deviations.
    • Identify the root cause: If drift is detected, it is crucial to identify its root cause. You can do this by examining the logs, analyzing system performance or observing the configuration alterations over time.
    • Rectify the drift: After determining the issue’s root, you should address it to align its compliance with the intended state. To do this, you may need to examine logs, examine configuration changes, and analyze system performance.
    • Automate the remediation process: To ensure that drift is automatically and quickly rectified, you should automate the remediation process. To achieve this, you can leverage a declarative compliance tool that can help you automatically implement the changes to bring the system back into conformity with the intended state.
    • Review and update policies: You should review and revise your policies to ensure they are still relevant and useful and stay updated. To achieve this, you may need to adjust the policies to reflect new compliance requirements or shift your business requirements.

    Summary

    By identifying drift in cloud environments, organizations can use policy-as-code to ensure that their cloud infrastructure is safe, compliant and secure. In this way, they can ensure compliance with industry best practices, regulatory standards and internal policies. This reduces the risks associated with configuration deviations and enables enterprises to maintain a secure and reliable cloud environment.

  • What Is a Cloud Operations Engineer?

    What Is a Cloud Operations Engineer?

    CloudOps, short for cloud operations, refers to the processes, tools and strategies employed to manage, monitor and optimize the performance, security and availability of cloud-based infrastructure, applications and services. It encompasses a set of best practices and methodologies that ensure the smooth functioning of cloud environments, including resource provisioning, configuration management, automation, deployment, monitoring and incident response. The primary goal of CloudOps is to maintain reliable, scalable and cost-effective cloud solutions that meet the needs of businesses and their end-users.

    What is a Cloud Operations Engineer?

    A cloud operations engineer is a professional responsible for managing, monitoring and optimizing cloud-based infrastructure, applications and services within an organization. They play a crucial role in ensuring the smooth functioning of cloud environments by handling tasks such as resource provisioning, configuration management, deployment, automation and incident response. 

    Additionally, they collaborate with development and IT teams to design and implement scalable, secure and cost-effective cloud solutions that align with business objectives and end-user requirements. Cloud operations engineers must possess a deep understanding of cloud computing technologies, platforms and best practices, as well as strong problem-solving, communication and collaboration skills.

    Responsibilities and Roles of a CloudOps Engineer

    A CloudOps engineer plays a vital role in managing and maintaining cloud infrastructure and services. Their responsibilities and roles might include:

    • Cloud infrastructure management: Design, deploy and maintain cloud infrastructure, ensuring optimal performance, security and scalability.
    • Cloud migration: Responsible for planning, executing and managing cloud migration of applications, workloads and infrastructure from on-premises or other cloud environments to a target cloud platform.
    • Resource provisioning: Allocate and manage cloud resources such as compute, storage and networking to meet the needs of applications and services.
    • Configuration management: Define and implement infrastructure configurations and manage changes to maintain consistency across environments.
    • Automation: Develop and implement automation scripts and tools to streamline and simplify repetitive tasks, improving efficiency and reducing human error.
    • Monitoring: Set up and maintain monitoring tools to track the performance, availability and security of cloud services, proactively identifying and resolving issues.
    • Incident response: Troubleshoot and resolve incidents, collaborating with development and IT teams to minimize downtime and maintain service quality.
    • Security and compliance: Implement and enforce security policies, best practices and compliance requirements to protect sensitive data and maintain cloud infrastructure integrity.
    • Cost optimization: Monitor and analyze cloud resource usage to identify opportunities for cost savings, recommending and implementing cost optimizations.
    • Backup and disaster recovery: Develop and maintain backup and disaster recovery plans to ensure data and application availability during unforeseen events.
    • Collaboration and communication: Work closely with development, IT and business teams to ensure alignment with project goals and provide guidance on cloud best practices.
    • Continuous improvement: Stay up-to-date with emerging cloud technologies, platforms and trends, continuously improving cloud operations and adapting to changing requirements.
    • Documentation: Create and maintain comprehensive documentation of cloud infrastructure, configurations, processes and procedures to ensure knowledge sharing and team collaboration.

    How to Become a CloudOps Engineer

    Becoming a CloudOps engineer typically requires a combination of education, relevant experience and certifications. The following steps can help you pursue a career as a CloudOps engineer:

    • Gain foundational knowledge: To become a CloudOps engineer, you will need a solid foundation in IT infrastructure, networking and cloud technologies. You can gain this knowledge through a degree in computer science or information technology or through certifications such as AWS Certified Solutions Architect or Microsoft Certified: Azure Solutions Architect.
    • Develop cloud skills: CloudOps engineers need to have expertise in cloud platforms such as AWS, Azure or Google Cloud. You can gain experience by working with cloud technologies in your current job or by creating personal projects on cloud platforms.
    • Learn automation tools: CloudOps engineers need to automate infrastructure and deployments using tools like Ansible, Puppet, Chef or Terraform. You can learn these tools through online courses or tutorials.
    • Develop soft skills: CloudOps engineers need strong communication, collaboration and problem-solving skills. You can develop these skills by working in team-based environments, attending networking events or conferences and taking courses or workshops in communication and collaboration.
    • Gain experience: To become a CloudOps engineer, you need hands-on experience in deploying, managing and monitoring cloud infrastructure. You can gain this experience by working on cloud projects, participating in hackathons or contributing to open source cloud projects.
    • Consider certification: CloudOps engineers can demonstrate their expertise through certification programs such as AWS Certified DevOps Engineer or Microsoft Certified: Azure DevOps Engineer. Certification can help you stand out in a competitive job market and demonstrate your proficiency in cloud technologies.

    Conclusion

    In conclusion, a cloud operations engineer is an essential professional in today’s cloud-centric IT landscape. They play a critical role in managing, monitoring, and optimizing cloud infrastructure, ensuring that applications and services run efficiently, securely and cost-effectively. 

    With a deep understanding of cloud technologies, platforms and best practices, they collaborate with various teams to align cloud solutions with business objectives and end-user needs. As cloud adoption continues to grow across industries, the demand for skilled cloud operations engineers is expected to rise, making it a promising career path for those seeking opportunities in the ever-evolving world of cloud computing.

  • FinOps Foundation’s FOCUS Aims to Standardize Cloud Billing

    FinOps Foundation’s FOCUS Aims to Standardize Cloud Billing

    Cloud spending is on the rise. As organizations shift more applications to the cloud, there is a growing need to optimize workloads and reduce spending when possible. But a lack of visibility into complex cloud bills often impedes these goals. This is compounded by a lack of data standardization across the multi-cloud landscape, with cloud vendors producing lengthy billing reports in various formats. With things so decentralized, FinOps practitioners often struggle to make sense of it all.

    FinOps Open Cost & Usage Specification (FOCUS) is set to change this status quo. FOCUS, an initiative sponsored by the FinOps Foundation, a Linux Foundation technical project, aims to create a standard, vendor-neutral specification for cloud cost, usage and billing data. A common schema for cloud billing should help unify cost data across cloud vendors and greatly aid FinOps objectives in areas like allocation, chargeback, budgeting and forecasting.

    I recently met with Udam Dewaraja, FOCUS lead at FinOps Foundation, to understand the context that led to creating the specification. Below, we’ll dive a little deeper into FOCUS, examining the details of the proposed specification and considering what benefits it will bring to cloud consumers and cloud service providers (CSPs).

    Why the Cloud Needs a Common Billing Schema

    The complexities of today’s cloud services can often create headaches around cloud billing. For instance, AWS alone has hundreds of services and potentially thousands of SKUs. Cloud service providers each have their own data formats and terminologies, which are reflected differently in detailed bills. And with the granularity of cloud metering, enterprises could be staring at hundreds of millions of rows of cost data. Now, extrapolate that across multiple cloud service providers, SaaS and APIs, and you can see the dilemma that the modern CFO faces.

    “There are many nuances that FinOps practitioners need to understand,” said Dewaraja. “And a lack of a common definition can be a scalability challenge as you add more things.”

    It’s relatively easy to spend money on more and more cloud-based consumption models. As Dewaraja said, “Everyone has the company credit card.” Of course, you can use tools like AWS Cost Explorer, Cloudability or Datadog to garner some insights to reduce spending. But to really get the fuller picture, you need to rationalize cost metrics across many areas. Plus, in 2023, The State of FinOps report found the widest variety of cloud cost management tools in use in the history of the report, indicating tooling sprawl.

    Overview of FinOps Foundation

    First, for those unfamiliar, the FinOps Foundation is a Linux Foundation-supported group that aims to help practitioners understand the discipline of FinOps. Representing a community of over ten thousand FinOps practitioners, the FinOps Foundation routinely shares best practices and oversees a FinOps framework. “FinOps is more of a cultural practice that should be adopted,” explained Dewaraja. As such, the initiative aims to oversee training, meetups and events and encourage executive buy-in around the FinOps concept.

    Understanding the FOCUS Specification

    So, what exactly does FOCUS entail? Well, FOCUS is an open source royalty-free specification for cloud billing data. It will provide standard dimensions, attributes, and terminologies for cloud billing data sets. To formalize the specification, the project contributors are actively considering the essential FinOps capabilities and what requirements these cloud bill data sets have in common.

    The specification itself has yet to be officially released and is still being formalized at the time of writing. Microsoft and Google have joined the initiative, as well as numerous cloud finance and cost analysis vendors. And while the group waits for additional CSPs to join, the community is building out open source validators and converters, said Dewaraja.

    Benefits of a Standard Cloud Billing Spec

    Cost management solutions are already available, but what’s not readily available is a method to make business decisions based on broader richer datasets, explained Dewaraja. This is where standardized cloud billing can greatly aid efforts by creating a more holistic picture.

    Specifically, this can aid objectives like allocation, chargeback, budgeting, and forecasting. Doing so is often an incremental process, explained Dewaraja—as you gather insights into data, you iteratively use this visibility to make improvements over time.

    Some other benefits of adopting FOCUS will likely be:

    Putting finance in engineering terms. In the cloud, finance is quite dynamic. And interestingly, the FinOps discipline is becoming engineering’s responsibility as part of the shift left approach, said Dewaraja. Engineers now need to understand this problem and how to architect their solutions cost-effectively. As such, bringing specifications into the mix will help put financial responsibility squarely on the shoulders of engineering terms.

    Bringing accountability to cloud consumption. A standard specification should empower essential FinOps capabilities that rely on usage segmentation like internal cost reporting, showback and chargeback. Because if invoices are just one big pie, there is no accountability for individual departments, and you can’t drive the behaviors you need, said Dewaraja.

    Win-win for practitioners and cloud vendors. With FOCUS, cloud providers will drive more trust in cost data, providing more visibility. And corporations using the cloud can make their adoption more efficient, likely influencing them to bring on more workloads. “Driving trust and understanding of this cloud data is mutually beneficial,” Dewaraja said.

    FOCUS Set to Improve FinOps

    Dewaraja, who previously spearheaded FinOps practices at Citigroup, knows all too well the strains of managing disparate cloud billing data. He oversaw a broad, dynamic cloud-native platform of distributed cloud and SaaS and sought to embrace platform engineering principles. In this environment, you need a holistic cost picture that associates various expenses to make the right calls, he said. “You really see success if you can plug this into the workflows of those developers and into business processes themselves,” he said.

    Theoretically, FOCUS could deliver on these goals. But although the project is gathering momentum, it’s still in an early stage. For the standard to gain a foothold, it will require greater awareness and more contribution from other CSPs. The specification is community-driven but is being led from the practitioner down, adopting corporate contribution licenses instead of individual developer contributions. Thus, cloud vendors and cost management vendors will likely lead the initial charge.

    FOCUS looks great in the long run but will require a little bit of work upfront to get folks on board. That said, it should be on the radar for 2024 roadmaps, said Dewaraja. To learn more about FOCUS and how to contribute, software providers should check out the FinOps Foundation materials.

  • Microsoft Adds Slew of Developer Tools to Azure

    Microsoft Adds Slew of Developer Tools to Azure

    At its Build 2023 conference this week, Microsoft announced general availability of an Azure Deployment Environment service that provides developers with a portal through which they can employ infrastructure-as-a-code (IaC) templates based on Terraform or Azure Resource Management files.

    In addition, Microsoft has enhanced Microsoft Dev Box, a workstation environment for developers that run Azure, to include a portal through which they can manage multiple environments.

    Scheduled to be generally available in July, Microsoft Dev Box is also previewing a capability that makes it possible to customize environments using configuration-as-code files stored in a Git repository. Microsoft is also previewing tighter integration between Dev Box and its Visual Studio integrated development environment (IDE) and Windows to further customize developers’ environments.

    Scott Guthrie, executive vice president of the cloud and artificial intelligence (AI) group at Microsoft, told conference attendees these and other tools for building cloud-native applications, along with GitHub Actions, make it simpler than ever to build applications in an Azure cloud service that includes an integrated continuous integration/continuous delivery (CI/CD) platform.

    Microsoft is also previewing GitHub Advanced Security for Azure DevOps that natively embeds automated security checks using scanning tools into the Azure DevOps platform.

    Other additions to the Azure environment include tools for centralizing the management of application programming interfaces (APIs) using an Azure API Center service that is available in preview.

    The Azure Event Grid has also been updated to enable it to use HTTP to pull delivery of discrete events in addition to pushing them.

    Finally, Microsoft is also previewing generative artificial intelligence (AI) capabilities to make it simpler to manage cloud costs by, for example, using prompts to summarize invoices or breakdown costs.

    Microsoft, along with its GitHub subsidiary, is at the forefront of applying generative AI to application development using tools like GitHub Copilot that developers can use to write better code faster. That copilot capability is now being disseminated across the portfolio of applications and services that Microsoft provides.

    It’s not yet clear what the downstream impact of such tools are just yet, but the overall quality of the applications being built should improve as, for example, fewer mistakes are made that can lead to vulnerabilities.

    Obviously, the way application software is built, constructed and secured is about to change in the generative AI era. Thanks to its alliance with OpenAI, Microsoft has a significant head start on rivals building generative AI platforms that can similarly be applied to improve developer productivity.

    In the meantime, DevOps teams should assume the volume of code simultaneously moving through pipelines is about to dramatically increase as developers become more productive. There will inevitably be more opportunities to apply multiple forms of AI to reduce any DevOps bottlenecks that might arise, but it’s clear most of the immediate benefits are being provided to developers.

    One way or another, however, the overall rate at which applications are built and deployed is about to change drastically. The issue is understanding how those advances will impact downstream DevOps workflows that were constructed for a different era of application development.

  • Quick Guide to Azure Storage Pricing

    Quick Guide to Azure Storage Pricing

    Azure Storage is a cloud-based storage service from Microsoft Azure. It provides a scalable, secure and highly available data storage solution for businesses of any size.

    The platform includes various services such as Azure Blob, which is a service for storing large amounts of unstructured data such as documents, images, videos and backups. Azure Files is a fully managed file share service for cloud-based applications. Azure Tables is a NoSQL key-value store used to store semi-structured data. Azure Queues is a messaging service used to store and process messages between applications or services.

    Finally, Azure Disks is a service that allows you to store and manage data in virtual hard disks used in virtual machines. With Azure Storage, businesses can easily manage their data, control access and scale their storage as per their needs without worrying about hardware or infrastructure management.

    How does Azure charge for storage space (bandwidth, capacity, number of transactions and quantity of transferred data)? What are the usage factors impacting Azure Storage fees: Region, account, access tier, replicas, egress?

    How Azure Storage Pricing Works

    Azure pricing for its storage services is based on several factors including storage capacity, data transfers and transaction counts. The cost of storage varies depending on the region and access tier selected, as well as the type of storage being used.

    For Azure Blob Storage, customers are charged for storage capacity, data egress (outbound data transfers) and the number of transactions (read, write and list operations).

    Azure Table Storage charges are based on the number of table entities stored and the amount of data transferred. Azure Queues are charged per transaction and data transfer, while Azure Disks are billed based on the provisioned capacity and the number of transactions.

    Several usage factors can impact Azure Storage fees, including the region where the data is stored, the storage account type, the access tier selected, the number of replicas and the amount of data transferred out of the storage account. 

    Overview of Azure Storage Pricing

    Blob Storage

    Azure Blob Storage pricing varies depending on the storage tier, data volume and whether customers opt for pay-as-you-go or reserved capacity.

    For pay-as-you-go pricing, Premium storage is charged at a flat rate of $0.015 per GB/month, while Cool and Archive storage are billed at $0.01 and $0.0099 per GB/month, respectively, regardless of data volume. Hot storage costs $0.018 per GB /month for the first 50 TB with the rate gradually decreasing with higher usage. For instance, customers who consume more than 500 TB/month pay $0.0166 per GB.

    Customers can also reserve blob storage in 100-TB or 1-PB blocks for one or three years. The prices for reserved storage vary by tier. A three-year Archive storage reservation costs $84 per 100 TB or $810 per PB/month, while Hot storage costs $1,244 per 100 TB/month for three years. For one-year reservations, Archive storage costs $91 per 100 TB or $883 per PB/month, and Hot storage costs $1,545 per 100 TB/month. For a one-year reservation, the cost of Hot storage is $15,050 per PB/month, and $11,963 per month for three years.

    Files

    Microsoft charges customers for Azure Files depending on the storage tier and whether they reserve storage capacity. Pay-as-you-go pricing ranges from $0.015 per GB/month for Cool storage to $0.016 per provisioned GiB/month for Premium file storage.

    Customers can commit to reserved storage capacity in increments of 10 TiB or 100 TiB for one or three years. A three-year reservation of 10 TiB Cool storage costs $101.39 per month, while one year costs $125.92 per month. However, the price increases significantly for the Premium offerings, with a three-year reservation of 10 TiB costing $1,081.34 per month and a one-year reservation costing $1,343.50 per month. 100 TiB reservations range from $983/month in the Cool tier for three years to $12,779.50/month in the Premium tier for one year.

    Nevertheless, there are other fees that add to the total cost such as read and write transactions, snapshots, metadata storage and early file deletion fees. These ancillary fees can impact the overall cost of using Azure Files and customers should account for them when estimating their expenses.

    Tables

    Azure Tables pricing is available in various tiers, including locally redundant storage (LRS), geo-redundant storage (GRS), read-access geo-redundant storage (RA-GRS), zone-redundant storage and geo-zone-redundant storage. 

    Customers are charged between $0.045 per GB/month for LRS and $0.01265 per GB/month for geozone-redundant read-only storage. Additionally, Microsoft charges a $0.0036 fee for each set of 10,000 transactions made on Azure Tables.

    Queues

    Compared to Blob and File Storage, the pricing model for Azure Queues is more straightforward. Customers are charged 4.5 cents to $0.075 cents per GB/month, depending on whether they choose local redundant storage (LRS) or read-access geozone-redundant storage (RA-GRS). Additionally, there is a $0.004 fee for every set of 10,000 operations in Class 1 and 2.

    Managed Disks

    Azure Managed Disks have numerous pricing options that can be overwhelming. The cost of using Managed Disks depends on several factors, including the disk size, performance, reservation period, capacity and disk type.

    The cheapest option for Managed Disks is the P1 disk, which costs $0.6 per month for 4 GiB of capacity in addition to $0.03 per mount/month. On the other hand, the P80 disk is the most expensive option, as it has a 32 TiB capacity costing $3,276.80 per month, in addition to $219/mount. However, the cost of a P80 disk can be reduced to $3,113/month (plus the $219/mount fee) with a reservation of one year.

    Azure Storage Vs. AWS Storage Pricing

    Here is a comparison of the pricing for similar storage services offered by Azure and AWS. As part of an Azure or AWS cost optimization strategy, you might consider combining services from both providers.

    Block Storage

    Amazon EBS is Amazon EC2’s basic storage device, with a price of $0.045 per GB for HDD and $0.10 per GB for SDD, with higher IOPS resulting in higher prices. The free tier includes 30 GB of SDD.

    For comparable regions in the U.S., Azure charges $0.05 per GB for HDD and $19.71 for 128 GB of SSD per month.

    Azure Managed Disks offer greater flexibility and easier backup than Azure Virtual Disks or Amazon EBS, with adjustable volume sizes and available IOPS. Managed Disks pricing starts at $19/month for a 128 GB disk, providing 500 IOPS.

    Object Storage

    Amazon S3 and Azure Blob Storage are both scalable services for storing objects (unstructured data) where data is stored together and cannot be changed in parts. Prices vary depending on the region, but for a comparable region in the U.S., Amazon S3 storage costs $0.023 per GB/month for the first 50 TB in addition to network usage fees. The free tier offers 5 GB of storage, while archive storage (Glacier) is $0.004 per GB/month.

    Azure Blob Storage starts at $0.0184 per GB/month for hot storage, although prices decrease to $0.01 per GB per month for cool storage and $0.002 for archive storage. Prices for both services increase for greater redundancy. 

    File Storage

    Amazon Elastic File System (EFS) is an equivalent service to Azure File storage and allows users to mount files to Amazon EC2 servers. It uses NFS V4.0/4.2 to mount files and is not available in all AWS regions (currently, four are supported), requiring a Direct Connect line or VPN for on-premises mounting.

    For a comparable U.S. region, Amazon EFS costs $0.30 per GB/month and the free tier offers 5GB on sign-up. Azure File Storage prices start at $0.06 per GB/month for local redundant storage, with prices increasing for more distributed redundancy setups. It is worth noting that both Azure and AWS charge additional fees for data access.

    Conclusion

    In conclusion, Azure Storage offers a range of services for storing and managing data in the cloud, each with its own pricing structure. While the pricing varies based on several factors including storage capacity, data transfers and transaction counts, it is essential to consider other usage factors, such as region, account, access tier, replicas and egress. 

    Customers should review these factors carefully and estimate their expenses to select the right storage service that meets their needs and budget. By doing so, they can leverage Azure Storage’s flexibility and scalability to store and manage their data efficiently in the cloud.

  • Using Microsoft Azure as a Backup Repository

    Using Microsoft Azure as a Backup Repository

    Every year, the amount of data that needs to be backed up increases. There are two ways to store backups – using on-premises facilities or trusting cloud-based solutions. Having an on-premises storage infrastructure can be costly for a company. This is because of the high prices for storage equipment and the expenses to maintain this equipment, e.g., hiring a storage engineer. Depending on a company’s rules, Microsoft Azure could be an alternative to traditional ways of keeping data. If the amount of data is relatively small (less than 4 TB), there is no problem with making use of this solution; you can either use Azure Blob as a CIFS or NFS storage, or you can create your own virtual machine by adding a disk to store your data and then configuring it as a file server. However, this blog post focuses on cases where space limits are more than 4 TB.

    Microsoft Azure allows you to create VMs and add disks with a maximum capacity of 4 TB for each disk. It is suitable for application servers but very often doesn’t work if you require a big file or backup server.

    However, we can exceed this limit by using Windows software RAID array. The idea is to use RAID5 as the software RAID which contains arrays starting from three disks.

    Why RAID5? Because it has a good read speed limit. It is also economical to use X disks, where X starts at three and you only lose one disk space for the total RAID result space. Furthermore, this should give you redundancy if one of your disks were to fail. However, if we build a RAID array in Azure, we already have the virtual disks as blocks of our array and they are protected by Microsoft Azure Cloud. Thus, the chance of failure in our RAID array (which is based on disks that are already in the RAID array) falls significantly.

    Here’s how to create an Azure-based backup and replication repository server.

    1. Log in to your Microsoft Azure account or register a new account at https://azure.microsoft.com/en-us/

    2. Create a new Windows Virtual Machine:

    In this example, we choose Windows 2008 R2 SP1, since we don’t require any advanced services, just software RAID5.

    3. Type in your VM name, username and the password according to MS Azure rules of complexity.

    4. Choose your VM size. Please note that the VM type depends on the number of attached disks. Thus, if you require more than 8 disks (over 24 GB in RAID 5), the only option is to choose DS2_V2 Standard:

    5. The settings for available storage are some of the main options for our future repository:

    At this point, you have created a new VM; now it is time to add disks. On the Disks page, you can add as many disks as you need for your future disk array. In our case, we will build software RAID5, which will require seven disks in total.

    We need to have our SMB share available from the internet, so we have to configure the Azure Firewall settings.

    In the Networking section, click the Add inbound port rule button and create a new rule allowing you to make port 445 (SMB) accessible to everyone (for testing). Alternatively, you can only make the port accessible to the backup software instance’s external IP as in the Source field (for production).

    The next step is to connect to the VM via RDP using the IP and credentials you created during the VM creation process.

    Next, you need to create a RAID5 volume. In Storage Manager, you`ll see all 4 TB disks you attached. Right click Disk Management > All Tasks > New Raid 5 Volume:

    Select all available 4 TB disks to add them to your array and move them to the right column:

    Follow the standard wizard; choose a letter for the disk and perform a quick format by NTFS.

    Next, create an SMB share on a new logical disk and add write permissions to the future repository.

    At this stage, you can use your entire 24 TB volume as a remote repository in your backup system.

    If you use the NAKIVO backup to Azure solution, just add a new repository using the server’s IP path to your share as well as the credentials you set for the SMB share:

    Then, create a new job using the new repository as the backup target and run it.

    Conclusion

    The amount of data organizations need to back up is growing exponentially. Therefore, it is important to store that data in the most economical way. Using a data backup system like NAKIVO and RAID5, you can make sure your VMs are reliably protected and that there is enough space to store them in Azure.

  • Grafana Labs Reins in Cost of Storing Observability Data

    Grafana Labs Reins in Cost of Storing Observability Data

    Grafana Labs today made available an ability to customize what observability data is collected via its managed cloud service. This capability can help reduce storage costs in addition to enhancing the dashboard Grafana provides to make it simpler to track metrics usage.

    Wayne Jin, vice president of product marketing for Grafana Labs, said both capabilities would make it easier for DevOps teams to track and ultimately reduce the cost of storing the time-series data collected via its cloud platform.

    As more DevOps teams use an observability platform to continuously collect metrics like traces and log data, they are discovering that the cost of storing all that high-cardinality data can easily spin out of control.

    Grafana Labs is now making available an Adaptive Metrics capability that enables DevOps teams to fine-tune what data is collected and stored. However, in the event of an issue, DevOps teams can also turn back on all data the platform can collect to aid an investigation of the root cause of disruption, noted Jin.

    The Cardinality Management dashboards Grafana Labs made available last year for paid tiers of its cloud service are now being made available to users of the free tier of service as well, Jin said. This will further help reduce storage costs by making it easier to identify metrics that are being collected but not used by anyone on the DevOps team, he added.

    The Adaptive Metrics aggregation engine can then be applied to transform these metrics at ingestion into lower cardinality versions. Unused or partially used labels are stripped from incoming metrics, reducing the total count of time-series data collected. Adaptive Metrics also recommends aggregations based on an organization’s historic usage patterns, and DevOps teams can choose which aggregation rules to apply. Dashboards, alerts and historic queries are guaranteed to continue to work as they did before aggregation, with no rewrites needed.

    Based on results reported by early users, Grafana Labs reported that Grafana Cloud Adaptive Metrics could eliminate an estimated 20-50% of the time-series data collected with no perceived impact on observability.

    The Grafana Cloud service relies on an instance of open source Grafana Mimir to store data collected in a format originally defined for the open source Prometheus monitoring tool. While Prometheus is widely employed in Kubernetes environments, it’s also starting to gain traction as a monitoring platform in legacy monolithic environments. The amount of data being collected via that platform over time is steadily increasing. The Grafana Labs approach makes it possible for DevOps teams to reduce that data whenever there is no usage that needs to be investigated, for example.

    It’s still early days as far as adoption of observability is concerned, but in uncertain economic times, there’s generally a lot more sensitivity to the total cost of IT. Many organizations are especially focused on reducing the cost of storing data in the cloud to rein in monthly spending on cloud services. Regardless of the motivation, collecting and storing data that isn’t needed is never the best use of cloud storage resources that, from a cost perspective, are just as finite in the cloud.