devopsy.kinsta.cloud

Category: Infrastructure/Networking

  • Cloud Drift Detection With Policy-as-Code

    Cloud Drift Detection With Policy-as-Code

    Cloud configurations can change and change often. Introducing new technologies, releasing new features and supporting new business requirements entail a constant flow of configuration changes in web application development.

    However, drift occurs regardless of how well-designed your IaC implementation is. The term “drift” is used to denote a state in which the actual state of your infrastructure deviates from the configuration.

    This article examines cloud drift detection, why it occurs and how to remediate it.

    Understanding the Problem

    Cloud configurations are prone to change and can change frequently. Businesses often use infrastructure-as-code (IaC) to manage cloud provisioning changes. IaC makes it easier and more reliable for organizations to manage and facilitate changes to the cloud deployment process.

    However, inconsistencies in your IaC deployment process can lead to uncertainty about how and where your resources are provisioned, controlled and protected, resulting in lower productivity. No matter how robust your IaC implementation, drift can creep in.

    Changes to infrastructure that are made independently of the code responsible for provisioning can lead to considerable drift, and if not adequately monitored, can lead to significant security concerns.

    What is Drift?

    In the context of infrastructure and application management, drift refers to the system state in which the actual configuration and state of the system deviate from the intended or expected configuration and state.

    Drift can be caused by software updates, manual changes and incorrect configurations. Drift management is an essential aspect of infrastructure and application management. By detecting and correcting drift, organizations can ensure that their systems remain stable, secure and compliant with their policies and industry standards.

    Configuration Drift: Definition, Causes and Examples

    Configuration drift can creep in over time despite consistently building and configuring your servers. It refers to the state of the system when configuration changes are not in sync with the value previously set.

    For example, configuration drift can occur when you are yet to document the changes made to the production environment. This results in the production and staging environments getting out of sync.

    Some of the other causes of configuration drift are configuration changes because of applying patches and updating network equipment, the addition of new resources to the network and lack of clarity about the desired state of the system.

    You can leverage configuration drift management tools such as Netwrix and Aqua to detect configuration drift.

    Infrastructure Drift: Definition, Causes and Examples

    Infrastructure drift refers to a phenomenon in which the actual state of the infrastructure differs from its desired state because the defined configuration, settings or properties of the infrastructure and the actual state of the provisioned resources differ.

    Infrastructure drift may result from manual changes, updates outside of configuration management processes, conflicting IaC code, changes in network configurations, changes due to security settings and configuration differences during system upgrades and maintenance.

    To detect infrastructure drift, you can take advantage of infrastructure drift detection tools such as Terraform, CloudQuery and driftctl.

    Infrastructure Drift and Configuration Drift: How do they compare?

    Both configuration and infrastructure drift can lead to security vulnerabilities, compliance issues and operational challenges. Although infrastructure drift and configuration drift are related, they aren’t the same.

    Infrastructure drift refers to inconsistencies or discrepancies between the intended or desired state of the infrastructure and the actual state of the resources provided. Configuration drift refers to inconsistencies or discrepancies between the current configuration of a component or system and the intended or desired configuration.

    Infrastructure drift encompasses everything from physical components to virtual components to networks, storage and computing resources. Configuration drift, on the other hand, focuses on individual components or systems within the infrastructure and their configuration settings.

    What is Drift Detection?

    Drift detection in the context of cloud infrastructures typically compares the actual state of resources deployed in the cloud environment with the defined state described in IaC templates.

    Typically, this is automated by tools, scripts or services that analyze and compare configurations, resource states or event logs. These tools are adept at issuing notifications or alerts when discrepancies are detected.

    By detecting discrepancies, organizations can identify and resolve inconsistencies, security vulnerabilities and compliance violations. Consequently, the risks related to security breaches are reduced and remediation actions become easier to implement.

    Why Policy-as-Code for Cloud Drift Detection?

    Policy-as-code provides several benefits when used for cloud drift detection:

    • Early detection of drift: Policy-as-code helps detect drift early detection by comparing the current state of the cloud environment with the desired state defined in the policies. This helps identify problems and solve issues before they escalate to a catastrophic level.
    • Automated enforcement: Policy-as-code enables automated enforcement of policies, ensuring that the cloud environment complies with the defined rules and requirements. This helps reduce the potential risks associated with human errors.
    • Faster remediation: Policy-as-code enables faster remediation by automatically detecting drift and taking the necessary remediation actions such as rolling back changes, updating configurations or notifying relevant teams. This facilitates faster response times and reduces the risks associated with security breaches and downtime.
    • Enforce best practices, policies and guidelines: With policy-as-code, best practices, policies and security guidelines can be implemented automatically, reducing security and compliance risks.

    Steps for Drift Detection Using Policy-as-Code

    Here are the steps that you need to follow to detect and fix drift in your infrastructure:

    1. Define your policies: You should first define the policies that should be used to describe the desired state or the baseline of your cloud environment and cover security and compliance aspects of your cloud environment.
    2. Codify and deploy policies: Then these policies should be codified and stored in the version control system. The next step should be to deploy the policies to your cloud environment using a policy engine.
    3. Identify configuration changes: Once your policies have been deployed, the next step should be to identify any new resources that may introduce risks and compare them to the baseline you’ve defined.
    4. Evaluate and remediate: You should evaluate the state of the environment against the defined policies to detect any drift. As soon as any drift is detected, you should take corrective actions to remediate the issues by rolling back changes, updating configurations or notifying the relevant team members.
    5. Review policies: To ensure that the policies are up-to-date and relevant, it is imperative that you review them often and make changes to policies as needed.
    6. Update and deploy: The last step should be to update the code files and then deploy the solution yet again.

    Build a Successful Drift Detection Strategy

    Maintaining your infrastructure at the desired state, ensuring compliance and mitigating risks associated with configuration drifts requires a successful drift detection strategy. Here are key strategies you can adopt to build a successful drift detection strategy.

    Understand When Drift Becomes a Risk

    Drift can profoundly impact a system’s stability, security and compliance. When your server configuration deviates from its intended configuration, it might stop working or become vulnerable to security threats.

    Most importantly, this might be a significant threat to security if you don’t have proper monitoring in place. To prevent infrastructure drift and update your security tools, it is important to evaluate and communicate your adoption of IaC.

    Detecting Drift: Detecting drift Between IaC and the Cloud

    To mitigate the risks associated with drift, configurations that have been changed in real-time must be identified and reset. You can achieve this programmatically and continuously by comparing configuration changes between IaaS and IaC. Programmatically identifying the configuration changes, i.e., detecting the changes to the configuration using code, is the best strategy to reduce drift-related risks.

    From Drift Detection to Drift Remediation: Responding to drift

    By detecting drift, you’ve won half the battle–you should now devise strategies to remediate it. By keeping an eye on configuration drift and responding to it promptly, you ensure the consistency and reliability of your infrastructure, minimize security risks and keep your system in a consistent state.

    Figure 1: Detect and Remediate Drift in the Cloud

    It’s important to have effective configuration drift response procedures in place to ensure the stability and security of your environment and also keep it effective for change management and control. You can respond to drift in the following ways:

    • Develop mechanisms to detect and report configuration drift
    • Understand the underlying causes of drift
    • Documenting the drift, including affected resources, configuration deviations and impacts
    • Resolve the configuration drift and return the system to the desired state
    • Following change management processes and obtaining approvals prior to implementing changes
    • Monitor the system to ensure drift has been resolved and the configuration remains stable

    Key Strategies for Detecting Drifts Between IaC and the Cloud

    There are a few strategies that can be adopted for detecting drift.

    Detect Configuration Changes That may Introduce Risks

    To identify configuration changes that can introduce risks, you should monitor and analyze changes proactively. Here are some of the key strategies to do so:

    • Create a baseline configuration for your system.
    • Review the system logs regularly for any suspicious or unexpected configuration changes.
    • To capture detailed information about configuration changes, comprehensive logging is helpful.
    • Implement a change management process for managing configuration changes.
    • Take advantage of automated configuration management tools such as Chef, Puppet, Ansible, etc.
    • Leverage threat intelligence to ensure that you are always up to speed on the most recent vulnerabilities, security threats and attack trends.
    • Conduct training to increase awareness about configuration changes’ potential consequences and risks.

    Establish Baselines and Safely Auto-Remediate Drift

    Establishing baselines and safely auto-remediating drift involves setting a reference point for the desired state of your infrastructure and automatically adjusting any detected drift back into compliance with the baseline.

    You can maintain the desired state of your infrastructure, reduce manual effort and quickly resolve configuration drift by establishing baselines and implementing security processes for automatic remediation.

    This minimizes the impact of drift on your systems while improving your environment’s security, compliance and stability.

    Track all Configuration Changes in Your Cloud Environment

    Tracking all configuration changes to your cloud environment is critical to maintaining visibility, accountability and security. It can help you improve visibility, detect unauthorized or unintended changes and maintain compliance and accountability.

    This way, you can proactively manage your cloud infrastructure, identify potential security risks and ensure its integrity and stability.

    Below are the key strategies for tracking cloud configuration changes at a glance:

    • Ensure detailed logging for all cloud components, including infrastructure, applications and security
    • Track API calls and activities within your cloud environment with cloud monitoring and auditing
    • Set up alerts and notifications for critical deviations from established baselines
    • Implement configuration changes through formal change management processes
    • Manage cloud configuration with tools such as AWS Config and Azure Automation

    Managing Drift in Production

    To effectively manage drift in production, it is crucial to consistently observe the configuration and state of production systems, keeping them in line with the policies that outline the desired state.

    Here are a few guidelines to help you reduce production drift:

    • Specify the required state: Define the intended state of the production environment as a set of policies using declarative compliance tools.
    • Monitor continuously: To determine any deviations, use a monitoring tool to monitor the production environment or compare the system’s present state to the desired state to identify any deviations.
    • Identify the root cause: If drift is detected, it is crucial to identify its root cause. You can do this by examining the logs, analyzing system performance or observing the configuration alterations over time.
    • Rectify the drift: After determining the issue’s root, you should address it to align its compliance with the intended state. To do this, you may need to examine logs, examine configuration changes, and analyze system performance.
    • Automate the remediation process: To ensure that drift is automatically and quickly rectified, you should automate the remediation process. To achieve this, you can leverage a declarative compliance tool that can help you automatically implement the changes to bring the system back into conformity with the intended state.
    • Review and update policies: You should review and revise your policies to ensure they are still relevant and useful and stay updated. To achieve this, you may need to adjust the policies to reflect new compliance requirements or shift your business requirements.

    Summary

    By identifying drift in cloud environments, organizations can use policy-as-code to ensure that their cloud infrastructure is safe, compliant and secure. In this way, they can ensure compliance with industry best practices, regulatory standards and internal policies. This reduces the risks associated with configuration deviations and enables enterprises to maintain a secure and reliable cloud environment.

  • US DoJ Makes PyPI Give Up User Data ¦ Tape Storage: Not Dead

    US DoJ Makes PyPI Give Up User Data ¦ Tape Storage: Not Dead

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: PyPI complies with a “string of subpoenas,” and LTO continues to grow, despite predictions of its demise. (more…)

  • FinOps Foundation’s FOCUS Aims to Standardize Cloud Billing

    FinOps Foundation’s FOCUS Aims to Standardize Cloud Billing

    Cloud spending is on the rise. As organizations shift more applications to the cloud, there is a growing need to optimize workloads and reduce spending when possible. But a lack of visibility into complex cloud bills often impedes these goals. This is compounded by a lack of data standardization across the multi-cloud landscape, with cloud vendors producing lengthy billing reports in various formats. With things so decentralized, FinOps practitioners often struggle to make sense of it all.

    FinOps Open Cost & Usage Specification (FOCUS) is set to change this status quo. FOCUS, an initiative sponsored by the FinOps Foundation, a Linux Foundation technical project, aims to create a standard, vendor-neutral specification for cloud cost, usage and billing data. A common schema for cloud billing should help unify cost data across cloud vendors and greatly aid FinOps objectives in areas like allocation, chargeback, budgeting and forecasting.

    I recently met with Udam Dewaraja, FOCUS lead at FinOps Foundation, to understand the context that led to creating the specification. Below, we’ll dive a little deeper into FOCUS, examining the details of the proposed specification and considering what benefits it will bring to cloud consumers and cloud service providers (CSPs).

    Why the Cloud Needs a Common Billing Schema

    The complexities of today’s cloud services can often create headaches around cloud billing. For instance, AWS alone has hundreds of services and potentially thousands of SKUs. Cloud service providers each have their own data formats and terminologies, which are reflected differently in detailed bills. And with the granularity of cloud metering, enterprises could be staring at hundreds of millions of rows of cost data. Now, extrapolate that across multiple cloud service providers, SaaS and APIs, and you can see the dilemma that the modern CFO faces.

    “There are many nuances that FinOps practitioners need to understand,” said Dewaraja. “And a lack of a common definition can be a scalability challenge as you add more things.”

    It’s relatively easy to spend money on more and more cloud-based consumption models. As Dewaraja said, “Everyone has the company credit card.” Of course, you can use tools like AWS Cost Explorer, Cloudability or Datadog to garner some insights to reduce spending. But to really get the fuller picture, you need to rationalize cost metrics across many areas. Plus, in 2023, The State of FinOps report found the widest variety of cloud cost management tools in use in the history of the report, indicating tooling sprawl.

    Overview of FinOps Foundation

    First, for those unfamiliar, the FinOps Foundation is a Linux Foundation-supported group that aims to help practitioners understand the discipline of FinOps. Representing a community of over ten thousand FinOps practitioners, the FinOps Foundation routinely shares best practices and oversees a FinOps framework. “FinOps is more of a cultural practice that should be adopted,” explained Dewaraja. As such, the initiative aims to oversee training, meetups and events and encourage executive buy-in around the FinOps concept.

    Understanding the FOCUS Specification

    So, what exactly does FOCUS entail? Well, FOCUS is an open source royalty-free specification for cloud billing data. It will provide standard dimensions, attributes, and terminologies for cloud billing data sets. To formalize the specification, the project contributors are actively considering the essential FinOps capabilities and what requirements these cloud bill data sets have in common.

    The specification itself has yet to be officially released and is still being formalized at the time of writing. Microsoft and Google have joined the initiative, as well as numerous cloud finance and cost analysis vendors. And while the group waits for additional CSPs to join, the community is building out open source validators and converters, said Dewaraja.

    Benefits of a Standard Cloud Billing Spec

    Cost management solutions are already available, but what’s not readily available is a method to make business decisions based on broader richer datasets, explained Dewaraja. This is where standardized cloud billing can greatly aid efforts by creating a more holistic picture.

    Specifically, this can aid objectives like allocation, chargeback, budgeting, and forecasting. Doing so is often an incremental process, explained Dewaraja—as you gather insights into data, you iteratively use this visibility to make improvements over time.

    Some other benefits of adopting FOCUS will likely be:

    Putting finance in engineering terms. In the cloud, finance is quite dynamic. And interestingly, the FinOps discipline is becoming engineering’s responsibility as part of the shift left approach, said Dewaraja. Engineers now need to understand this problem and how to architect their solutions cost-effectively. As such, bringing specifications into the mix will help put financial responsibility squarely on the shoulders of engineering terms.

    Bringing accountability to cloud consumption. A standard specification should empower essential FinOps capabilities that rely on usage segmentation like internal cost reporting, showback and chargeback. Because if invoices are just one big pie, there is no accountability for individual departments, and you can’t drive the behaviors you need, said Dewaraja.

    Win-win for practitioners and cloud vendors. With FOCUS, cloud providers will drive more trust in cost data, providing more visibility. And corporations using the cloud can make their adoption more efficient, likely influencing them to bring on more workloads. “Driving trust and understanding of this cloud data is mutually beneficial,” Dewaraja said.

    FOCUS Set to Improve FinOps

    Dewaraja, who previously spearheaded FinOps practices at Citigroup, knows all too well the strains of managing disparate cloud billing data. He oversaw a broad, dynamic cloud-native platform of distributed cloud and SaaS and sought to embrace platform engineering principles. In this environment, you need a holistic cost picture that associates various expenses to make the right calls, he said. “You really see success if you can plug this into the workflows of those developers and into business processes themselves,” he said.

    Theoretically, FOCUS could deliver on these goals. But although the project is gathering momentum, it’s still in an early stage. For the standard to gain a foothold, it will require greater awareness and more contribution from other CSPs. The specification is community-driven but is being led from the practitioner down, adopting corporate contribution licenses instead of individual developer contributions. Thus, cloud vendors and cost management vendors will likely lead the initial charge.

    FOCUS looks great in the long run but will require a little bit of work upfront to get folks on board. That said, it should be on the radar for 2024 roadmaps, said Dewaraja. To learn more about FOCUS and how to contribute, software providers should check out the FinOps Foundation materials.

  • Microsoft Adds Slew of Developer Tools to Azure

    Microsoft Adds Slew of Developer Tools to Azure

    At its Build 2023 conference this week, Microsoft announced general availability of an Azure Deployment Environment service that provides developers with a portal through which they can employ infrastructure-as-a-code (IaC) templates based on Terraform or Azure Resource Management files.

    In addition, Microsoft has enhanced Microsoft Dev Box, a workstation environment for developers that run Azure, to include a portal through which they can manage multiple environments.

    Scheduled to be generally available in July, Microsoft Dev Box is also previewing a capability that makes it possible to customize environments using configuration-as-code files stored in a Git repository. Microsoft is also previewing tighter integration between Dev Box and its Visual Studio integrated development environment (IDE) and Windows to further customize developers’ environments.

    Scott Guthrie, executive vice president of the cloud and artificial intelligence (AI) group at Microsoft, told conference attendees these and other tools for building cloud-native applications, along with GitHub Actions, make it simpler than ever to build applications in an Azure cloud service that includes an integrated continuous integration/continuous delivery (CI/CD) platform.

    Microsoft is also previewing GitHub Advanced Security for Azure DevOps that natively embeds automated security checks using scanning tools into the Azure DevOps platform.

    Other additions to the Azure environment include tools for centralizing the management of application programming interfaces (APIs) using an Azure API Center service that is available in preview.

    The Azure Event Grid has also been updated to enable it to use HTTP to pull delivery of discrete events in addition to pushing them.

    Finally, Microsoft is also previewing generative artificial intelligence (AI) capabilities to make it simpler to manage cloud costs by, for example, using prompts to summarize invoices or breakdown costs.

    Microsoft, along with its GitHub subsidiary, is at the forefront of applying generative AI to application development using tools like GitHub Copilot that developers can use to write better code faster. That copilot capability is now being disseminated across the portfolio of applications and services that Microsoft provides.

    It’s not yet clear what the downstream impact of such tools are just yet, but the overall quality of the applications being built should improve as, for example, fewer mistakes are made that can lead to vulnerabilities.

    Obviously, the way application software is built, constructed and secured is about to change in the generative AI era. Thanks to its alliance with OpenAI, Microsoft has a significant head start on rivals building generative AI platforms that can similarly be applied to improve developer productivity.

    In the meantime, DevOps teams should assume the volume of code simultaneously moving through pipelines is about to dramatically increase as developers become more productive. There will inevitably be more opportunities to apply multiple forms of AI to reduce any DevOps bottlenecks that might arise, but it’s clear most of the immediate benefits are being provided to developers.

    One way or another, however, the overall rate at which applications are built and deployed is about to change drastically. The issue is understanding how those advances will impact downstream DevOps workflows that were constructed for a different era of application development.

  • Improving DevX Gives Your Business a Boost

    Improving DevX Gives Your Business a Boost

    As organizations transform and grow, different methodologies and development tools can come together, creating an extra level of complexity for your mainframe application development teams. In a perfect world, developers would be able to focus on delivering solutions without worrying about all the infrastructure necessary to make that happen. Building a better developer experience, or DevX, can make that a reality.

    DevX is about meeting developers where they are, at every stage of their career, delivering compatibility with and connectivity to the tools and devices that they want to use so they can focus on what they do best: Coding and releasing high-quality mainframe applications faster and more frequently and fostering better productivity and more innovation.

    Automation is integral to removing friction for developers. A modern integrated development environment (IDE) with built-in automation allows developers to find and manage code more efficiently and with greater quality and move quickly and easily from coding to editing to debugging to testing.

    Modern tooling can simplify and accelerate how developers work by giving them a visual representation of their code layout in a single-pane-of-glass view. They can move quickly in a natural, easy way with more information at their fingertips while built-in automation offers assistance at the right moment without getting in the way.

    This can seem difficult when dealing with mainframe application development and longtime mainframe developers who are still working with tools from the 1980s haven’t directly benefitted from the intervening decades of improvements. What they might not realize is that the improvements made for other platforms are available to mainframe developers, too. By engaging them in your DevX modernization efforts, you can improve their experience—and that of their team members.

    Elements of a Productive Mainframe Developer Experience

    • A modern DevX adopts flexible agile processes that give developers predictable capabilities and enable smaller, iterative changes with faster feedback. It also syncs distributed and mainframe development teams to foster greater collaboration for additional process improvements.
    • A better DevX is platform-agnostic, creating a development environment where developers can focus on the functionality of the applications they’re building instead of the deployment mechanism, leveling the playing field for every generation of developer.
    • With an improved DevX, developers can achieve the full potential of DevOps. Implementing DevOps for source code management (SCM) functions allows developers to understand the scope of any changes needed before they start coding, which helps speed approvals through concurrent work streams.
    • The single biggest boost to productivity—and an integral component of DevX—is shift left automated testing that enables developers at all skill levels to test new code immediately and find and fix bugs earlier in the development cycle when they’re easier to fix—and before they reach the end user. Continuous testing also automatically prepares test data within the DevOps toolchain.
    • An improved DevX delivers operational insights, so developers aren’t just catching issues but also understanding and prioritizing them, working hand in hand with the operations team to make the overall system better. It also empowers newer developers to address and correct any issues that arise in legacy code without poring over hundreds of modules to find what they need to improve. DevX tools work together to give developers an opportunity to leverage operational data in a whole new way.

    Once you have the tools, your mainframe developers need to be supported by a culture and education that teaches them how to use them and understand the benefits they offer. If you implement new tooling that nobody will use, the transformation is for naught. It’s important to start with the user experience and find the right tools to address it rather than starting with a tool and forcing its use just because it’s new. Continuous improvement is an important part of this; otherwise, you could bring the mainframe developer experience from the 1980s up to today’s standards only to have it freeze in place. This is why metrics and analytics are so important.

    As you speed up delivery, you also want to make sure the quality improves. With an improved DevX, you’re consistently finding defects earlier in the life cycle, which results in safer, more efficient delivery of better-quality code. By tracking velocity with metrics such as user stories delivered in one sprint, you can gauge how those changes are making a difference. You can also demonstrate how automation helps eliminate human error, increases code drops, and improves business agility.

    If you’re giving your mainframe developers everything they need to do their jobs easily and more efficiently, they’re exponentially more likely to adopt the tool and advocate for it—because a better DevX improves the day-to-day employee experience for longstanding and next-generation developers. By removing the drudgery of tasks that take developers away from writing code, you’re unleashing their creative and enabling them to achieve even greater levels of innovation—and you’re also improving your hiring and retention practices.

    Improving the mainframe DevX pays dividends across your business, from your developers to your customers to your stakeholders, so you can bring more new features to market faster and enable a competitive response for revenue-generating solutions. The mainframe is here to stay. Make sure you’re building a modern development environment that empowers your developers to respond nimbly in the face of change and make change your advantage.

  • Boomi Brings Generative AI to Integration Platform

    Boomi Brings Generative AI to Integration Platform

    Boomi announced today it is applying generative artificial intelligence (AI) to enable organizations to automate integrations via a natural language interface.

    Boomi CEO Steve Lucas said Boomi AI makes it much simpler for professional and citizen developers alike to create integrations at scale using generative AI platforms to invoke the low-code engine that Boomi already provides to create integrations between thousands of applications.

    Those generative AI capabilities will complement a wide range of AI capabilities based on machine learning algorithms that are already embedded in the Boomi platform, he added.

    Generative AI’s Impact on Integration and Automation – Steve Lucas, Boomi from Techstrong.TV on Vimeo.

    Boomi has been widely used to drive business process integration via an integration platform-as-a-service (iPaaS) environment it delivers as a managed service. The low-code platform the company created significantly reduces the need to rely on professional developers to write procedural code using Java, for example, to create those integrations.

    Generative AI capabilities make it possible for almost anyone of any skill level to create those integrations across multiple types of application programming interfaces (APIs), noted Lucas.

    That capability also makes it possible for Boomi to notify end users when, for example, a data attribute in an application has changed and ask them if they would like to apply that change across their entire application environment.

    In addition, Boomi AI can also interpret the intent of a prompt to better orchestrate processes in addition to providing an audit history in natural language that explains how decisions were made and what outcomes occurred as a result. As part of that effort, Boomi algorithms are also trained to avoid biases and unfairness with established ethical guidelines.

    Boomi’s AI engine is trained using anonymized data collected from approximately 20,000 customers that have created more than 200 million integrations using the Boomi platform. Those integrations have been exposed to third-party large language models (LLMs) that the company is using to train Boomi AI, said Lucas. Boomi is not storing any customer data to train its AI models.
    DevOps teams, of course, spend hours creating integrations that should be much simpler to create, test and validate with the rise of generative AI. In fact, the number of integrations being created to drive a wide range of digital business transformation initiatives should substantially increase.

    In the meantime, DevOps teams would be well-advised to start evaluating which tasks they manually perform today that could instead be automated by a generative AI platform. Many of the routine integrations that DevOps teams create today using low-level APIs will be increasingly automated by end users. The ultimate goal should be to identify more complex tasks that previously DevOps teams would not have attempted simply because there wasn’t enough time.

    It’s still too early to determine what impact generative AI will have on the way IT is managed, but DevOps teams that are typically committed to automating as many processes as possible should be at the forefront of adoption. The challenge is going to be simply keeping track of the rate of change that might soon be enabled by these platforms.

  • Quick Guide to Azure Storage Pricing

    Quick Guide to Azure Storage Pricing

    Azure Storage is a cloud-based storage service from Microsoft Azure. It provides a scalable, secure and highly available data storage solution for businesses of any size.

    The platform includes various services such as Azure Blob, which is a service for storing large amounts of unstructured data such as documents, images, videos and backups. Azure Files is a fully managed file share service for cloud-based applications. Azure Tables is a NoSQL key-value store used to store semi-structured data. Azure Queues is a messaging service used to store and process messages between applications or services.

    Finally, Azure Disks is a service that allows you to store and manage data in virtual hard disks used in virtual machines. With Azure Storage, businesses can easily manage their data, control access and scale their storage as per their needs without worrying about hardware or infrastructure management.

    How does Azure charge for storage space (bandwidth, capacity, number of transactions and quantity of transferred data)? What are the usage factors impacting Azure Storage fees: Region, account, access tier, replicas, egress?

    How Azure Storage Pricing Works

    Azure pricing for its storage services is based on several factors including storage capacity, data transfers and transaction counts. The cost of storage varies depending on the region and access tier selected, as well as the type of storage being used.

    For Azure Blob Storage, customers are charged for storage capacity, data egress (outbound data transfers) and the number of transactions (read, write and list operations).

    Azure Table Storage charges are based on the number of table entities stored and the amount of data transferred. Azure Queues are charged per transaction and data transfer, while Azure Disks are billed based on the provisioned capacity and the number of transactions.

    Several usage factors can impact Azure Storage fees, including the region where the data is stored, the storage account type, the access tier selected, the number of replicas and the amount of data transferred out of the storage account. 

    Overview of Azure Storage Pricing

    Blob Storage

    Azure Blob Storage pricing varies depending on the storage tier, data volume and whether customers opt for pay-as-you-go or reserved capacity.

    For pay-as-you-go pricing, Premium storage is charged at a flat rate of $0.015 per GB/month, while Cool and Archive storage are billed at $0.01 and $0.0099 per GB/month, respectively, regardless of data volume. Hot storage costs $0.018 per GB /month for the first 50 TB with the rate gradually decreasing with higher usage. For instance, customers who consume more than 500 TB/month pay $0.0166 per GB.

    Customers can also reserve blob storage in 100-TB or 1-PB blocks for one or three years. The prices for reserved storage vary by tier. A three-year Archive storage reservation costs $84 per 100 TB or $810 per PB/month, while Hot storage costs $1,244 per 100 TB/month for three years. For one-year reservations, Archive storage costs $91 per 100 TB or $883 per PB/month, and Hot storage costs $1,545 per 100 TB/month. For a one-year reservation, the cost of Hot storage is $15,050 per PB/month, and $11,963 per month for three years.

    Files

    Microsoft charges customers for Azure Files depending on the storage tier and whether they reserve storage capacity. Pay-as-you-go pricing ranges from $0.015 per GB/month for Cool storage to $0.016 per provisioned GiB/month for Premium file storage.

    Customers can commit to reserved storage capacity in increments of 10 TiB or 100 TiB for one or three years. A three-year reservation of 10 TiB Cool storage costs $101.39 per month, while one year costs $125.92 per month. However, the price increases significantly for the Premium offerings, with a three-year reservation of 10 TiB costing $1,081.34 per month and a one-year reservation costing $1,343.50 per month. 100 TiB reservations range from $983/month in the Cool tier for three years to $12,779.50/month in the Premium tier for one year.

    Nevertheless, there are other fees that add to the total cost such as read and write transactions, snapshots, metadata storage and early file deletion fees. These ancillary fees can impact the overall cost of using Azure Files and customers should account for them when estimating their expenses.

    Tables

    Azure Tables pricing is available in various tiers, including locally redundant storage (LRS), geo-redundant storage (GRS), read-access geo-redundant storage (RA-GRS), zone-redundant storage and geo-zone-redundant storage. 

    Customers are charged between $0.045 per GB/month for LRS and $0.01265 per GB/month for geozone-redundant read-only storage. Additionally, Microsoft charges a $0.0036 fee for each set of 10,000 transactions made on Azure Tables.

    Queues

    Compared to Blob and File Storage, the pricing model for Azure Queues is more straightforward. Customers are charged 4.5 cents to $0.075 cents per GB/month, depending on whether they choose local redundant storage (LRS) or read-access geozone-redundant storage (RA-GRS). Additionally, there is a $0.004 fee for every set of 10,000 operations in Class 1 and 2.

    Managed Disks

    Azure Managed Disks have numerous pricing options that can be overwhelming. The cost of using Managed Disks depends on several factors, including the disk size, performance, reservation period, capacity and disk type.

    The cheapest option for Managed Disks is the P1 disk, which costs $0.6 per month for 4 GiB of capacity in addition to $0.03 per mount/month. On the other hand, the P80 disk is the most expensive option, as it has a 32 TiB capacity costing $3,276.80 per month, in addition to $219/mount. However, the cost of a P80 disk can be reduced to $3,113/month (plus the $219/mount fee) with a reservation of one year.

    Azure Storage Vs. AWS Storage Pricing

    Here is a comparison of the pricing for similar storage services offered by Azure and AWS. As part of an Azure or AWS cost optimization strategy, you might consider combining services from both providers.

    Block Storage

    Amazon EBS is Amazon EC2’s basic storage device, with a price of $0.045 per GB for HDD and $0.10 per GB for SDD, with higher IOPS resulting in higher prices. The free tier includes 30 GB of SDD.

    For comparable regions in the U.S., Azure charges $0.05 per GB for HDD and $19.71 for 128 GB of SSD per month.

    Azure Managed Disks offer greater flexibility and easier backup than Azure Virtual Disks or Amazon EBS, with adjustable volume sizes and available IOPS. Managed Disks pricing starts at $19/month for a 128 GB disk, providing 500 IOPS.

    Object Storage

    Amazon S3 and Azure Blob Storage are both scalable services for storing objects (unstructured data) where data is stored together and cannot be changed in parts. Prices vary depending on the region, but for a comparable region in the U.S., Amazon S3 storage costs $0.023 per GB/month for the first 50 TB in addition to network usage fees. The free tier offers 5 GB of storage, while archive storage (Glacier) is $0.004 per GB/month.

    Azure Blob Storage starts at $0.0184 per GB/month for hot storage, although prices decrease to $0.01 per GB per month for cool storage and $0.002 for archive storage. Prices for both services increase for greater redundancy. 

    File Storage

    Amazon Elastic File System (EFS) is an equivalent service to Azure File storage and allows users to mount files to Amazon EC2 servers. It uses NFS V4.0/4.2 to mount files and is not available in all AWS regions (currently, four are supported), requiring a Direct Connect line or VPN for on-premises mounting.

    For a comparable U.S. region, Amazon EFS costs $0.30 per GB/month and the free tier offers 5GB on sign-up. Azure File Storage prices start at $0.06 per GB/month for local redundant storage, with prices increasing for more distributed redundancy setups. It is worth noting that both Azure and AWS charge additional fees for data access.

    Conclusion

    In conclusion, Azure Storage offers a range of services for storing and managing data in the cloud, each with its own pricing structure. While the pricing varies based on several factors including storage capacity, data transfers and transaction counts, it is essential to consider other usage factors, such as region, account, access tier, replicas and egress. 

    Customers should review these factors carefully and estimate their expenses to select the right storage service that meets their needs and budget. By doing so, they can leverage Azure Storage’s flexibility and scalability to store and manage their data efficiently in the cloud.

  • Using Microsoft Azure as a Backup Repository

    Using Microsoft Azure as a Backup Repository

    Every year, the amount of data that needs to be backed up increases. There are two ways to store backups – using on-premises facilities or trusting cloud-based solutions. Having an on-premises storage infrastructure can be costly for a company. This is because of the high prices for storage equipment and the expenses to maintain this equipment, e.g., hiring a storage engineer. Depending on a company’s rules, Microsoft Azure could be an alternative to traditional ways of keeping data. If the amount of data is relatively small (less than 4 TB), there is no problem with making use of this solution; you can either use Azure Blob as a CIFS or NFS storage, or you can create your own virtual machine by adding a disk to store your data and then configuring it as a file server. However, this blog post focuses on cases where space limits are more than 4 TB.

    Microsoft Azure allows you to create VMs and add disks with a maximum capacity of 4 TB for each disk. It is suitable for application servers but very often doesn’t work if you require a big file or backup server.

    However, we can exceed this limit by using Windows software RAID array. The idea is to use RAID5 as the software RAID which contains arrays starting from three disks.

    Why RAID5? Because it has a good read speed limit. It is also economical to use X disks, where X starts at three and you only lose one disk space for the total RAID result space. Furthermore, this should give you redundancy if one of your disks were to fail. However, if we build a RAID array in Azure, we already have the virtual disks as blocks of our array and they are protected by Microsoft Azure Cloud. Thus, the chance of failure in our RAID array (which is based on disks that are already in the RAID array) falls significantly.

    Here’s how to create an Azure-based backup and replication repository server.

    1. Log in to your Microsoft Azure account or register a new account at https://azure.microsoft.com/en-us/

    2. Create a new Windows Virtual Machine:

    In this example, we choose Windows 2008 R2 SP1, since we don’t require any advanced services, just software RAID5.

    3. Type in your VM name, username and the password according to MS Azure rules of complexity.

    4. Choose your VM size. Please note that the VM type depends on the number of attached disks. Thus, if you require more than 8 disks (over 24 GB in RAID 5), the only option is to choose DS2_V2 Standard:

    5. The settings for available storage are some of the main options for our future repository:

    At this point, you have created a new VM; now it is time to add disks. On the Disks page, you can add as many disks as you need for your future disk array. In our case, we will build software RAID5, which will require seven disks in total.

    We need to have our SMB share available from the internet, so we have to configure the Azure Firewall settings.

    In the Networking section, click the Add inbound port rule button and create a new rule allowing you to make port 445 (SMB) accessible to everyone (for testing). Alternatively, you can only make the port accessible to the backup software instance’s external IP as in the Source field (for production).

    The next step is to connect to the VM via RDP using the IP and credentials you created during the VM creation process.

    Next, you need to create a RAID5 volume. In Storage Manager, you`ll see all 4 TB disks you attached. Right click Disk Management > All Tasks > New Raid 5 Volume:

    Select all available 4 TB disks to add them to your array and move them to the right column:

    Follow the standard wizard; choose a letter for the disk and perform a quick format by NTFS.

    Next, create an SMB share on a new logical disk and add write permissions to the future repository.

    At this stage, you can use your entire 24 TB volume as a remote repository in your backup system.

    If you use the NAKIVO backup to Azure solution, just add a new repository using the server’s IP path to your share as well as the credentials you set for the SMB share:

    Then, create a new job using the new repository as the backup target and run it.

    Conclusion

    The amount of data organizations need to back up is growing exponentially. Therefore, it is important to store that data in the most economical way. Using a data backup system like NAKIVO and RAID5, you can make sure your VMs are reliably protected and that there is enough space to store them in Azure.

  • Consider Performance, Growth & Budget When Buying Data Analytics

    Consider Performance, Growth & Budget When Buying Data Analytics

    So, your business needs to invest in data analytics technology to improve efficiencies, competitive advantage or business outcomes. There are a ton of things to consider, but underlying each decision are three driving factors:

    • Performance – The need to meet service level agreements.
    • Growth – The need to accommodate success, as well as deal with inevitable industry changes.
    • Budget – The need to do both of the above while keeping costs low enough to not erode the profit gain from analytics.

    These three forces underpin nearly every technology selection decision, but what frequently isn’t noticed is how much these three drivers interact.

    1. Concurrency and Growth

    Test both for current levels of concurrency and the level you hope to achieve in the next few years as the organization grows.

    When evaluating benchmarks between vendors, consider that benchmarks are designed to make that company’s software look good. For example, if they are good at concurrency, you will see various levels of concurrency represented in the benchmark. If the benchmark shows only one workload at a time, or maybe ten, you can safely assume that concurrency is probably not something that technology is good at. Similarly, if the benchmark is done at 10 TB scale, the vendor probably isn’t that good at high-scale analytics.

    When bringing vendors in for a proof-of-concept, one mistake I’ve seen (more times than I can count) is testing the capabilities of the software with only one user at a time. People do not line up politely to use analytics one at a time.

    In addition to current levels of concurrency, over time, hopefully, the organization will grow, and the number of both internal analytics users and external customers will increase. Testing for the number of users you have now may not uncover problems that will invariably reveal themselves when a lot of people hit the system at once. The last thing your organization needs is to turn away customers because they’re overloading your systems.

    2. Performance & Price

    Evaluate price/performance, not performance alone.

    Everyone understands the importance of performance. Not one user has ever gone to a data architect or engineer and requested for their analytics to “please execute more slowly.” But performance isn’t simply how fast the analytical software can process queries, how fast the data pipelines can move data or how fast the response is between data input and reaction output. There’s also a monetary aspect of performance to bear in mind.

    In a recent benchmark, three data management and analytics stacks were compared with 60 simulated concurrent users/workloads. The difference in queries per hour on 250 TB of data was minor, perhaps 16 queries per hour for the slowest and 23 queries per hour for the fastest.

    (Mcknight Benchmark – SaaS Data Analytics Platform Comparison – vendor names removed)

    It looks like performance is just not that big a consideration for this technology. But there is no such thing as an unlimited budget. Let’s look at how much it would cost to get this level of performance out of each stack.

    (Mcknight Benchmark – SaaS Data Analytics Platform Comparison – vendor names removed)

    Price-performance is a far more useful metric of value to the organization. How much the software can do is irrelevant if the budget limits how much the business can afford to do with it.

    3. Software Efficiency

    Consider how much efficiency and control you’re willing to trade for ease-of-use.

    When you analyze data in the cloud, you’re not just paying for software, you’re also paying for hardware, a.k.a. cloud compute instances. Bundling the two costs in one bill makes things easier. But it also means that cloud providers make more money the more instances, or compute nodes, you use. You trade ease-of-use in billing for a higher price. And they no longer have a financial incentive to provide efficient software.

    Similarly, to make a cloud-based analytics software scale up in performance and concurrency with zero effort on your part, the software has to use only defaults for all settings internally. It can scale only by throwing more compute at the problem; compute that you’re paying for every time you use it. You’re trading control of the software–tuning, scaling guidelines, shut down conditions, etc.–for ease-of-use, again for a higher price.

    Throwing more compute at a problem isn’t always the best solution. Recently, I saw a POC where the customer implementation included 278 EC2 nodes on Amazon cloud. The organization didn’t initiate the POC because of inefficiency. They were having performance problems, even with that much compute power. A competing product was able to run on only nine EC2 nodes with the same workload and better performance. The company was paying 25 times as much for the compute hardware because it was convenient.

    Be aware of the tradeoffs.

    4. Affordable Scalability

    Paying for only what you use can become impractical when you use more and more.

    The volume of data that is being created, consumed and captured is growing exponentially. By 2025, it is estimated that nearly 30% of data being generated will be real-time. Instead of pulling data in batches, data will be pulled in continuously. Queries will be responded to in seconds, not hours, or in milliseconds, not seconds.

    Going to the cloud can make tremendous sense when a company or a workload in that company is new. But studies show that as businesses scale and growth slows, the pressure the cloud puts on business margins eventually outweighs its benefits. This is why most enterprises today are considering repatriating some or all of their workloads off the cloud.

    5. Flexibility for Change

    Make certain your stack will continue to work even as conditions change, and you won’t have to rebuild from scratch to do the same thing in a different location.

    When you’re building a long-term architectural strategy, you want to ensure you cover as many bases as possible because, the fact is, you really don’t know what the future holds. Ten years ago, maybe you had a Hadoop on-premises strategy or a data warehouse appliance. Today, you may have a cloud-focused strategy. Tomorrow, maybe you’ll move some or all of those workloads back on-premises due to performance, security, costs, regulations or other reasons.

    This is why you want flexibility that doesn’t mandate a “cloud-only” or an “on-premises-only” or a “this-cloud-only” architecture and which doesn’t integrate with only the same vendors’ software. You want an architecture that can work equally well both on the cloud as well as on-premises, hybrid or multi-cloud, providing the flexibility of going where business needs are best met.

    Containerization offers possibly the highest level of flexibility in deployment, but it can also add to deployment complexity, so again, be aware of what you’re trading. If the software you choose is platform-agnostic—not tied to any particular location, ecosystem or deployment model—then that will make future choices a lot more flexible.

    Regardless of what analytics you choose, what you decide to build or what use cases you have to meet, striving for balance in performance, growth and budget will create a strong data architecture, now and for the future:

      1. The analytics architecture is high-performance: it should meet SLAs, get things done on time and meet or exceed expectations and requirements.
      2. The analytics architecture is resilient to growth and change; even if there is rapid growth or sudden or extreme changes, it’s future-ready.
      3. The analytics architecture should control costs and live within an allocated budget.

    Keep these guidelines in mind when choosing an analytics provider. The decisions you make today will have direct ramifications on the growth, profitability and success of the business tomorrow.

  • Survey: Much Greater Focus on Reining in IT Costs

    Survey: Much Greater Focus on Reining in IT Costs

    A survey from Vega Cloud found 62% of respondents are more closely monitoring all IT costs, with more than half (52%) more closely evaluating their cloud service provider options.

    The survey polled 100 vice presidents from organizations with more than 1,000 employees and was published today by Vega Cloud, a provider of a platform for managing multiple clouds.

    Overall, nearly half of respondents (48%) said they expected to reduce their IT spending a little or a lot in 2023, with 65% identifying the reduction of cloud spending as a specific goal. Three quarters (75%) are looking to cut cloud costs by 10-50% in 2023, the survey found.

    A full 93% reported they have invested at least a fair amount of money on tools to optimize cloud costs, with 82% having invested in FinOps staff or people with FinOps experience. The challenge is that 70% of respondents said it was difficult to find people with these skill sets, the survey found.

    The best way for organizations to reduce cloud costs is to become more informed about what they’re spending (40%), and 33% said that the best approach is to consistently optimize use of cloud resources. A total of 61% identified reserved instances/savings plans as the best way to optimize cloud use, while 55% said optimization was best achieved through waste elimination followed closely by right-sizing (54%).

    Vega Cloud CEO Kris Bliesner said it’s clear the days when organizations allowed developers to consume cloud resources at will are coming to an end as cloud services consume a much larger percentage of the IT budget. In fact, the survey noted that 91% of respondents are spending at least one million dollars annually on cloud infrastructure alone.

    The issue is that finance departments not only want to reduce cloud spend but also want cloud spending to become more predictable from a budget perspective. Nearly half of survey respondents (45%) said they’d received a surprise in their cloud bills “a number of times” over the past 12 months, with 70% of respondents noting surprises have happened at least a half dozen times in the past year.

    A total of 40% admitted they had exceeded budgets multiple times in the past year, and 38% said they had to modify their forecast cloud spending budget numerous times over the past 12 months. More than half of respondents (55%) said database costs are the metric that matters most when looking at cloud usage followed by bandwidth costs (51%), storage costs (49%) and compute costs (47%). Nearly two-thirds (64%) said they track metrics at least somewhat often, while 46% said they have somewhat optimized their cloud use. A total of 11% said they’re not optimized at all.

    A full 88% said they’re using artificial intelligence (AI) at least somewhat to optimize cloud use and costs, with nearly three-quarters (74%) noting they think ChatGPT or a similar generative AI tool could be used to optimize cloud use/costs further. Specifically, 42% said they thought it could improve decision making and 30% said it could automate decision making.

    Among the cloud service providers, Google is cited as doing the most to help customers manage and reduce cloud costs (38%), followed by Amazon Web Services (34%) and Microsoft (28%)—but a full 85% said they could all be doing more.

    Of course, changing economic conditions are naturally driving organizations to look harder at cloud spending. Well over half of respondents (56%) said they are concerned about the economy, and 95% of those respondents said those concerns have, at least somewhat, impacted their IT strategy.

    It’s not clear whether organizations will continue to exercise greater control over cloud spending should the economy recover. For now, at least, there’s clearly more desire for control than ever.