devopsy.kinsta.cloud

Category: IT as Code

  • Microsoft Adds Slew of Developer Tools to Azure

    Microsoft Adds Slew of Developer Tools to Azure

    At its Build 2023 conference this week, Microsoft announced general availability of an Azure Deployment Environment service that provides developers with a portal through which they can employ infrastructure-as-a-code (IaC) templates based on Terraform or Azure Resource Management files.

    In addition, Microsoft has enhanced Microsoft Dev Box, a workstation environment for developers that run Azure, to include a portal through which they can manage multiple environments.

    Scheduled to be generally available in July, Microsoft Dev Box is also previewing a capability that makes it possible to customize environments using configuration-as-code files stored in a Git repository. Microsoft is also previewing tighter integration between Dev Box and its Visual Studio integrated development environment (IDE) and Windows to further customize developers’ environments.

    Scott Guthrie, executive vice president of the cloud and artificial intelligence (AI) group at Microsoft, told conference attendees these and other tools for building cloud-native applications, along with GitHub Actions, make it simpler than ever to build applications in an Azure cloud service that includes an integrated continuous integration/continuous delivery (CI/CD) platform.

    Microsoft is also previewing GitHub Advanced Security for Azure DevOps that natively embeds automated security checks using scanning tools into the Azure DevOps platform.

    Other additions to the Azure environment include tools for centralizing the management of application programming interfaces (APIs) using an Azure API Center service that is available in preview.

    The Azure Event Grid has also been updated to enable it to use HTTP to pull delivery of discrete events in addition to pushing them.

    Finally, Microsoft is also previewing generative artificial intelligence (AI) capabilities to make it simpler to manage cloud costs by, for example, using prompts to summarize invoices or breakdown costs.

    Microsoft, along with its GitHub subsidiary, is at the forefront of applying generative AI to application development using tools like GitHub Copilot that developers can use to write better code faster. That copilot capability is now being disseminated across the portfolio of applications and services that Microsoft provides.

    It’s not yet clear what the downstream impact of such tools are just yet, but the overall quality of the applications being built should improve as, for example, fewer mistakes are made that can lead to vulnerabilities.

    Obviously, the way application software is built, constructed and secured is about to change in the generative AI era. Thanks to its alliance with OpenAI, Microsoft has a significant head start on rivals building generative AI platforms that can similarly be applied to improve developer productivity.

    In the meantime, DevOps teams should assume the volume of code simultaneously moving through pipelines is about to dramatically increase as developers become more productive. There will inevitably be more opportunities to apply multiple forms of AI to reduce any DevOps bottlenecks that might arise, but it’s clear most of the immediate benefits are being provided to developers.

    One way or another, however, the overall rate at which applications are built and deployed is about to change drastically. The issue is understanding how those advances will impact downstream DevOps workflows that were constructed for a different era of application development.

  • Red Hat Enhances Insights to Simplify RHEL Management

    Red Hat Enhances Insights to Simplify RHEL Management

    This week at its Red Hat Summit event, Red Hat introduced an expanded set of capabilities to Red Hat Insights that simplify management of Red Hat Enterprise Linux (RHEL).

    Managing Linux across the hybrid cloud is complex, and it’s made even more so because of current macroeconomic conditions and the challenge of finding and retaining highly skilled DevOps and IT professionals, said Gunnar Hellekson, vice president and general manager, Red Hat Enterprise Linux, Red Hat.

    The enhanced Insights capabilities add a layer of abstraction to help reduce enterprise Linux complexity across the hybrid cloud and make RHEL management easier by lowering skill level requirements for managing Linux server estates, the company said.

    Using a single UI, entry-level systems administrators and IT support teams can access Red Hat Insights’ predictive analytics to identify potential bugs, misconfigurations or security vulnerabilities and remediate them. Systems administrators of all skill levels can detect, assess and push fixes for potential problems without a deep understanding of Linux management systems like Red Hat Satellite Server and without having to interact with the command line, the company said.

    These enhancements help organizations maintain services and systems even in uncertain macroeconomic times when hiring is constrained. Red Hat research also claimed that these enhanced capabilities enabled IT organizations to find IT issues up to 90% faster and remediate them nearly 66% faster across the hybrid cloud.

    A Red Hat Insights image builder service also helps lower the skills level needed to manage Linux estates by enabling IT teams to build standardized, optimized operating system images, including the creation of their own ‘golden images’ that adhere to organization-specific needs for security and compliance. These images can be deployed across public clouds, hybrid clouds, data centers and the edge using the same console.

    As the economic downturn continues, organizations are looking to do more with less and identify ways to make the most of the resources, talent and skillsets they already have. Red Hat is making the case for adding a layer of abstraction to allow less-highly-skilled IT and DevOps team members to contribute to managing Linux estates across hybrid, multi-cloud and edge deployments.

    “The demands of managing Linux across the hybrid cloud are stretching the ability of resource-constrained IT departments to keep pace,” said Hellekson in a statement. “CIOs need to be able to extend existing skillsets and lower the skill barriers for overseeing Linux estates, which is exactly what the management capabilities of Red Hat Insights are designed to do.”

  • Red Hat Moves to Secure Software Supply Chains

    Red Hat Moves to Secure Software Supply Chains

    Red Hat today announced a portfolio of cloud services designed to better secure software supply chains. The expanded portfolio includes Red Hat Trusted Application Pipeline to secure continuous integration/continuous delivery (CI/CD) workflows and Red Hat Trusted Content offerings to better secure software components.

    Announced at the Red Hat Summit, the Red Hat Trusted Software Supply Chain combines a suite of curated open source tools into a cloud service managed by Red Hat. The Red Hat Trusted Application Pipeline service is based on sigstore, an emerging de facto standard for cryptographically signing code to ensure immutability that Red Hat helped drive, while Red Hat Trusted Content provides access to a set of software components that Red Hat has vetted.

    Red Hat Trusted Software Supply Chain also provides access to existing tools such as Quay, a registry for containers, and the Advanced Cluster Security (ACS) platform for securing Kubernetes clusters.

    Red Hat Trusted Software Supply Chain is scheduled to be made available in preview in the next few weeks, and will also track the provenance of components. In addition, the service will monitor emerging threats to software components and alert users of new and emerging risks. The service will also surface recommendations for remediating vulnerabilities it identifies based on best practices defined by Red Hat.

    Finally, Red Hat Trusted Application Pipeline also promises to make it simpler to automate the deployment of secure code with a few clicks within the context of an integrated continuous integration/continuous delivery (CI/CD) pipeline. DevOps team can also import Git repositories and configure container-native continuous build, test and deployment pipelines, inspect source code and transitive dependencies, autogenerate software bill of materials (SBOMs) within builds and verify and promote container images. An enterprise contract policy engine can consistently enforce compliance with the Supply Chain Levels for Software Artifacts (SLSA) framework.

    Finding ways to implement DevSecOps workflows is a major challenge for any organization. Red Hat is making a case for a managed cloud service based on a collection of security software and services that a DevOps team would otherwise have to implement themselves.

    Tasked with securing increasingly complex software development environments, it’s not clear whether DevOps teams will opt to rely on a cloud service, but given the general lack of security expertise among software development teams, the fastest way to secure a software supply chain for most organizations is to rely on a service.

    Regardless of approach, of course, DevSecOps will soon become a standard element of application development workflows. Once that occurs, it’s not clear whether DevOps will simply become DevSecOps or whether security will be assumed to be an integrated element of any set of DevOps best practices.

    Sarwar Raza, vice president and general manager for cloud services at Red Hat, said regardless of approach, it’s clear that implicit trust is no longer good enough. Organizations of all sizes will need to be able to demonstrate that they have maintained a chain of custody for the software components used to build an application, he noted.

    One way or another, the overall security of downstream applications should eventually improve as the processes used to construct software are increasingly locked down.

  • Improving DevX Gives Your Business a Boost

    Improving DevX Gives Your Business a Boost

    As organizations transform and grow, different methodologies and development tools can come together, creating an extra level of complexity for your mainframe application development teams. In a perfect world, developers would be able to focus on delivering solutions without worrying about all the infrastructure necessary to make that happen. Building a better developer experience, or DevX, can make that a reality.

    DevX is about meeting developers where they are, at every stage of their career, delivering compatibility with and connectivity to the tools and devices that they want to use so they can focus on what they do best: Coding and releasing high-quality mainframe applications faster and more frequently and fostering better productivity and more innovation.

    Automation is integral to removing friction for developers. A modern integrated development environment (IDE) with built-in automation allows developers to find and manage code more efficiently and with greater quality and move quickly and easily from coding to editing to debugging to testing.

    Modern tooling can simplify and accelerate how developers work by giving them a visual representation of their code layout in a single-pane-of-glass view. They can move quickly in a natural, easy way with more information at their fingertips while built-in automation offers assistance at the right moment without getting in the way.

    This can seem difficult when dealing with mainframe application development and longtime mainframe developers who are still working with tools from the 1980s haven’t directly benefitted from the intervening decades of improvements. What they might not realize is that the improvements made for other platforms are available to mainframe developers, too. By engaging them in your DevX modernization efforts, you can improve their experience—and that of their team members.

    Elements of a Productive Mainframe Developer Experience

    • A modern DevX adopts flexible agile processes that give developers predictable capabilities and enable smaller, iterative changes with faster feedback. It also syncs distributed and mainframe development teams to foster greater collaboration for additional process improvements.
    • A better DevX is platform-agnostic, creating a development environment where developers can focus on the functionality of the applications they’re building instead of the deployment mechanism, leveling the playing field for every generation of developer.
    • With an improved DevX, developers can achieve the full potential of DevOps. Implementing DevOps for source code management (SCM) functions allows developers to understand the scope of any changes needed before they start coding, which helps speed approvals through concurrent work streams.
    • The single biggest boost to productivity—and an integral component of DevX—is shift left automated testing that enables developers at all skill levels to test new code immediately and find and fix bugs earlier in the development cycle when they’re easier to fix—and before they reach the end user. Continuous testing also automatically prepares test data within the DevOps toolchain.
    • An improved DevX delivers operational insights, so developers aren’t just catching issues but also understanding and prioritizing them, working hand in hand with the operations team to make the overall system better. It also empowers newer developers to address and correct any issues that arise in legacy code without poring over hundreds of modules to find what they need to improve. DevX tools work together to give developers an opportunity to leverage operational data in a whole new way.

    Once you have the tools, your mainframe developers need to be supported by a culture and education that teaches them how to use them and understand the benefits they offer. If you implement new tooling that nobody will use, the transformation is for naught. It’s important to start with the user experience and find the right tools to address it rather than starting with a tool and forcing its use just because it’s new. Continuous improvement is an important part of this; otherwise, you could bring the mainframe developer experience from the 1980s up to today’s standards only to have it freeze in place. This is why metrics and analytics are so important.

    As you speed up delivery, you also want to make sure the quality improves. With an improved DevX, you’re consistently finding defects earlier in the life cycle, which results in safer, more efficient delivery of better-quality code. By tracking velocity with metrics such as user stories delivered in one sprint, you can gauge how those changes are making a difference. You can also demonstrate how automation helps eliminate human error, increases code drops, and improves business agility.

    If you’re giving your mainframe developers everything they need to do their jobs easily and more efficiently, they’re exponentially more likely to adopt the tool and advocate for it—because a better DevX improves the day-to-day employee experience for longstanding and next-generation developers. By removing the drudgery of tasks that take developers away from writing code, you’re unleashing their creative and enabling them to achieve even greater levels of innovation—and you’re also improving your hiring and retention practices.

    Improving the mainframe DevX pays dividends across your business, from your developers to your customers to your stakeholders, so you can bring more new features to market faster and enable a competitive response for revenue-generating solutions. The mainframe is here to stay. Make sure you’re building a modern development environment that empowers your developers to respond nimbly in the face of change and make change your advantage.

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    LinkedIn

    Samsung Electronics America
    Mountain View, California
    Lead Site Reliability Engineer (SRE) – DevOps Specialist
    $180,000 to $210,000

    Indeed.com

    Federal Reserve Bank of New York
    New York, New York
    DevOps Engineer
    $178,100 to $207,900

    CareerBuilder.com

    SAIC
    Chantilly, Virginia
    DevOps Advisor
    $130,000 to $200,000

    Dice.com

    Saicon Consultants
    Alpharetta, Georgia
    DevOps Engineer
    $110,000 to $140,000

    Simply Hired.com

    Hewlett-Packard Enterprise
    Fort Collins, Colorado
    DevOps Automation Engineer
    $109,400 – $251,500

  • Infrastructure-as-Code Pitfalls in Platform Engineering

    Infrastructure-as-Code Pitfalls in Platform Engineering

    In the context of cloud operations, the last decade was ruled by DevOps and infrastructure-as-code (IaC). But what is true DevOps? Is it developers running their own operations? Is it operators learning development skills to automate infrastructure provisioning using infrastructure-as-code (IaC) tools like Terraform?

    Though DevOps is widely understood to incorporate both of these, software developers are far from fully understanding or operating infrastructure. IaC is purely a DevOps tool. When an organization claims to have automated its infrastructure, they mean automation for the efficiency of the DevOps teams, not automation for developers. Developers are still left waiting on DevOps for days for even small infrastructure changes.

    While there were efficiencies gained by the current DevOps model, there is an increasing acknowledgment that we require a better approach to enable developer self-service. This has led to the platform engineering discipline.

    The high-level goals of platform engineering are:

    • Developer self-service for significant parts of infrastructure updates without DevOps subject matter expertise.
    • Built-in security and compliance controls.

    Infrastructure-as-Code (IaC) Limitations in Platform Engineering

    A DuploCloud survey of 40+ enterprises that have made substantial investments in platform engineering showed that the prevalent approach is for DevOps teams to build a DevOps platform with IaC as a core underlying technology. They create templates for the organization’s use cases and publish them in a CI/CD pipeline or a self-service catalog.

    Here are the top reasons why platforms that build on top of IaC are failing these platform engineering goals:

    1. IaC templates are rigid and not flexible enough to meet changing developer requirements: It is true that DevOps teams can anticipate cloud infrastructure topologies to some extent and have IaC templates for those with a few customizable parameters. But in a microservices world, there are thousands of other workflows and topologies possible based on changing application needs and security controls. Large data centers and edge infrastructure make this even more complex. A manual approach, relying solely on DevOps personnel to constantly build and update myriad combinations in static scripts simply can’t scale.
    2. Scripting tools can’t build life cycle management: In cloud operations, people-triggered changes are only a subset of possible use cases. Many asynchronous operations need to be continuously performed. These range from detection of configurations, from desired state to reverting, or complex configuration scenarios where individual components have to be set up asynchronously and brought together later. It could be as simple as a certain component going down and needing to be restored. IaC is a script that runs when triggered to completion. It has no active life cycle to operate continuously in the background.
    3. Inability to build a concept of an environment: In any orchestration system, users have a concept of the environment they want to build. When they log in to the platform and navigate to their environment, they expect to update as well as view the state and aspects of resources in that environment, be it the provisioning status, metrics, logs, faults, audit logs, compliance posture, etc. They may want to perform debugging functions, such as restarting services, SSHinginto a VM or accessing a resource’s cloud console (S3 Console, for example) using access control boundaries within that environment. For example, in Kubernetes, you choose the namespace to be the environment and management software like Rancher on top of K8S to provide these functions. If we must replicate this same concept across a broad infrastructure-level platform, we cannot do it using only IaC. We need something with a scope that spans all cloud operations, like Kubernetes. Terraform is a configuration updating and management software, not an orchestration system.

    Many platform teams have tried to work around these problems with point solutions by using a disparate set of jobs for certain aspects of life cycle management, building a thin UI shim on top of cloud accounts for visualization of resources while redirecting to other systems like Datadog for logging, metrics and alerts. But for most of the use cases, the DevOps team is still very much in the operations workflow. This completely defeats the concept of developer self-service as well as continuous compliance goals.

    Learning From Other Successful Platforms

    Two recent examples of successful cloud platforms are Amazon Web Services (AWS) in the context of infrastructure-as-a-service (IaaS) and Kubernetes for container orchestration. These are distributed system implementations using higher-level programming languages like Java and Go. You can’t build such complex systems using scripts and jobs.

    Building a true DevOps orchestration platform requires a systems design approach. It also takes expert systems engineers and many years to build and mature, like it did for K8s and IaaS in the Public Cloud.

    Conclusion

    Since platform engineering teams need systems design expertise that is likely outside of the scope of the typical DevOps role, we need the software development function to intervene with distributed systems programmers and many years of investment. Most organizations trying to build in-house platforms don’t have or don’t want to make this type of investment, as it may not be their core business. Megascale tech companies like Facebook, Uber, Netflix, etc. will likely build an in-house solution, as they have the talent and scale, but for the masses, it is likely that the solution will be an off-the-shelf product from an ISV.

    The industry has solved IaaS with public cloud, container orchestration with Kubernetes and observability with platforms like Datadog. The platform engineering category is heading in the same direction.

  • What’s the Difference Between DevOps and Platform Engineering?

    What’s the Difference Between DevOps and Platform Engineering?

    The DevOps space is buzzing with a new trend—platform engineering. Platform engineering has been gaining steam over the last few years but has really peaked in interest lately. In fact, according to a study conducted by Puppet Labs, 51% of organizations have adopted platform engineering or plan to do so in the next year. But what exactly is platform engineering, and how is it different than DevOps?

    Below, we’ll explore the nuances between these two concepts. We’ll review what DevOps and platform engineering are and see how they’re similar and dissimilar. We’ll also outline the benefits of platform engineering and suggest some best practices to make the most of this emerging discipline.

    What is DevOps?

    DevOps is a broad topic, but it generally involves efforts to increase software release efficiency and frequency. Some of the hallmarks of the DevOps discipline include reducing siloes between teams and increasing collaboration. DevOps is often directly associated with various tools that automate the process of moving code from a local environment into production, also known as continuous integration and continuous delivery (CI/CD). Monitoring, incident management and configuration management tools also support the DevOps process.

    “DevOps reflects more of a philosophy and a way of working,” said David Sandilands, senior solutions architect for Puppet. DevOps enables more of a shift left mentality. Instead of having operations teams “guarding the gates,” developers work in partnership with Ops. In doing so, groups can decrease build time significantly, he said. This style helps engineers incrementally and rapidly release new features, replacing old waterfall development cycles.

    The ultimate goal of DevOps is to automate the software life cycle and shorten the feedback loop. Thus, DevOps takes a life cycle approach to shipping applications more efficiently, streamlining phases such as build, release, configure, monitor, plan, create and verify. Microservices architecture and containerization also often play a crucial role in enabling DevOps.

    What is Platform Engineering?

    Platform engineering is all about developing a self-service internal platform that serves the needs of developer consumers. Although DevOps opened things up and empowered individual developers with advanced automation, it left organizations with scattered tooling and processes, especially at large enterprises. For example, various infrastructure-as-code (IaC) configurations might exist for similar functions within the same organization but be unknown to engineers in different departments. Not to mention, overseeing a growing DevOps pipeline means added responsibilities around continual updates and security.

    Platform engineering places more control in the hands of a central platform team rather than individual developers. These teams typically treat the platform as a product. They do so by evaluating the needs of developers across an organization, building out internal self-service tools that suit the needs of most consumers and maintaining the platform.

    The 2023 State of DevOps Report found that this self-service approach typically handles areas like deployment, operation of infrastructure, monitoring production applications and security and compliance. Other capabilities, such as abstracting Kubernetes or overseeing secrets management, may also play a role in these efforts.

    Benefits of Platform Engineering

    • Improve developer experience: Platform engineering can enhance the experience a developer has with their software build and release systems. A better developer experience can do loads to improve morale and create a more productive engineering atmosphere.
    • Standardize DevOps practices: By bringing together common DevOps approaches into a shared platform, organizations can begin to standardize with reusable build processes and automated infrastructure. This can unite an organization on common runtimes, observability methods and deployment processes. This can streamline the introduction of DevOps practices to an organization.
    • Secure the DevOps pipeline: Various vulnerabilities exist within tooling that comprises modern CI/CD chains and cloud-native open source software. By outsourcing DevOps responsibility to a shared platform team, the organization can coalesce around more secure practices and avoid overlooking important updates or holes.
    • Supply additional guardrails: Platform teams might also assist in the acquisition of new tools as well as outlining and enforcing policies for software infrastructure. While this may come at the cost of customization, additional guardrails can help avoid shadow IT and governance around internal development. Letting platform teams handle it could make maintenance easier across an organization.

    Platform Engineering: Under the DevOps Umbrella

    I don’t see platform engineering as replacing DevOps. Instead, I view platform engineering as falling under the umbrella of DevOps since platform engineering takes DevOps processes and makes them more reusable and self-service. Both use a lot of automation and share many of the same end goals.

    However, there are nuances between the two approaches that reflect the recent trends in the IT market. With DevOps, the market was more interested in shifting responsibility left and empowering individual developers to oversee the entire software life cycle. Now, as sprawl and shadow IT emerge, the pendulum seems to be shifting back toward a bit more centralization and governance with platform engineering.

    Platform engineering is typically more of an in-house development initiative. So, what are some best practices IT divisions should remember to get the most out of platform engineering? Hope Lynch, senior director of platform at Cloudbees, shared some great strategies to consider that revolve around treating the platform as a product. Doing this involves accepting feedback from developers, not controlling everything and always maintaining a focus on developer experience.

    Hopefully, by following these tips, organizations should have a better grasp of the emerging discipline of platform engineering and have the confidence to begin introducing platform-based concepts within their own environments.

  • Boomi Brings Generative AI to Integration Platform

    Boomi Brings Generative AI to Integration Platform

    Boomi announced today it is applying generative artificial intelligence (AI) to enable organizations to automate integrations via a natural language interface.

    Boomi CEO Steve Lucas said Boomi AI makes it much simpler for professional and citizen developers alike to create integrations at scale using generative AI platforms to invoke the low-code engine that Boomi already provides to create integrations between thousands of applications.

    Those generative AI capabilities will complement a wide range of AI capabilities based on machine learning algorithms that are already embedded in the Boomi platform, he added.

    Generative AI’s Impact on Integration and Automation – Steve Lucas, Boomi from Techstrong.TV on Vimeo.

    Boomi has been widely used to drive business process integration via an integration platform-as-a-service (iPaaS) environment it delivers as a managed service. The low-code platform the company created significantly reduces the need to rely on professional developers to write procedural code using Java, for example, to create those integrations.

    Generative AI capabilities make it possible for almost anyone of any skill level to create those integrations across multiple types of application programming interfaces (APIs), noted Lucas.

    That capability also makes it possible for Boomi to notify end users when, for example, a data attribute in an application has changed and ask them if they would like to apply that change across their entire application environment.

    In addition, Boomi AI can also interpret the intent of a prompt to better orchestrate processes in addition to providing an audit history in natural language that explains how decisions were made and what outcomes occurred as a result. As part of that effort, Boomi algorithms are also trained to avoid biases and unfairness with established ethical guidelines.

    Boomi’s AI engine is trained using anonymized data collected from approximately 20,000 customers that have created more than 200 million integrations using the Boomi platform. Those integrations have been exposed to third-party large language models (LLMs) that the company is using to train Boomi AI, said Lucas. Boomi is not storing any customer data to train its AI models.
    DevOps teams, of course, spend hours creating integrations that should be much simpler to create, test and validate with the rise of generative AI. In fact, the number of integrations being created to drive a wide range of digital business transformation initiatives should substantially increase.

    In the meantime, DevOps teams would be well-advised to start evaluating which tasks they manually perform today that could instead be automated by a generative AI platform. Many of the routine integrations that DevOps teams create today using low-level APIs will be increasingly automated by end users. The ultimate goal should be to identify more complex tasks that previously DevOps teams would not have attempted simply because there wasn’t enough time.

    It’s still too early to determine what impact generative AI will have on the way IT is managed, but DevOps teams that are typically committed to automating as many processes as possible should be at the forefront of adoption. The challenge is going to be simply keeping track of the rate of change that might soon be enabled by these platforms.

  • Quick Guide to Azure Storage Pricing

    Quick Guide to Azure Storage Pricing

    Azure Storage is a cloud-based storage service from Microsoft Azure. It provides a scalable, secure and highly available data storage solution for businesses of any size.

    The platform includes various services such as Azure Blob, which is a service for storing large amounts of unstructured data such as documents, images, videos and backups. Azure Files is a fully managed file share service for cloud-based applications. Azure Tables is a NoSQL key-value store used to store semi-structured data. Azure Queues is a messaging service used to store and process messages between applications or services.

    Finally, Azure Disks is a service that allows you to store and manage data in virtual hard disks used in virtual machines. With Azure Storage, businesses can easily manage their data, control access and scale their storage as per their needs without worrying about hardware or infrastructure management.

    How does Azure charge for storage space (bandwidth, capacity, number of transactions and quantity of transferred data)? What are the usage factors impacting Azure Storage fees: Region, account, access tier, replicas, egress?

    How Azure Storage Pricing Works

    Azure pricing for its storage services is based on several factors including storage capacity, data transfers and transaction counts. The cost of storage varies depending on the region and access tier selected, as well as the type of storage being used.

    For Azure Blob Storage, customers are charged for storage capacity, data egress (outbound data transfers) and the number of transactions (read, write and list operations).

    Azure Table Storage charges are based on the number of table entities stored and the amount of data transferred. Azure Queues are charged per transaction and data transfer, while Azure Disks are billed based on the provisioned capacity and the number of transactions.

    Several usage factors can impact Azure Storage fees, including the region where the data is stored, the storage account type, the access tier selected, the number of replicas and the amount of data transferred out of the storage account. 

    Overview of Azure Storage Pricing

    Blob Storage

    Azure Blob Storage pricing varies depending on the storage tier, data volume and whether customers opt for pay-as-you-go or reserved capacity.

    For pay-as-you-go pricing, Premium storage is charged at a flat rate of $0.015 per GB/month, while Cool and Archive storage are billed at $0.01 and $0.0099 per GB/month, respectively, regardless of data volume. Hot storage costs $0.018 per GB /month for the first 50 TB with the rate gradually decreasing with higher usage. For instance, customers who consume more than 500 TB/month pay $0.0166 per GB.

    Customers can also reserve blob storage in 100-TB or 1-PB blocks for one or three years. The prices for reserved storage vary by tier. A three-year Archive storage reservation costs $84 per 100 TB or $810 per PB/month, while Hot storage costs $1,244 per 100 TB/month for three years. For one-year reservations, Archive storage costs $91 per 100 TB or $883 per PB/month, and Hot storage costs $1,545 per 100 TB/month. For a one-year reservation, the cost of Hot storage is $15,050 per PB/month, and $11,963 per month for three years.

    Files

    Microsoft charges customers for Azure Files depending on the storage tier and whether they reserve storage capacity. Pay-as-you-go pricing ranges from $0.015 per GB/month for Cool storage to $0.016 per provisioned GiB/month for Premium file storage.

    Customers can commit to reserved storage capacity in increments of 10 TiB or 100 TiB for one or three years. A three-year reservation of 10 TiB Cool storage costs $101.39 per month, while one year costs $125.92 per month. However, the price increases significantly for the Premium offerings, with a three-year reservation of 10 TiB costing $1,081.34 per month and a one-year reservation costing $1,343.50 per month. 100 TiB reservations range from $983/month in the Cool tier for three years to $12,779.50/month in the Premium tier for one year.

    Nevertheless, there are other fees that add to the total cost such as read and write transactions, snapshots, metadata storage and early file deletion fees. These ancillary fees can impact the overall cost of using Azure Files and customers should account for them when estimating their expenses.

    Tables

    Azure Tables pricing is available in various tiers, including locally redundant storage (LRS), geo-redundant storage (GRS), read-access geo-redundant storage (RA-GRS), zone-redundant storage and geo-zone-redundant storage. 

    Customers are charged between $0.045 per GB/month for LRS and $0.01265 per GB/month for geozone-redundant read-only storage. Additionally, Microsoft charges a $0.0036 fee for each set of 10,000 transactions made on Azure Tables.

    Queues

    Compared to Blob and File Storage, the pricing model for Azure Queues is more straightforward. Customers are charged 4.5 cents to $0.075 cents per GB/month, depending on whether they choose local redundant storage (LRS) or read-access geozone-redundant storage (RA-GRS). Additionally, there is a $0.004 fee for every set of 10,000 operations in Class 1 and 2.

    Managed Disks

    Azure Managed Disks have numerous pricing options that can be overwhelming. The cost of using Managed Disks depends on several factors, including the disk size, performance, reservation period, capacity and disk type.

    The cheapest option for Managed Disks is the P1 disk, which costs $0.6 per month for 4 GiB of capacity in addition to $0.03 per mount/month. On the other hand, the P80 disk is the most expensive option, as it has a 32 TiB capacity costing $3,276.80 per month, in addition to $219/mount. However, the cost of a P80 disk can be reduced to $3,113/month (plus the $219/mount fee) with a reservation of one year.

    Azure Storage Vs. AWS Storage Pricing

    Here is a comparison of the pricing for similar storage services offered by Azure and AWS. As part of an Azure or AWS cost optimization strategy, you might consider combining services from both providers.

    Block Storage

    Amazon EBS is Amazon EC2’s basic storage device, with a price of $0.045 per GB for HDD and $0.10 per GB for SDD, with higher IOPS resulting in higher prices. The free tier includes 30 GB of SDD.

    For comparable regions in the U.S., Azure charges $0.05 per GB for HDD and $19.71 for 128 GB of SSD per month.

    Azure Managed Disks offer greater flexibility and easier backup than Azure Virtual Disks or Amazon EBS, with adjustable volume sizes and available IOPS. Managed Disks pricing starts at $19/month for a 128 GB disk, providing 500 IOPS.

    Object Storage

    Amazon S3 and Azure Blob Storage are both scalable services for storing objects (unstructured data) where data is stored together and cannot be changed in parts. Prices vary depending on the region, but for a comparable region in the U.S., Amazon S3 storage costs $0.023 per GB/month for the first 50 TB in addition to network usage fees. The free tier offers 5 GB of storage, while archive storage (Glacier) is $0.004 per GB/month.

    Azure Blob Storage starts at $0.0184 per GB/month for hot storage, although prices decrease to $0.01 per GB per month for cool storage and $0.002 for archive storage. Prices for both services increase for greater redundancy. 

    File Storage

    Amazon Elastic File System (EFS) is an equivalent service to Azure File storage and allows users to mount files to Amazon EC2 servers. It uses NFS V4.0/4.2 to mount files and is not available in all AWS regions (currently, four are supported), requiring a Direct Connect line or VPN for on-premises mounting.

    For a comparable U.S. region, Amazon EFS costs $0.30 per GB/month and the free tier offers 5GB on sign-up. Azure File Storage prices start at $0.06 per GB/month for local redundant storage, with prices increasing for more distributed redundancy setups. It is worth noting that both Azure and AWS charge additional fees for data access.

    Conclusion

    In conclusion, Azure Storage offers a range of services for storing and managing data in the cloud, each with its own pricing structure. While the pricing varies based on several factors including storage capacity, data transfers and transaction counts, it is essential to consider other usage factors, such as region, account, access tier, replicas and egress. 

    Customers should review these factors carefully and estimate their expenses to select the right storage service that meets their needs and budget. By doing so, they can leverage Azure Storage’s flexibility and scalability to store and manage their data efficiently in the cloud.

  • Using Microsoft Azure as a Backup Repository

    Using Microsoft Azure as a Backup Repository

    Every year, the amount of data that needs to be backed up increases. There are two ways to store backups – using on-premises facilities or trusting cloud-based solutions. Having an on-premises storage infrastructure can be costly for a company. This is because of the high prices for storage equipment and the expenses to maintain this equipment, e.g., hiring a storage engineer. Depending on a company’s rules, Microsoft Azure could be an alternative to traditional ways of keeping data. If the amount of data is relatively small (less than 4 TB), there is no problem with making use of this solution; you can either use Azure Blob as a CIFS or NFS storage, or you can create your own virtual machine by adding a disk to store your data and then configuring it as a file server. However, this blog post focuses on cases where space limits are more than 4 TB.

    Microsoft Azure allows you to create VMs and add disks with a maximum capacity of 4 TB for each disk. It is suitable for application servers but very often doesn’t work if you require a big file or backup server.

    However, we can exceed this limit by using Windows software RAID array. The idea is to use RAID5 as the software RAID which contains arrays starting from three disks.

    Why RAID5? Because it has a good read speed limit. It is also economical to use X disks, where X starts at three and you only lose one disk space for the total RAID result space. Furthermore, this should give you redundancy if one of your disks were to fail. However, if we build a RAID array in Azure, we already have the virtual disks as blocks of our array and they are protected by Microsoft Azure Cloud. Thus, the chance of failure in our RAID array (which is based on disks that are already in the RAID array) falls significantly.

    Here’s how to create an Azure-based backup and replication repository server.

    1. Log in to your Microsoft Azure account or register a new account at https://azure.microsoft.com/en-us/

    2. Create a new Windows Virtual Machine:

    In this example, we choose Windows 2008 R2 SP1, since we don’t require any advanced services, just software RAID5.

    3. Type in your VM name, username and the password according to MS Azure rules of complexity.

    4. Choose your VM size. Please note that the VM type depends on the number of attached disks. Thus, if you require more than 8 disks (over 24 GB in RAID 5), the only option is to choose DS2_V2 Standard:

    5. The settings for available storage are some of the main options for our future repository:

    At this point, you have created a new VM; now it is time to add disks. On the Disks page, you can add as many disks as you need for your future disk array. In our case, we will build software RAID5, which will require seven disks in total.

    We need to have our SMB share available from the internet, so we have to configure the Azure Firewall settings.

    In the Networking section, click the Add inbound port rule button and create a new rule allowing you to make port 445 (SMB) accessible to everyone (for testing). Alternatively, you can only make the port accessible to the backup software instance’s external IP as in the Source field (for production).

    The next step is to connect to the VM via RDP using the IP and credentials you created during the VM creation process.

    Next, you need to create a RAID5 volume. In Storage Manager, you`ll see all 4 TB disks you attached. Right click Disk Management > All Tasks > New Raid 5 Volume:

    Select all available 4 TB disks to add them to your array and move them to the right column:

    Follow the standard wizard; choose a letter for the disk and perform a quick format by NTFS.

    Next, create an SMB share on a new logical disk and add write permissions to the future repository.

    At this stage, you can use your entire 24 TB volume as a remote repository in your backup system.

    If you use the NAKIVO backup to Azure solution, just add a new repository using the server’s IP path to your share as well as the credentials you set for the SMB share:

    Then, create a new job using the new repository as the backup target and run it.

    Conclusion

    The amount of data organizations need to back up is growing exponentially. Therefore, it is important to store that data in the most economical way. Using a data backup system like NAKIVO and RAID5, you can make sure your VMs are reliably protected and that there is enough space to store them in Azure.