devopsy.kinsta.cloud

Category: DevOps in the Cloud

  • Microsoft Adds Slew of Developer Tools to Azure

    Microsoft Adds Slew of Developer Tools to Azure

    At its Build 2023 conference this week, Microsoft announced general availability of an Azure Deployment Environment service that provides developers with a portal through which they can employ infrastructure-as-a-code (IaC) templates based on Terraform or Azure Resource Management files.

    In addition, Microsoft has enhanced Microsoft Dev Box, a workstation environment for developers that run Azure, to include a portal through which they can manage multiple environments.

    Scheduled to be generally available in July, Microsoft Dev Box is also previewing a capability that makes it possible to customize environments using configuration-as-code files stored in a Git repository. Microsoft is also previewing tighter integration between Dev Box and its Visual Studio integrated development environment (IDE) and Windows to further customize developers’ environments.

    Scott Guthrie, executive vice president of the cloud and artificial intelligence (AI) group at Microsoft, told conference attendees these and other tools for building cloud-native applications, along with GitHub Actions, make it simpler than ever to build applications in an Azure cloud service that includes an integrated continuous integration/continuous delivery (CI/CD) platform.

    Microsoft is also previewing GitHub Advanced Security for Azure DevOps that natively embeds automated security checks using scanning tools into the Azure DevOps platform.

    Other additions to the Azure environment include tools for centralizing the management of application programming interfaces (APIs) using an Azure API Center service that is available in preview.

    The Azure Event Grid has also been updated to enable it to use HTTP to pull delivery of discrete events in addition to pushing them.

    Finally, Microsoft is also previewing generative artificial intelligence (AI) capabilities to make it simpler to manage cloud costs by, for example, using prompts to summarize invoices or breakdown costs.

    Microsoft, along with its GitHub subsidiary, is at the forefront of applying generative AI to application development using tools like GitHub Copilot that developers can use to write better code faster. That copilot capability is now being disseminated across the portfolio of applications and services that Microsoft provides.

    It’s not yet clear what the downstream impact of such tools are just yet, but the overall quality of the applications being built should improve as, for example, fewer mistakes are made that can lead to vulnerabilities.

    Obviously, the way application software is built, constructed and secured is about to change in the generative AI era. Thanks to its alliance with OpenAI, Microsoft has a significant head start on rivals building generative AI platforms that can similarly be applied to improve developer productivity.

    In the meantime, DevOps teams should assume the volume of code simultaneously moving through pipelines is about to dramatically increase as developers become more productive. There will inevitably be more opportunities to apply multiple forms of AI to reduce any DevOps bottlenecks that might arise, but it’s clear most of the immediate benefits are being provided to developers.

    One way or another, however, the overall rate at which applications are built and deployed is about to change drastically. The issue is understanding how those advances will impact downstream DevOps workflows that were constructed for a different era of application development.

  • Red Hat Moves to Secure Software Supply Chains

    Red Hat Moves to Secure Software Supply Chains

    Red Hat today announced a portfolio of cloud services designed to better secure software supply chains. The expanded portfolio includes Red Hat Trusted Application Pipeline to secure continuous integration/continuous delivery (CI/CD) workflows and Red Hat Trusted Content offerings to better secure software components.

    Announced at the Red Hat Summit, the Red Hat Trusted Software Supply Chain combines a suite of curated open source tools into a cloud service managed by Red Hat. The Red Hat Trusted Application Pipeline service is based on sigstore, an emerging de facto standard for cryptographically signing code to ensure immutability that Red Hat helped drive, while Red Hat Trusted Content provides access to a set of software components that Red Hat has vetted.

    Red Hat Trusted Software Supply Chain also provides access to existing tools such as Quay, a registry for containers, and the Advanced Cluster Security (ACS) platform for securing Kubernetes clusters.

    Red Hat Trusted Software Supply Chain is scheduled to be made available in preview in the next few weeks, and will also track the provenance of components. In addition, the service will monitor emerging threats to software components and alert users of new and emerging risks. The service will also surface recommendations for remediating vulnerabilities it identifies based on best practices defined by Red Hat.

    Finally, Red Hat Trusted Application Pipeline also promises to make it simpler to automate the deployment of secure code with a few clicks within the context of an integrated continuous integration/continuous delivery (CI/CD) pipeline. DevOps team can also import Git repositories and configure container-native continuous build, test and deployment pipelines, inspect source code and transitive dependencies, autogenerate software bill of materials (SBOMs) within builds and verify and promote container images. An enterprise contract policy engine can consistently enforce compliance with the Supply Chain Levels for Software Artifacts (SLSA) framework.

    Finding ways to implement DevSecOps workflows is a major challenge for any organization. Red Hat is making a case for a managed cloud service based on a collection of security software and services that a DevOps team would otherwise have to implement themselves.

    Tasked with securing increasingly complex software development environments, it’s not clear whether DevOps teams will opt to rely on a cloud service, but given the general lack of security expertise among software development teams, the fastest way to secure a software supply chain for most organizations is to rely on a service.

    Regardless of approach, of course, DevSecOps will soon become a standard element of application development workflows. Once that occurs, it’s not clear whether DevOps will simply become DevSecOps or whether security will be assumed to be an integrated element of any set of DevOps best practices.

    Sarwar Raza, vice president and general manager for cloud services at Red Hat, said regardless of approach, it’s clear that implicit trust is no longer good enough. Organizations of all sizes will need to be able to demonstrate that they have maintained a chain of custody for the software components used to build an application, he noted.

    One way or another, the overall security of downstream applications should eventually improve as the processes used to construct software are increasingly locked down.

  • OpenAI to go Open Source — Elon Musk was a ‘Huge Idiot’ ¦ Mojo Risin’

    OpenAI to go Open Source — Elon Musk was a ‘Huge Idiot’ ¦ Mojo Risin’

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Sources say it’s an open source future for OpenAI, and improving AI perf with Mojo. (more…)

  • ServiceNow Adds Observability Platform to SaaS Portfolio

    ServiceNow Adds Observability Platform to SaaS Portfolio

    At its Knowledge 2023 conference, ServiceNow launched an observability service that runs on its software-as-a-service (SaaS) platform.

    Based on the observability platform the company gained with its acquisition of Lightstep, the ServiceNow Cloud Observability platform makes it possible to collect and analyze data collected from software running on infrastructure and also from workflows created using ServiceNow applications.

    Pablo Stern, senior vice president and general manager for technology workflows products at ServiceNow, said the goal is to make it simpler to investigate IT issues involving applications and workflows that have multiple dependencies. That makes it possible to quickly diagnose an issue, determine the root cause and then dispatch the right site reliability engineering (SRE) team to troubleshoot it, he added.

    ServiceNow Cloud Observability uses both open source OpenTelemety agent software to collect logs, metrics and traces in addition to a log management tool that ServiceNow gained last year with its acquisition of Era Software.

    In addition, ServiceNow is employing a Service Graph Connector that makes it simpler to pull data from, for example, OpenTelemetry and Kubernetes clusters into the ServiceNow IT Operations Management (ITOM) without requiring any additional tooling. ServiceNow is also making available its Unified Query Language to make it possible for IT teams to query the data residing in ServiceNow Cloud Observability.

    As application environments become more complex, the need for entire IT teams to have access to an observability platform is growing. While observability has always been a core tenet of DevOps best practices, Stern said it’s clear that these capabilities also need to be made available to IT administrators. In fact, as observability becomes more democratized, it will drive the rise of services operations that will enable IT teams to be better aligned around specific processes, he added.

    Most IT organizations today employ a wide range of tools that enable them to monitor a set of pre-defined metrics. Observability platforms, in contrast, enable IT teams to collect data from systems in a way that can be investigated using a query tool.

    It’s still early days as far as the adoption of observability platforms is concerned, but as the collection of logs, metrics and traces becomes more unified, an opportunity to better converge IT practices based on DevOps principles and the ITIL framework that many IT administrators rely on to manage IT will emerge.

    The challenge, regardless of what methodology is used, is that IT teams need to be able to frame the queries they use to discover the root cause of an IT issue before any potential disruption occurs. In the long term, however, it’s expected that machine learning algorithms will leverage the data collected by observability platforms to automatically identify those issues.

    Regardless of how an IT issue arises, the metric that matters most is how quickly it can be resolved—no matter who in the IT team is tasked with resolving it. The issue is finding a way to provide IT teams with better tools to achieve that goal.

  • Quick Guide to Azure Storage Pricing

    Quick Guide to Azure Storage Pricing

    Azure Storage is a cloud-based storage service from Microsoft Azure. It provides a scalable, secure and highly available data storage solution for businesses of any size.

    The platform includes various services such as Azure Blob, which is a service for storing large amounts of unstructured data such as documents, images, videos and backups. Azure Files is a fully managed file share service for cloud-based applications. Azure Tables is a NoSQL key-value store used to store semi-structured data. Azure Queues is a messaging service used to store and process messages between applications or services.

    Finally, Azure Disks is a service that allows you to store and manage data in virtual hard disks used in virtual machines. With Azure Storage, businesses can easily manage their data, control access and scale their storage as per their needs without worrying about hardware or infrastructure management.

    How does Azure charge for storage space (bandwidth, capacity, number of transactions and quantity of transferred data)? What are the usage factors impacting Azure Storage fees: Region, account, access tier, replicas, egress?

    How Azure Storage Pricing Works

    Azure pricing for its storage services is based on several factors including storage capacity, data transfers and transaction counts. The cost of storage varies depending on the region and access tier selected, as well as the type of storage being used.

    For Azure Blob Storage, customers are charged for storage capacity, data egress (outbound data transfers) and the number of transactions (read, write and list operations).

    Azure Table Storage charges are based on the number of table entities stored and the amount of data transferred. Azure Queues are charged per transaction and data transfer, while Azure Disks are billed based on the provisioned capacity and the number of transactions.

    Several usage factors can impact Azure Storage fees, including the region where the data is stored, the storage account type, the access tier selected, the number of replicas and the amount of data transferred out of the storage account. 

    Overview of Azure Storage Pricing

    Blob Storage

    Azure Blob Storage pricing varies depending on the storage tier, data volume and whether customers opt for pay-as-you-go or reserved capacity.

    For pay-as-you-go pricing, Premium storage is charged at a flat rate of $0.015 per GB/month, while Cool and Archive storage are billed at $0.01 and $0.0099 per GB/month, respectively, regardless of data volume. Hot storage costs $0.018 per GB /month for the first 50 TB with the rate gradually decreasing with higher usage. For instance, customers who consume more than 500 TB/month pay $0.0166 per GB.

    Customers can also reserve blob storage in 100-TB or 1-PB blocks for one or three years. The prices for reserved storage vary by tier. A three-year Archive storage reservation costs $84 per 100 TB or $810 per PB/month, while Hot storage costs $1,244 per 100 TB/month for three years. For one-year reservations, Archive storage costs $91 per 100 TB or $883 per PB/month, and Hot storage costs $1,545 per 100 TB/month. For a one-year reservation, the cost of Hot storage is $15,050 per PB/month, and $11,963 per month for three years.

    Files

    Microsoft charges customers for Azure Files depending on the storage tier and whether they reserve storage capacity. Pay-as-you-go pricing ranges from $0.015 per GB/month for Cool storage to $0.016 per provisioned GiB/month for Premium file storage.

    Customers can commit to reserved storage capacity in increments of 10 TiB or 100 TiB for one or three years. A three-year reservation of 10 TiB Cool storage costs $101.39 per month, while one year costs $125.92 per month. However, the price increases significantly for the Premium offerings, with a three-year reservation of 10 TiB costing $1,081.34 per month and a one-year reservation costing $1,343.50 per month. 100 TiB reservations range from $983/month in the Cool tier for three years to $12,779.50/month in the Premium tier for one year.

    Nevertheless, there are other fees that add to the total cost such as read and write transactions, snapshots, metadata storage and early file deletion fees. These ancillary fees can impact the overall cost of using Azure Files and customers should account for them when estimating their expenses.

    Tables

    Azure Tables pricing is available in various tiers, including locally redundant storage (LRS), geo-redundant storage (GRS), read-access geo-redundant storage (RA-GRS), zone-redundant storage and geo-zone-redundant storage. 

    Customers are charged between $0.045 per GB/month for LRS and $0.01265 per GB/month for geozone-redundant read-only storage. Additionally, Microsoft charges a $0.0036 fee for each set of 10,000 transactions made on Azure Tables.

    Queues

    Compared to Blob and File Storage, the pricing model for Azure Queues is more straightforward. Customers are charged 4.5 cents to $0.075 cents per GB/month, depending on whether they choose local redundant storage (LRS) or read-access geozone-redundant storage (RA-GRS). Additionally, there is a $0.004 fee for every set of 10,000 operations in Class 1 and 2.

    Managed Disks

    Azure Managed Disks have numerous pricing options that can be overwhelming. The cost of using Managed Disks depends on several factors, including the disk size, performance, reservation period, capacity and disk type.

    The cheapest option for Managed Disks is the P1 disk, which costs $0.6 per month for 4 GiB of capacity in addition to $0.03 per mount/month. On the other hand, the P80 disk is the most expensive option, as it has a 32 TiB capacity costing $3,276.80 per month, in addition to $219/mount. However, the cost of a P80 disk can be reduced to $3,113/month (plus the $219/mount fee) with a reservation of one year.

    Azure Storage Vs. AWS Storage Pricing

    Here is a comparison of the pricing for similar storage services offered by Azure and AWS. As part of an Azure or AWS cost optimization strategy, you might consider combining services from both providers.

    Block Storage

    Amazon EBS is Amazon EC2’s basic storage device, with a price of $0.045 per GB for HDD and $0.10 per GB for SDD, with higher IOPS resulting in higher prices. The free tier includes 30 GB of SDD.

    For comparable regions in the U.S., Azure charges $0.05 per GB for HDD and $19.71 for 128 GB of SSD per month.

    Azure Managed Disks offer greater flexibility and easier backup than Azure Virtual Disks or Amazon EBS, with adjustable volume sizes and available IOPS. Managed Disks pricing starts at $19/month for a 128 GB disk, providing 500 IOPS.

    Object Storage

    Amazon S3 and Azure Blob Storage are both scalable services for storing objects (unstructured data) where data is stored together and cannot be changed in parts. Prices vary depending on the region, but for a comparable region in the U.S., Amazon S3 storage costs $0.023 per GB/month for the first 50 TB in addition to network usage fees. The free tier offers 5 GB of storage, while archive storage (Glacier) is $0.004 per GB/month.

    Azure Blob Storage starts at $0.0184 per GB/month for hot storage, although prices decrease to $0.01 per GB per month for cool storage and $0.002 for archive storage. Prices for both services increase for greater redundancy. 

    File Storage

    Amazon Elastic File System (EFS) is an equivalent service to Azure File storage and allows users to mount files to Amazon EC2 servers. It uses NFS V4.0/4.2 to mount files and is not available in all AWS regions (currently, four are supported), requiring a Direct Connect line or VPN for on-premises mounting.

    For a comparable U.S. region, Amazon EFS costs $0.30 per GB/month and the free tier offers 5GB on sign-up. Azure File Storage prices start at $0.06 per GB/month for local redundant storage, with prices increasing for more distributed redundancy setups. It is worth noting that both Azure and AWS charge additional fees for data access.

    Conclusion

    In conclusion, Azure Storage offers a range of services for storing and managing data in the cloud, each with its own pricing structure. While the pricing varies based on several factors including storage capacity, data transfers and transaction counts, it is essential to consider other usage factors, such as region, account, access tier, replicas and egress. 

    Customers should review these factors carefully and estimate their expenses to select the right storage service that meets their needs and budget. By doing so, they can leverage Azure Storage’s flexibility and scalability to store and manage their data efficiently in the cloud.

  • Using Microsoft Azure as a Backup Repository

    Using Microsoft Azure as a Backup Repository

    Every year, the amount of data that needs to be backed up increases. There are two ways to store backups – using on-premises facilities or trusting cloud-based solutions. Having an on-premises storage infrastructure can be costly for a company. This is because of the high prices for storage equipment and the expenses to maintain this equipment, e.g., hiring a storage engineer. Depending on a company’s rules, Microsoft Azure could be an alternative to traditional ways of keeping data. If the amount of data is relatively small (less than 4 TB), there is no problem with making use of this solution; you can either use Azure Blob as a CIFS or NFS storage, or you can create your own virtual machine by adding a disk to store your data and then configuring it as a file server. However, this blog post focuses on cases where space limits are more than 4 TB.

    Microsoft Azure allows you to create VMs and add disks with a maximum capacity of 4 TB for each disk. It is suitable for application servers but very often doesn’t work if you require a big file or backup server.

    However, we can exceed this limit by using Windows software RAID array. The idea is to use RAID5 as the software RAID which contains arrays starting from three disks.

    Why RAID5? Because it has a good read speed limit. It is also economical to use X disks, where X starts at three and you only lose one disk space for the total RAID result space. Furthermore, this should give you redundancy if one of your disks were to fail. However, if we build a RAID array in Azure, we already have the virtual disks as blocks of our array and they are protected by Microsoft Azure Cloud. Thus, the chance of failure in our RAID array (which is based on disks that are already in the RAID array) falls significantly.

    Here’s how to create an Azure-based backup and replication repository server.

    1. Log in to your Microsoft Azure account or register a new account at https://azure.microsoft.com/en-us/

    2. Create a new Windows Virtual Machine:

    In this example, we choose Windows 2008 R2 SP1, since we don’t require any advanced services, just software RAID5.

    3. Type in your VM name, username and the password according to MS Azure rules of complexity.

    4. Choose your VM size. Please note that the VM type depends on the number of attached disks. Thus, if you require more than 8 disks (over 24 GB in RAID 5), the only option is to choose DS2_V2 Standard:

    5. The settings for available storage are some of the main options for our future repository:

    At this point, you have created a new VM; now it is time to add disks. On the Disks page, you can add as many disks as you need for your future disk array. In our case, we will build software RAID5, which will require seven disks in total.

    We need to have our SMB share available from the internet, so we have to configure the Azure Firewall settings.

    In the Networking section, click the Add inbound port rule button and create a new rule allowing you to make port 445 (SMB) accessible to everyone (for testing). Alternatively, you can only make the port accessible to the backup software instance’s external IP as in the Source field (for production).

    The next step is to connect to the VM via RDP using the IP and credentials you created during the VM creation process.

    Next, you need to create a RAID5 volume. In Storage Manager, you`ll see all 4 TB disks you attached. Right click Disk Management > All Tasks > New Raid 5 Volume:

    Select all available 4 TB disks to add them to your array and move them to the right column:

    Follow the standard wizard; choose a letter for the disk and perform a quick format by NTFS.

    Next, create an SMB share on a new logical disk and add write permissions to the future repository.

    At this stage, you can use your entire 24 TB volume as a remote repository in your backup system.

    If you use the NAKIVO backup to Azure solution, just add a new repository using the server’s IP path to your share as well as the credentials you set for the SMB share:

    Then, create a new job using the new repository as the backup target and run it.

    Conclusion

    The amount of data organizations need to back up is growing exponentially. Therefore, it is important to store that data in the most economical way. Using a data backup system like NAKIVO and RAID5, you can make sure your VMs are reliably protected and that there is enough space to store them in Azure.

  • AWS Converts Cedar Policy-as-Code Tool to Open Source Project

    AWS Converts Cedar Policy-as-Code Tool to Open Source Project

    At the Open Source Summit North America conference today, Amazon Web Services (AWS) announced it is making Cedar, a language for defining permissions as policies that includes automated reasoning to mathematically prove an IT environment is secure, available as an open source project.

    In addition, AWS launched SnapChange, an open source fuzz testing tool that cybersecurity researchers can employ to discover vulnerabilities by replaying physical memory snapshots in a KVM virtual machine. Fuzz testing uncovers software security issues by monitoring how the system behaves while processing random data.

    David Nalley, director of open source marketing for AWS, said both projects are part of an ongoing AWS effort to contribute intellectual property to the open source community that plays a pivotal role in the applications that organizations deploy on the AWS cloud.

    AWS has been employing Cedar to provide IT organizations with the ability to write authorization policies as code that can be deployed anywhere. It is most widely used within the managed AWS Verified Permissions service that manages those processes on behalf of organizations that deploy workloads on its cloud.

    Cedar differs from other policy-as-code approaches because it employs automated reasoning, a computer science discipline that is considered a subset of artificial intelligence (AI), to enable machines to prove theorems and checks. That latter capability is especially critical to enable organizations to mathematically prove that compliance mandates have been continuously met, noted Nalley.

    Otherwise, IT organizations become wholly dependent on an assessment of intermittent audits conducted by humans that are likely to arrive at varying subjective conclusions, he noted.

    The latest open source contributions from AWS come at a time when there is more scrutiny being applied to how open source software is developed and employed. The scrutiny increased in the wake of a series of vulnerability disclosures involving open source software, such as the Log4j monitoring software project.

    The maintainers of many open source software projects often don’t have the resources or capabilities required to immediately patch a vulnerability every time it is discovered. That’s become especially problematic, because more research into open source software vulnerabilities is being conducted; it’s only a matter of time before a vulnerability is detected in software that was largely created by developers that have limited cybersecurity expertise. Enterprises that consume open source software need to pay careful attention to what dependencies they are creating when that software gets incorporated into an application environment. A patch for a zero-day vulnerability that is discovered might not be readily available.

    Hopefully, consumers of open source software will become more involved in its development by contributing code or making available the financial resources that maintainers need to devote time and effort to creating a timely patch to a vulnerability. In the meantime, however, they should also be reviewing what open source software is included in applications running in production environments. Cybercriminals are increasingly looking to discover and exploit vulnerabilities in not just existing applications but not also in components they know are likely to be employed across a wide range of downstream applications.

    As such, finding ways to apply more cybersecurity expertise to application development by embracing DevSecOps practices has become an imperative rather than a recommendation.

  • Grafana Labs Reins in Cost of Storing Observability Data

    Grafana Labs Reins in Cost of Storing Observability Data

    Grafana Labs today made available an ability to customize what observability data is collected via its managed cloud service. This capability can help reduce storage costs in addition to enhancing the dashboard Grafana provides to make it simpler to track metrics usage.

    Wayne Jin, vice president of product marketing for Grafana Labs, said both capabilities would make it easier for DevOps teams to track and ultimately reduce the cost of storing the time-series data collected via its cloud platform.

    As more DevOps teams use an observability platform to continuously collect metrics like traces and log data, they are discovering that the cost of storing all that high-cardinality data can easily spin out of control.

    Grafana Labs is now making available an Adaptive Metrics capability that enables DevOps teams to fine-tune what data is collected and stored. However, in the event of an issue, DevOps teams can also turn back on all data the platform can collect to aid an investigation of the root cause of disruption, noted Jin.

    The Cardinality Management dashboards Grafana Labs made available last year for paid tiers of its cloud service are now being made available to users of the free tier of service as well, Jin said. This will further help reduce storage costs by making it easier to identify metrics that are being collected but not used by anyone on the DevOps team, he added.

    The Adaptive Metrics aggregation engine can then be applied to transform these metrics at ingestion into lower cardinality versions. Unused or partially used labels are stripped from incoming metrics, reducing the total count of time-series data collected. Adaptive Metrics also recommends aggregations based on an organization’s historic usage patterns, and DevOps teams can choose which aggregation rules to apply. Dashboards, alerts and historic queries are guaranteed to continue to work as they did before aggregation, with no rewrites needed.

    Based on results reported by early users, Grafana Labs reported that Grafana Cloud Adaptive Metrics could eliminate an estimated 20-50% of the time-series data collected with no perceived impact on observability.

    The Grafana Cloud service relies on an instance of open source Grafana Mimir to store data collected in a format originally defined for the open source Prometheus monitoring tool. While Prometheus is widely employed in Kubernetes environments, it’s also starting to gain traction as a monitoring platform in legacy monolithic environments. The amount of data being collected via that platform over time is steadily increasing. The Grafana Labs approach makes it possible for DevOps teams to reduce that data whenever there is no usage that needs to be investigated, for example.

    It’s still early days as far as adoption of observability is concerned, but in uncertain economic times, there’s generally a lot more sensitivity to the total cost of IT. Many organizations are especially focused on reducing the cost of storing data in the cloud to rein in monthly spending on cloud services. Regardless of the motivation, collecting and storing data that isn’t needed is never the best use of cloud storage resources that, from a cost perspective, are just as finite in the cloud.

  • Consider Performance, Growth & Budget When Buying Data Analytics

    Consider Performance, Growth & Budget When Buying Data Analytics

    So, your business needs to invest in data analytics technology to improve efficiencies, competitive advantage or business outcomes. There are a ton of things to consider, but underlying each decision are three driving factors:

    • Performance – The need to meet service level agreements.
    • Growth – The need to accommodate success, as well as deal with inevitable industry changes.
    • Budget – The need to do both of the above while keeping costs low enough to not erode the profit gain from analytics.

    These three forces underpin nearly every technology selection decision, but what frequently isn’t noticed is how much these three drivers interact.

    1. Concurrency and Growth

    Test both for current levels of concurrency and the level you hope to achieve in the next few years as the organization grows.

    When evaluating benchmarks between vendors, consider that benchmarks are designed to make that company’s software look good. For example, if they are good at concurrency, you will see various levels of concurrency represented in the benchmark. If the benchmark shows only one workload at a time, or maybe ten, you can safely assume that concurrency is probably not something that technology is good at. Similarly, if the benchmark is done at 10 TB scale, the vendor probably isn’t that good at high-scale analytics.

    When bringing vendors in for a proof-of-concept, one mistake I’ve seen (more times than I can count) is testing the capabilities of the software with only one user at a time. People do not line up politely to use analytics one at a time.

    In addition to current levels of concurrency, over time, hopefully, the organization will grow, and the number of both internal analytics users and external customers will increase. Testing for the number of users you have now may not uncover problems that will invariably reveal themselves when a lot of people hit the system at once. The last thing your organization needs is to turn away customers because they’re overloading your systems.

    2. Performance & Price

    Evaluate price/performance, not performance alone.

    Everyone understands the importance of performance. Not one user has ever gone to a data architect or engineer and requested for their analytics to “please execute more slowly.” But performance isn’t simply how fast the analytical software can process queries, how fast the data pipelines can move data or how fast the response is between data input and reaction output. There’s also a monetary aspect of performance to bear in mind.

    In a recent benchmark, three data management and analytics stacks were compared with 60 simulated concurrent users/workloads. The difference in queries per hour on 250 TB of data was minor, perhaps 16 queries per hour for the slowest and 23 queries per hour for the fastest.

    (Mcknight Benchmark – SaaS Data Analytics Platform Comparison – vendor names removed)

    It looks like performance is just not that big a consideration for this technology. But there is no such thing as an unlimited budget. Let’s look at how much it would cost to get this level of performance out of each stack.

    (Mcknight Benchmark – SaaS Data Analytics Platform Comparison – vendor names removed)

    Price-performance is a far more useful metric of value to the organization. How much the software can do is irrelevant if the budget limits how much the business can afford to do with it.

    3. Software Efficiency

    Consider how much efficiency and control you’re willing to trade for ease-of-use.

    When you analyze data in the cloud, you’re not just paying for software, you’re also paying for hardware, a.k.a. cloud compute instances. Bundling the two costs in one bill makes things easier. But it also means that cloud providers make more money the more instances, or compute nodes, you use. You trade ease-of-use in billing for a higher price. And they no longer have a financial incentive to provide efficient software.

    Similarly, to make a cloud-based analytics software scale up in performance and concurrency with zero effort on your part, the software has to use only defaults for all settings internally. It can scale only by throwing more compute at the problem; compute that you’re paying for every time you use it. You’re trading control of the software–tuning, scaling guidelines, shut down conditions, etc.–for ease-of-use, again for a higher price.

    Throwing more compute at a problem isn’t always the best solution. Recently, I saw a POC where the customer implementation included 278 EC2 nodes on Amazon cloud. The organization didn’t initiate the POC because of inefficiency. They were having performance problems, even with that much compute power. A competing product was able to run on only nine EC2 nodes with the same workload and better performance. The company was paying 25 times as much for the compute hardware because it was convenient.

    Be aware of the tradeoffs.

    4. Affordable Scalability

    Paying for only what you use can become impractical when you use more and more.

    The volume of data that is being created, consumed and captured is growing exponentially. By 2025, it is estimated that nearly 30% of data being generated will be real-time. Instead of pulling data in batches, data will be pulled in continuously. Queries will be responded to in seconds, not hours, or in milliseconds, not seconds.

    Going to the cloud can make tremendous sense when a company or a workload in that company is new. But studies show that as businesses scale and growth slows, the pressure the cloud puts on business margins eventually outweighs its benefits. This is why most enterprises today are considering repatriating some or all of their workloads off the cloud.

    5. Flexibility for Change

    Make certain your stack will continue to work even as conditions change, and you won’t have to rebuild from scratch to do the same thing in a different location.

    When you’re building a long-term architectural strategy, you want to ensure you cover as many bases as possible because, the fact is, you really don’t know what the future holds. Ten years ago, maybe you had a Hadoop on-premises strategy or a data warehouse appliance. Today, you may have a cloud-focused strategy. Tomorrow, maybe you’ll move some or all of those workloads back on-premises due to performance, security, costs, regulations or other reasons.

    This is why you want flexibility that doesn’t mandate a “cloud-only” or an “on-premises-only” or a “this-cloud-only” architecture and which doesn’t integrate with only the same vendors’ software. You want an architecture that can work equally well both on the cloud as well as on-premises, hybrid or multi-cloud, providing the flexibility of going where business needs are best met.

    Containerization offers possibly the highest level of flexibility in deployment, but it can also add to deployment complexity, so again, be aware of what you’re trading. If the software you choose is platform-agnostic—not tied to any particular location, ecosystem or deployment model—then that will make future choices a lot more flexible.

    Regardless of what analytics you choose, what you decide to build or what use cases you have to meet, striving for balance in performance, growth and budget will create a strong data architecture, now and for the future:

      1. The analytics architecture is high-performance: it should meet SLAs, get things done on time and meet or exceed expectations and requirements.
      2. The analytics architecture is resilient to growth and change; even if there is rapid growth or sudden or extreme changes, it’s future-ready.
      3. The analytics architecture should control costs and live within an allocated budget.

    Keep these guidelines in mind when choosing an analytics provider. The decisions you make today will have direct ramifications on the growth, profitability and success of the business tomorrow.

  • Survey: Much Greater Focus on Reining in IT Costs

    Survey: Much Greater Focus on Reining in IT Costs

    A survey from Vega Cloud found 62% of respondents are more closely monitoring all IT costs, with more than half (52%) more closely evaluating their cloud service provider options.

    The survey polled 100 vice presidents from organizations with more than 1,000 employees and was published today by Vega Cloud, a provider of a platform for managing multiple clouds.

    Overall, nearly half of respondents (48%) said they expected to reduce their IT spending a little or a lot in 2023, with 65% identifying the reduction of cloud spending as a specific goal. Three quarters (75%) are looking to cut cloud costs by 10-50% in 2023, the survey found.

    A full 93% reported they have invested at least a fair amount of money on tools to optimize cloud costs, with 82% having invested in FinOps staff or people with FinOps experience. The challenge is that 70% of respondents said it was difficult to find people with these skill sets, the survey found.

    The best way for organizations to reduce cloud costs is to become more informed about what they’re spending (40%), and 33% said that the best approach is to consistently optimize use of cloud resources. A total of 61% identified reserved instances/savings plans as the best way to optimize cloud use, while 55% said optimization was best achieved through waste elimination followed closely by right-sizing (54%).

    Vega Cloud CEO Kris Bliesner said it’s clear the days when organizations allowed developers to consume cloud resources at will are coming to an end as cloud services consume a much larger percentage of the IT budget. In fact, the survey noted that 91% of respondents are spending at least one million dollars annually on cloud infrastructure alone.

    The issue is that finance departments not only want to reduce cloud spend but also want cloud spending to become more predictable from a budget perspective. Nearly half of survey respondents (45%) said they’d received a surprise in their cloud bills “a number of times” over the past 12 months, with 70% of respondents noting surprises have happened at least a half dozen times in the past year.

    A total of 40% admitted they had exceeded budgets multiple times in the past year, and 38% said they had to modify their forecast cloud spending budget numerous times over the past 12 months. More than half of respondents (55%) said database costs are the metric that matters most when looking at cloud usage followed by bandwidth costs (51%), storage costs (49%) and compute costs (47%). Nearly two-thirds (64%) said they track metrics at least somewhat often, while 46% said they have somewhat optimized their cloud use. A total of 11% said they’re not optimized at all.

    A full 88% said they’re using artificial intelligence (AI) at least somewhat to optimize cloud use and costs, with nearly three-quarters (74%) noting they think ChatGPT or a similar generative AI tool could be used to optimize cloud use/costs further. Specifically, 42% said they thought it could improve decision making and 30% said it could automate decision making.

    Among the cloud service providers, Google is cited as doing the most to help customers manage and reduce cloud costs (38%), followed by Amazon Web Services (34%) and Microsoft (28%)—but a full 85% said they could all be doing more.

    Of course, changing economic conditions are naturally driving organizations to look harder at cloud spending. Well over half of respondents (56%) said they are concerned about the economy, and 95% of those respondents said those concerns have, at least somewhat, impacted their IT strategy.

    It’s not clear whether organizations will continue to exercise greater control over cloud spending should the economy recover. For now, at least, there’s clearly more desire for control than ever.