devopsy.kinsta.cloud

Tag: data management

  • There’s no One-Size-Fits-All Data Mobility Strategy

    There’s no One-Size-Fits-All Data Mobility Strategy

    Whether you’re a Fortune 500 enterprise or a fledgling startup, there’s one thing every business can expect to confront: Change. This is resoundingly true when it comes to data management. Even greenfield operations will, sooner rather than later, need to change how and where they leverage their data. After all, every business—including those launched with the most cutting-edge technology—is built on a series of choices and compromises. As technologies and business priorities change, so should your data mobility strategy.

    In a world of ever-changing variables, what’s the right data mobility strategy? There is no one-size-fits-all solution. It does, however, help to start with a solid foundation on which to build your strategy.

    With the right platform that allows for flexibility and change, it’s time to build the structural framework for your data mobility strategy. When creating the framework for your data mobility strategy, there are a few considerations to keep in mind, including your desired use of on-premise storage, your hybrid cloud strategy and “supercomputing” capabilities, and the storage area network (SAN) needed based on your organization’s needs. Let’s review what they might mean for you:

    Developing a Hybrid Cloud Strategy

    The reality is that the hybrid cloud is now achievable for most organizations. According to Flexera’s 2023 State of the Cloud Report, 72% of organizations have embraced a hybrid cloud strategy. Your data mobility strategy should reflect that.

    With a proper hybrid cloud environment, the user experience should be seamless regardless of where your data exists, whether that’s on-premise, in a private cloud, or in the public cloud. As the potential for more dynamic resource allocation grows, your data mobility strategy inherently becomes more critical. Data mobility, in effect, now equates to computing mobility.

    Leveraging the “Supercomputer” in the Cloud

    Data mobility becomes even more imperative as entities gain more opportunities to take advantage of “supercomputing” capabilities.

    The computing power that exists in the cloud now is tremendous, and thanks to modern dynamic cloud pricing, even smaller organizations can tap into it. According to one estimate, the cloud-based high-performance computing market should reach $55 billion by 2030, growing at a CAGR of 7% from 2023 to 2030.

    So for example, if you’re a FinTech startup, you could pay by the minute to leverage high-performance computing capabilities to test your trading algorithm. Previously, these powerful capabilities were only available to the biggest customers — think NASA — and came with astronomical costs.

    On-Premise Solutions Aren’t Going Away

    Even as cloud services and capabilities become more accessible and more valuable, on-premise operations aren’t going to disappear. Customers have all kinds of reasons for keeping data on-premise, whether it’s for compliance purposes, costs or the desire to leverage existing infrastructure. Certain workloads, meanwhile, simply don’t make as much sense on the cloud.

    The reality is that even the most compelling alternatives couldn’t eliminate on-premise operations, because business ecosystems and industries evolve organically. Think about it: Many businesses still use landlines, and workers still use desktop computers. Nearly half a century after marketers first coined the term the “paperless office,” it took a full-blown pandemic to push our collaboration tools to the point where we are finally, basically, paperless.

    Fiber Channel and iSCSI

    When it comes to data mobility, how does an organization decide whether to adopt fiber, iSCSI or both? As organizations manage workloads that require low latency and other networking capabilities, they’re going to be investing heavily in this area.

    Traditionally, a significant amount of data management is fiber-channel-based. With iSCSI, an organization doesn’t need to invest in separate physical infrastructure, such as switches or cables. Moreover, iSCSI has become more attractive to entities moving workloads to the cloud. A whole range of products, from Pure Storage, NetApp and even cloud platforms like Microsoft Azure, offer iSCSI-based storage in the cloud. Your data mobility strategy should consider how to move from fiber channel to iSCSI and back again.

    Change is the Only Constant

    Ultimately, no one can predict all of the major technological innovations that will impact their business down the road. To stay nimble, you need a data mobility strategy that you can execute with the team you have — whether their primary expertise is in storage, networking or elsewhere. This can be especially challenging in large organizations, with teams that have to meticulously plan their projects and stay accountable for their budgets. Now more than ever, it’s important to have a solid foundation on which to build a strategy that fits your needs.

  • Data Management: Cloud Technology or Data Centers?

    Data Management: Cloud Technology or Data Centers?

    Traditionally, larger enterprises have relied on in-house server hardware equipment for their data management needs, optimized to run high-performing applications with a certain degree of assured security and reliability. 

    However, the high costs and technical expertise required to manage these solutions effectively often render them inaccessible to startups and emerging businesses. But, as the old saying goes, necessity is the mother of invention. 

    With the growing demand for cost-effective, efficient data management solutions, the tech industry has witnessed the advent of cloud technology and data center colocation rental services. These alternatives pave the way for startups and small businesses to handle large volumes of data without straining their budgets.

    This article will compare and contrast cloud technology and data center services to help business leaders make well-informed decisions about their data management needs and the solutions that can help them.

    Understanding Data Center Colocation Services

    Data center colocation services offer an enticing solution for companies that require robust data management capabilities without incurring high capital expenditure for securing storage hardware, along with storing and powering them. 

    Essentially, this solution is like renting a space for your data in a specialized facility. These data centers provide a secure environment with the necessary power, cooling, and physical security for your server, storage, and networking equipment.

    In opting for colocation services, companies can leverage state-of-the-art data center infrastructure without the burden of high upfront costs and ongoing maintenance responsibilities associated with owning a private data center. This also allows businesses to focus more on their core competencies rather than the intricacies of data center management.

    However, data centers come with their own set of unique challenges. For instance, they often require a certain degree of technical expertise on the part of the business to manage their servers. This means hiring or training staff to handle server management, increasing the overall cost of this solution.

    Additionally, data centers’ effectiveness can sometimes be influenced by physical proximity. The closer your operations are to the data center, the better the performance can be. This geographical factor can pose problems for businesses operating in multiple locations or managing data across different data centers.

    The Power of Cloud Technology

    In contrast to relying on data centers, cloud technology offers businesses of all kinds and sizes immense scalability and flexibility in terms of data management solutions. 

    As a service that delivers computing infrastructure, software, or storage as a service via a network connection, typically the Internet, cloud technology eliminates the need for physical proximity to servers. It is quickly scalable, both upwards and downwards, as per business needs, ensuring that you only pay for what you use.

    Moreover, the ability of cloud technology to integrate various services can simplify business operations and enhance customer engagement. For instance, many businesses are now offering cloud-based text-to-pay systems that allow customers to pay from any device, providing ease of transaction while simultaneously gathering customer data in the cloud.

    Cloud services can also provide robust disaster recovery and business continuity solutions. However, businesses often cite concerns about data security and privacy in the cloud. Although most cloud service providers implement best-in-class security protocols, the very nature of the cloud—being an off-premises solution—necessitates a high level of trust in the provider.

    AI and Data Management Solutions

    If your business relies heavily on AI applications, you must understand how your data management solution supports this need. AI requires high-quality, well-managed data to function effectively. Here, cloud technology often presents an edge over traditional data centers.

    Many cloud service providers offer built-in tools and services designed to better facilitate AI implementation, including pre-trained machine learning models and AI platforms as a service. These resources can dramatically reduce the time, costs, and complexity that come with implementing AI, particularly for small to mid-sized businesses without extensive in-house AI expertise.

    On the other hand, while data centers might not have these built-in tools, they offer robust performance capabilities for AI workloads and often provide more control over the data, which is vital for sensitive AI applications.

    Keep in mind you can turn to other server-based solutions for storing data as well. For example, if your application relies on a significant amount of PDF content to store data and function effectively, hosting those files on a server such as a server-side PDF SDK will allow you to process those documents on the server via an API. This will then allow you to autoamte your document processing and convert large volumes of your PDF files. 

    Web Hosting Requirements

    Your current web hosting requirements can also significantly impact the choice between data center services and cloud technology. Shared hosting services may be sufficient for businesses with low-traffic websites or applications, and if you don’t need to store a lot of data, but as your traffic increases and the website grows, so do your hosting requirements. 

    Cloud hosting solutions offer excellent scalability, allowing your website to grow with your business. As traffic surges, cloud servers can easily handle the increase, maintaining website performance and user experience.

    Conversely, with a dedicated data center, you’re generally limited to the capacity of the server you rent. To increase your capacity, you may need to physically upgrade the hardware, a process that can result in downtime and potential loss of revenue.

    Considering Cost Implications

    The overall cost of your chosen solution will be a key factor when deciding between data center colocation services and cloud technology. Each service has unique cost structures that can vary between providers, and understanding these can significantly impact your company’s financial health and resource allocation.

    With data center colocation services, the costs are generally more predictable. You pay a fixed amount based on your rented server space and the bandwidth you use. This stability can be advantageous for budgeting purposes, especially for businesses with steady data usage and processing needs. However, you may also incur additional expenses for any necessary hardware upgrades, maintenance, or increased bandwidth usage.

    What’s more, there’s the indirect cost of employing or training staff to manage the servers, plus the potential costs of transportation to and from the data center, particularly for businesses not located near their rented server space. These costs can add up and should be factored into the total cost of operating within a data center.

    Cloud services, conversely, operate on a pay-as-you-go model. This solution can be much more cost-effective for businesses with shifting data needs since you can readily scale up or down based on demand, ultimately only paying for the resources you actually use.

    Choosing the Right Path for Your Organization

    Unfortunately, there’s no one-size-fits-all solution for effective data management. Each business has unique needs, requirements, and considerations that necessitate a carefully tailored approach. When choosing between cloud technology and data center colocation services, business leaders must carefully consider several key factors to determine the right solution for them.

    If scalability, flexibility, and AI integration rank high in your data management strategy, cloud technology presents an excellent solution with its on-demand resources and array of built-in tools. On the other hand, data center colocation services provide robust performance capabilities, greater control over data, and predictable costs—features that may appeal to businesses with more consistent data management needs.

    Understanding your current web hosting requirements and the potential cost implications of each solution are also crucial elements to consider in your decision-making process. The demand for efficient, secure, and cost-effective data management solutions will only continue to grow as data storage needs expand rapidly.

  • Why Your Data Lake Strategy is Broken for Observability

    Why Your Data Lake Strategy is Broken for Observability

    Someone once said, “Life is better at the lake.” That may be true in the physical world, but when it comes to using data lakes for observability, that’s not typically the case.

    Just about every organization in every industry has a digital presence, whether it’s a website, an app or another digital service. This makes observability a universal need; it helps teams monitor software performance, optimize performance, detect potential security issues and quickly monitor for bugs if and when they occur. These responsibilities typically fall on engineering teams and require as close to real-time response as possible.

    Some organizations also have business intelligence (BI) needs, aimed at generating insights from multiple data sources for business decision-making. These responsibilities typically fall on business decision-makers; they are not real-time decisions, but rather made by looking back over long time periods.

    All-In on a Data Lake Strategy

    Companies that go all-in on a data lake strategy may ignore the needs of the engineering teams working on the real-time mission-critical aspects of the business and force all of the company’s data, queries, dashboards, etc. into a data lake solution. That doesn’t actually meet the needs of the engineering teams. The result? Engineering teams use unsanctioned solutions, fracturing the company’s data strategy from the ground up.

    By using a standards-based observability solution built on open source InfluxDB 3.0, companies can give their engineers the observability tools they need while preserving their data lake strategy.

    Here are three main challenges that teams come across when using data lakes for time-series, and how to solve the problem with InfluxDB 3.0.

    Problem One: Inefficient and Slow Ingest
    Data sources generate time-series data at an incredibly high rate, easily producing hundreds or thousands of data points every second; some sources even do so as quickly as every nanosecond. Data lakes can store massive volumes of data, but they aren’t optimized to quickly ingest and make it readable on a millisecond time scale.

    Data lake ingest jobs can be measured in minutes or even hours, which is far too late for any meaningful interventions that can prevent catastrophes such as downtime. Customers will notice the outage before the data is even ingested into the data lake, much less acted upon.

    Problem Two: Time-Series Life Cycle
    Full-fidelity data is necessary for real-time analytics and observability, but the value of the full-fidelity data fades over time. Storing the full-fidelity data in a data lake may be convenient, but it is costly and not necessary for typical BI purposes.

    InfluxDB is designed for the time-series data life cycle. This life cycle entails:

    1. The leading edge of data. That is, data written in the last two hours or so. This is the data that is most likely to be queried, so the full fidelity data is kept in memory ready to query instantly.
    2. Compacted data. That is, data older than the leading edge that is still likely to be queried and requires full fidelity is compacted off to disk in object store, which still permits fast querying but does not incur the costs of storing data in memory.
    3. Retention policies. That is, data that has lost its value in its full fidelity form. For example, for observability data, data older than a few weeks can be aggregated into a much smaller data footprint. Then, this aggregated data is ingested into the company’s data lake while the full fidelity data is automatically dropped from InfluxDB 3.0.

    Without such a system, it is necessary to store the full fidelity data in the data lake and incur those massive costs without a reasonable way to manage the life cycle of the data. With such a system, the correct amount of observability data can be ingested into your data lake to support your BI needs.

    Problem Three: Slower Query Performance
    Engineers can experience a high degree of frustration when they are troubleshooting an issue in real-time, but their dashboards won’t load or their queries won’t return in a reasonable amount of time. Data lakes simply are not designed to produce fast results for observability, and so query performance is a primary complaint of engineers trying to use a data lake for observability. Waiting minutes for a query to return when you are running a BI report may be annoying, but it could spell extended downtime for an engineering team.

    Fortunately, InfluxDB is built on the Apache Arrow standards, which means that it can be directly queried by your engineering team. Via protocols such as Delta Sharing and Iceberg, anyone can query into InfluxDB 3.0 from their data lake and join the real-time observability data stored in InfluxDB 3.0 with the data lake data. This is akin to having your cake and eating it too.

    The Bottom Line

    If you are trying to use a data lake strategy to fulfill your engineering team’s observability needs, you are going to frustrate those teams because the data lake is not designed for data ingestion, the data life cycle, or the query needs of those teams. This will cause those teams to create rogue solutions that are inconsistent with your company’s data lake strategy.

    By providing those teams with open source InfluxDB 3.0, engineering teams will have a first-class observability platform that is designed to interoperate sensibly with your company’s data lake. This enables the critical mission of your observability platform while maintaining your overall data lake strategy for your BI needs.

  • How to Achieve Data Mesh With Four Key Principles

    How to Achieve Data Mesh With Four Key Principles

    Today’s world is becoming increasingly data-driven, which means businesses need to refine their practices and decision-making processes with the use of trustworthy data. However, this has proven challenging as organizations often don’t know what to do with the abundance of data available to them. According to a recent survey from Snow Software, 89% of IT leaders receive actionable data, yet 60% feel overwhelmed by the volume at their fingertips. These companies are both thriving in data collection and are bogged down by the sheer quantity available. While traditional data strategies have not yet adapted to meet modern business needs as the amount of data continues to grow, there’s a strong opportunity for companies to transform their data warehouses and data lakes and optimize the flow of data to consumers through an emergent approach called data mesh. Data mesh forges a path for companies to improve their business strategy and make better, faster decisions by leveraging the insights buried within their data.

    Data Mesh and its Benefits

    At its core, data mesh is an approach to data management that enables organizations to evolve from the pitfalls of legacy, centralized data architecture toward a decentralized, domain-driven design at scale. Breaking that down, data mesh is designed to help companies manage complex data networks by focusing on decentralized data ownership, treating data as a product, and providing a uniform infrastructure capable of sharing data across the company landscape. It improves upon legacy data management, which is often time-consuming, error-prone, unsustainable and unscalable. Unlike these traditional methods, data mesh gives domain-driven ownership to experts who strategize how it can best serve the company, as opposed to just dumping all data in one place and waiting for a centralized function to process it and make it available to the business.

    A data mesh approach also enables organizations to democratize their data by allowing users to acquire trusted data when they need it, all while automating the complexity and being shielded by security and governance guardrails. It removes bottlenecks and helps businesses glean the data-driven insights they need at the scale and speed of their organization. This way, organizations can maximize the value of their data by ensuring it is clean, vetted and readily available to support business initiatives.

    The Four Principles of Implementation

    So how can organizations embrace this approach? The framework of data mesh is centered on four guiding principles to help organizations get the most out of their data. Leveraging these principles will help your organization unlock the benefits of a data mesh.

    ● Domain-driven ownership: As opposed to separating data from your organization’s subject matter experts, the first pillar of data mesh implementation involves putting domain experts in charge of their data ecosystem. This way, data will be cleaned and enriched at the source before it’s made available throughout the organization. Domain owners will maintain the quality of the data and keep all necessary facts and documentation available for audits, effectively combining data with business talent.
    ● Data as a product: The job of the data team is to provide the data that the company needs for whatever purpose, be it making decisions, building personalized products, or detecting fraud. By treating data as a product, the company is giving its data a vision, strategy and product roadmap, ensuring its value is executable.
    ● Self-service data infrastructure: A business is made up of logically autonomous domains; each domain not only supports a business function, product or process but also produces data products and therefore requires a data infrastructure. Self-service infrastructure ensures business domains don’t have to deal with the underlying complexity of networking, security, compute and storage requirements.
    ● Federated computational governance: Standards for data governance are defined centrally, but local domain teams are empowered and equipped to apply to these standards however is most suitable for their specific environment within predefined guardrails. The key to achieving this for policy, security, quality and definition at scale is automation and integration throughout the landscape of the data infrastructure.

    By enacting these principles within your organization, your data mesh architecture is sure to provide the most value for your data.

    Data mesh is an optimal solution for any business currently struggling to find value in its data. Traditional methods are time-consuming, unsustainable, unscalable and error-prone. Data mesh is an approach designed to transcend these barriers, decentralizing much of the IT team’s heavy lifting and putting the responsibility of managing data on individual business domains. With clear ownership, domain experts can develop deep data knowledge and expertise and ensure that it’s trustworthy and valuable to your organization before making it available for consumption. This way, enterprises can prioritize self-service and empower data consumers with access to accurate, high-quality data when needed.

  • Apache Druid’s Role in Modern Data Analytics

    Apache Druid’s Role in Modern Data Analytics

    From Twitter and Netflix to Salesforce and Confluent, modern analytics applications are being built by developers across the world’s leading digital businesses to create digital tools that serve real-time analytics to hundreds or even thousands of concurrent users.

    Looking more closely at what these apps do reveals some critical, strategic technology decision-making. In particular, why do developers increasingly turn to Apache Druid instead of the myriad of other databases out there?

    In general, they look to Druid to build apps that deliver on a range of objectives, including creating operational visibility at scale, customer-facing analytics, rapid-drill down exploration and real-time ML pipelines. Let’s take a look at each of these in turn:

    Operational Visibility at Scale

    Building apps for operational visibility can include a wide range of use cases, such as product analytics, digital operations, observability, fraud detection and IoT. In each situation, developers need to minimize the gap between events created and time-to-insight.

    From a technology standpoint, there are several databases that can analyze events in real-time, including stream processors like Apache Flink or ksqlDB, time-series databases like InfluxDB or TimeScaleDB and also key-value stores such as Redis. The problem is that all of these technologies share the same inherent limitation for analytics: They aren’t able to process OLAP-style queries effectively at any meaningful scale.

    For instance, if the app in question calls for analyzing time and non-time-based dimensions over the past month, assuming ingestion was one million events per second, that would equate to more than 2.6 trillion events. Any database that supports stream ingestion can query the current status (setting aside the challenge to handle ingestion at this scale) and, to an extent, basic aggregation of numbers and counters. Difficulties can arise, however, if the app requires aggregation across all of these events, “group by” on non-time-based attributes or high concurrency. Using the wrong database leaves users looking at the ‘spinning wheel of death’ as they wait for results.

    In these circumstances, devs turn to Apache Druid. A real-world example explains how: Salesforce engineers built an analytics app using Druid to monitor their product experience. The app enables engineers, product owners and customer service reps to query any combination of dimensions, filters and aggregations on real-time logs for performance analysis, trend analysis and troubleshooting. As a result, their query results are returned in seconds with billions or even trillions of streaming events ingested every day.

    Customer-Facing Analytics

    On the face of things, the technologies offered by Twitter, Forescout and Atlassian don’t have a huge amount in common. Look beneath the surface, however, and they all apply analytics to more than just internal decision-making. They have also created analytics apps and processes to provide their customers with powerful insights as part of a value-added service or a core product offering 

    What does that teach other organizations? When building analytics for external users, a whole different set of tech challenges must be addressed. While concurrency is clearly a key consideration, the database should also support current and future user growth with ease. The thing is, concurrency also has to be delivered economically, or it can have a huge impact on infrastructure costs.

    Similarly, delivering a good interactive experience is dependent on sub-second query response performance—it has become a minimum user requirement almost everywhere. Instead of fighting PostgreSQL or Hive or any other analytics database to meet the performance requirements at scale, Druid offers the performance devs and users expect.

    Take Confluent, for example, which is a popular event streaming platform. They built their cloud-based monitoring service, Confluent Health+, with Druid. Having initially used a different database, they discovered that, as their data volume grew, their legacy pipeline struggled to keep up with their data ingestion and query loads. By switching to Druid, they have been able to create the impressive user experience developers everywhere are focused on delivering.

    Rapid Drill-Down Exploration

    Generally speaking, the primary role of reports is to fulfill a simple requirement: Deliver data in a consumable form. What organizations are increasingly looking for, however, is a deeper level of insight and understanding that shows why something happened. They can take this information and apply it to a wide range of possibilities, such as solving, anticipating or preventing future problems or opportunities.

    Addressing these ‘why?’ questions means data has to be analyzed at tremendous speed to reveal root causes. While this is a simple proposition when the data set in question is small, when dealing with data drawn from today’s high-volume sources, such as cloud services, clicks and IoT/telemetry, the underlying database has to correlate a few to hundreds of dimensions across highly cardinal data with billions to trillions of rows. That presents a much more difficult challenge.

    Indeed, when data is analyzed at a huge scale, full table scans are simply too slow. Exacerbating the problem is the typical query-shaping techniques used to speed up performance have expensive tradeoffs:

    Looking at real-world use cases once again helps demonstrate the role of Apache Druid in situations such as these. For example, developers at cybersecurity company HUMAN built an analytics app for internet bot detection using Apache Druid. Previously, their cloud data warehouse was too slow to keep up with their need to drill into massive web traffic quickly. Today, their Apache-powered app gives their data scientists the ability to aggregate and filter across trillions of events instantly, meaning they can classify anomalies and train their machine learning models to detect real-time malicious activity.

    Real-Time ML Pipelines

    For many people, analytics is synonymous with viewing a synthesized data set presented via a UI like a chart, graph, report or dashboard. Increasingly, however, new operational workflows are completely automated without any human intervention or data visualization.  

    Google Maps is an example of this use case, given it optimizes routing based on real-time and historical traffic flow and patterns. Another example is Reddit’s use of Apache Druid to deliver real-time ad placement within milliseconds of a new visitor arriving on the app. In this context, there is no data visualization to interpret; it has been replaced by automated decisions derived by inference. These performance requirements play particularly well to the capabilities offered by Druid.

    Across a growing variety of demanding use cases, devs are adopting Druid to build inference-based capabilities into their apps. From diagnostics, recommendations and automated decisions, today’s organizations need instant query responses of high cardinality, alongside highly dimensional event data at scale and in real-time. Delivering performance levels like this is becoming increasingly important for businesses focused on innovation, quality of service and competitive advantage.

  • Snowflake Makes Case for Building and Deploying Apps on Platform

    Snowflake Makes Case for Building and Deploying Apps on Platform

    During a Snowday 2022 event, Snowflake this week moved to make it simpler to build applications on top of its cloud platform for managing data in addition to enhancing the overall security of the platform.

    The company has made generally available Snowpark, a framework for building Java, Scala, SQL or Python applications. Snowflake also committed to making generally available in early 2023 an open source Streamlit framework for building Python applications that Snowflake acquired earlier this year.

    Snowflake also revealed it is adding observability tools to the platform to make it easier to build, test, debug, deploy and monitor data pipelines, including alerting tools (private preview), logging capabilities (private preview), event tracing (private preview), task graphs and history (public preview).

    Torsten Grabs, director of product management for Snowflake, said those capabilities will make it easier to alert DevOps teams to issues that might arise with applications deployed on the Snowflake platform.

    At the same time, Snowflake is also making it simpler to build data pipelines by making a Serverless Tasks capability generally available. In addition, the company is previewing a Schema Inference engine that makes it simpler to ingest data and a Dynamic Tables capability to declaratively manage data pipelines.

    Other capabilities being added to the platform include a public preview of a Query Acceleration Service to make additional compute resources available on demand. Snowflake has also improved query efficiency with join elimination in addition to providing search optimizations.

    There are also now cost optimization tools that enable IT teams to run a cost-benefit analysis of Snowflake services in addition to tools that make it easier to replicate data across multiple cloud computing environments running the Snowflake platform.

    IT teams can now also replicate Streams and Tasks via a capability in public preview that enables data pipelines to seamlessly work on secondary Snowflake accounts and configure failovers using a graphical tool that is available in private beta.

    Finally, Snowflake has also enhanced its cross-cloud Snowgrid Capability to make it easier to share both public and private listings of data in addition to applying tags to and searching masked data.

    The latest editions to the Snowflake platform come at a time when more organizations are starting to adopt DataOps best practices to manage data. Snowflake is now making a case for building and deploying applications on top of a platform that automates the management of the data pipelines used by applications to access data.

    It’s not clear to what degree the Snowflake platform will be employed to host applications, but the company is clearly betting the volume of data residing in its cloud platform will attract a wide range of application developers that, in turn, will need to be supported by DevOps teams.

    Of course, Snowflake is not the only provider of a data warehouse or data lake that has similar ambitions. Regardless of platform, however, the one thing that is clear is that a convergence of DevOps and DataOps workflows is now underway as IT organizations look to streamline the management of the massive amounts of data that now reside in the cloud.

  • InfluxData Delivers Revamped Database for Real-Time Processing

    InfluxData Delivers Revamped Database for Real-Time Processing

    InfluxData has made good on a promise to deliver a next-generation time-series database that enables observability and monitoring tools used by DevOps teams to ingest high volumes of data more cost-efficiently.

    In addition, for the first time, the latest iteration of the InfluxDB database supports the de facto standard SQL query language. Developers can run unlimited time-series workloads and contextualize data by any dimension without restrictions using SQL or existing tools such as the InfluxDB application programming interface (API), Flux scripting language or the InfluxQL query language

    Brian Gilmore, director of internet-of-things (IoT) and emerging technologies for InfluxData, said InfluxDB is now based on a columnar engine, dubbed IOx, that leverages the open source Apache Arrow memory format and the Rust programming language accessed via a multi-tenant InfluxDB Cloud service that company manages on behalf of application developers. Use cases that require processing large volumes of data such as metrics, events, traces and other high cardinality data can now be queried in milliseconds, he noted.

    The goal is to make this latest incarnation of InfluxDB available via a managed cloud service before providing IT teams with a version they can install themselves.

    The arrival of a revamped InfluxDB architecture comes as the amount of data being generated across application environments continues to rapidly expand. Many DevOps teams are especially struggling with ways to process and store all the data being generated by cloud-native applications made up of microservices generating a steady stream of logs, metrics and traces.

    InfluxData, of course, has ambitions for its database that go beyond application development and deployment environments. Edge computing platforms, for example, will increasingly process and analyze data in real-time at the point where it is generated and consumed. Similarly, applications infused with machine and deep learning algorithms will require databases that can process data at scale. Regardless of the use case, however, it’s apparent more IT teams will be embracing DataOps best practices alongside DevOps to manage data as applications are deployed and continuously updated.

    It remains to be seen how many observability and monitoring tools might embrace this latest iteration of the InfluxDB database, but it’s clear DevOps teams should expect to see major advances as the underlying databases upon which these platforms depend continue to evolve. The goal is to ultimately eliminate the tradeoff that now often occurs between how much data is collected and analyzed versus the cost of processing and storing it.

    In the meantime, DevOps teams will need to decide whether they want to manage databases themselves as more managed services become easier to invoke in the age of the cloud. Most database selections these days are made by development teams that aren’t especially eager to take on the tasks normally performed by a database administrator. A managed service essentially outsources responsibility for managing a database to a services provider instead of requiring IT teams to hire a DBA themselves.

    One way or another, however, the speed at which data can now be processed in real-time should sharply reduce the need to rely on existing and cumbersome batch-oriented approaches to processing data.

  • Mezmo Adds Observability Pipeline to Analyze DevOps Data

    Mezmo Adds Observability Pipeline to Analyze DevOps Data

    At the KubeCon + CloudNativeCon North America conference this week, Mezmo launched an Observability Pipeline platform that promises to make it simpler to manage, enrich and correlate machine data.

    Previously known as LogDNA, Mezmo is expanding the scope of its reach to add the ability to augment and analyze data on top of its core log management platform. Over time, that capability will be extended to add support for metrics and traces using the open source OpenTelemetry agent software.

    Mezmo CEO Tucker Callaway said the goal is to provide a framework for centralizing the flow of observability data that is now being generated by a wide range of sources across a DevOps workflow. It’s clear that DevOps teams require a means to enrich and correlate high volumes of data in motion that goes beyond simply pre-processing so it can be stored more cost-effectively, he added.

    The Mezmo Observability Pipeline achieves that goal by ensuring the right data is flowing into the right systems in the right format for analysis, noted Callaway.

    As the volume of data being collected by DevOps teams continues to dramatically increase the need to embrace a set of DataOps best practices is becoming more apparent. DataOps has emerged recently to describe a set of data engineering practices that applies many DevOps principles to the way data should be managed. Mezmo’s Observability Pipeline aims to make it simpler to apply DataOps best practices specifically to the data collected by DevOps teams.

    Of course, each DevOps team will need to decide for themselves how much data they want or need to store long term. However, increasingly DevOps teams are holding onto data much longer as IT environments become more complex, noted Callaway. That creates a need to more efficiently analyze as much data as possible and store it more cost-effectively, he said. Those twin goals, however, often create a physics problem in the sense that they can be diametrically opposed.

    In the longer term, it’s not quite clear how much DevOps teams are embracing observability. Most DevOps teams make use of monitoring tools that enable them to track a set of pre-defined metrics. Observability platforms promise to make it easier for DevOps teams to query a wide range of data types to discover potential issues before they disrupt an IT environment. The challenge is that many DevOps teams may not know what queries to form to discover those issues. It’s likely that, over time, machine learning algorithms will be increasingly applied to observability data to make it simpler to surface potential issues.

    In the meantime, however, the need for DevOps teams to find ways to manage all the data being generated by multiple types of applications is becoming a higher priority. In fact, in the absence of any meaningful way to manage all that data, the benefits of an investment in an observability platform is likely to be limited, at best.

  • SaaS Data Backup and the API Bottleneck

    SaaS Data Backup and the API Bottleneck

    The need to protect SaaS data has never been greater. A recent global survey from Odaseva found that 51% of ransomware attacks are targeting SaaS data, and they are more likely to succeed (52%) than were attacks on cloud, endpoint and on-premises data. 

    But there are plenty of reasons beyond the threat of ransomware to protect SaaS data. Employees accidentally—and maliciously–delete data. And while outages are not common, they do happen, and when they do, it can cause a lot of pain. Atlassian, for instance, suffered a two-week outage in April. If the data isn’t backed up, the organization is completely at the mercy of the provider to get the service back online before they can access it.

    And though all respondents said their organizations say they have some form of SaaS data protection in place, 57% said they still have SaaS data that remains unprotected. Likely, that’s why only half of respondents whose SaaS data had been successfully encrypted by an attack said they were able to recover all of it, compared to more than eight in 10 (81%) who were able to do so for on-premises data.

    That’s not surprising, because backing up SaaS data is very different from on-premises data protection. For starters, of course, IT has limited control, at best, over the application and data. IT will need to rely on APIs to back up and restore SaaS data, and those API calls will have hard caps. SaaS providers almost always run their services on a multi-tenant architecture, so they need to make sure API resources are available for everyone. To accomplish this goal, they set hard caps on API calls, typically over a 24-hour period, which means that IT must carefully manage API usage for backup and restore operations. And with many different APIs with different capabilities and limitations to choose from, IT will need to make some hard, complex choices. 

    Navigating APIs With Hard Caps

    Managing API usage is important, and not just for backup. These same APIs are also vital for connecting other applications integrated into the SaaS application. And for core SaaS apps, such as CRM or ERP, there will likely be many such integrated applications that require the use of those APIs. 

    To get the best performance, IT needs to be very careful with the APIs it uses for backup and recovery. Let’s take Salesforce as an example. The Salesforce REST API can transfer up to one million records per hour, while the BULK API can do up to ten times more in the same amount of time. And if you use parallel calls to multiplex data, the BULK API can do up to 300 million. The upshot is that the API you use can make a huge difference in your recovery point objectives (RPOs) because transferring more data faster means more backups throughout the day.

    Unfortunately, Salesforce customers won’t be able to depend on BULK API for all their backup and recovery needs. Some objects can only be accessed via the REST or other APIs, for instance, and that BULK API will also be useful to other apps—you don’t want to hit that hard cap and leave other applications unable to connect. And you can’t always use the same API to write data as you do to read it.

    The SaaS backup infrastructure has to achieve a balance of API usage, which must be closely managed. And don’t forget about restore, which can consume a ton of API resources. Make sure you have enough to account for a big restore—you really don’t want to hit the API cap while you’re trying to get your users back in business after a data loss event.

    Lastly, APIs change over time and the backup system has to be able to adapt. You’ll need to make sure you keep up with API documentation so the system can be updated regularly; otherwise, data may go unprotected.

    It’s now becoming widely accepted that SaaS data needs to be protected, but building a SaaS backup solution that meets an organization’s RPOs and RTOs is more complex than many in IT may realize. A strategic plan for API management is a must-have for any SaaS backup management solution.