devopsy.kinsta.cloud

Tag: data analytics

  • Implementing Data Analytics in Multi-Cloud Environments

    Implementing Data Analytics in Multi-Cloud Environments

    Multi-cloud data analytics is more intricate, demanding a nuanced approach to ensure seamless operations and accurate insights. Coordinating analytics across multiple cloud providers introduces various issues, requiring careful consideration of implementation strategy. From cost management to performance variability and data orchestration, we will explore each challenge in detail and provide actionable insights on how to effectively navigate these hurdles, ensuring fool-proof data analytics infrastructure in multi-cloud solutions.

    Interoperability and Standards

    Interoperability challenges arise in multi-cloud data analytics due to varying APIs and standards across cloud providers. The lack of uniformity in data formats and communication protocols hinders seamless collaboration and data exchange. To address this, organizations should prioritize cloud services adhering to open standards, and advocate for industry-wide interoperability initiatives. Additionally, create an abstraction layer or middleware that acts as a bridge between different cloud providers. This abstraction layer helps standardize interactions, facilitating seamless data exchange and reducing dependencies on specific provider APIs. Thus, you enhance the flexibility of your multi-cloud environment, making it easier to integrate and switch between different cloud platforms.

    Vendor Lock-In

    Organizations risk becoming overly dependent on a single vendor, limiting flexibility and making migration complex. Utilizing specialized features or tools from one provider may face difficulties transitioning to another, which restricts the ability to choose more cost-effective or suitable solutions and may result in increased switching costs. To mitigate this, a multi-cloud strategy should be adopted, favoring standardized technologies and open APIs, allowing organizations to distribute workloads across multiple cloud providers and avoid being tied to a specific vendor’s ecosystem, enhancing flexibility and strategic decision-making. Design architectures that leverage abstraction layers and avoid tightly coupling with provider-specific services. This approach ensures flexibility, allowing your organization to choose and switch between cloud providers based on performance, cost and strategic considerations while minimizing the risk of vendor lock-in.

    Cost Management

    Each provider has unique pricing models and billing mechanisms, making it difficult for organizations to accurately estimate and compare costs. The dynamic nature of data analytics workloads, coupled with data transfer expenses between clouds, contributes to the complexity of budgeting. To cope with that, implement a centralized cloud cost management tool to monitor expenses across providers. Also, regularly review and optimize resource utilization to identify underutilized or over-provisioned instances. You may leverage reserved instances or spot instances for cost-effective computing and implement automation for scaling based on demand, ensuring resources are allocated efficiently. Lastly, negotiate pricing with cloud providers and explore discount options to secure favorable terms for long-term commitments.

    Performance Variability

    As providers have unique infrastructure, network configurations and service levels, inconsistent performance levels may occur and impact the reliability and predictability of analytics results. To mitigate these consequences, conduct thorough performance testing across different cloud environments, optimize queries for specific platforms, and consider workload distribution based on provider strengths. Utilizing performance monitoring tools helps track and analyze variations, enabling timely adjustments. Choosing cloud providers based on specific performance requirements and establishing performance service level agreements (SLAs) also mitigates the impact of performance variability in multi-cloud data analytics.

    Resource Orchestration

    Due to differences in orchestration tools, APIs, and resource management mechanisms cloud providers employ, compatibility issues often take place. The heterogeneity makes it complex to ensure seamless deployment, scaling, and management of analytics workloads. Organizations may consider adopting cloud-agnostic orchestration tools and frameworks to abstract the underlying differences among cloud providers, allowing for consistent resource management. Embracing infrastructure as code (IaC) practices and automation also streamlines resource orchestration, enabling organizations to deploy and manage analytics workloads more efficiently across various cloud environments.

    Data Movement and Latency

    Varied network architectures, geographical distances, and differing data transfer mechanisms contribute to increased latency and potential performance issues. Transferring significant datasets across clouds incurs additional costs and can lead to operational inefficiencies. To avoid that, organizations should optimize data movement processes by leveraging efficient transfer protocols, considering edge computing for proximity to data sources, and strategically distributing workloads based on data location. Implementing caching strategies and utilizing content delivery networks (CDNs) can help mitigate latency challenges, ensuring a more responsive and cost-effective multi-cloud data analytics environment.

    As a Footnote

    In conclusion, the journey of implementing data analytics in multi-cloud environments presents its unique set of challenges. Embracing open standards, adopting cloud-agnostic tools and fostering a culture of continuous improvement are essential components of overcoming them. Continuous monitoring, regular updates to governance frameworks, and strategic decision-making aligned with your business specifics and objectives are keys to reaping maximum benefits from multi-cloud data analytics. You may also take advantage of cloud computing consulting to get a tailored strategy from experts.

  • Multi-Tenant Database Strategies for Mixed Workloads and Real-Time SLAs

    Multi-Tenant Database Strategies for Mixed Workloads and Real-Time SLAs

    Multi-tenancy—where a single instance or multiple instances of one or more applications operate in a single shared infrastructure environment—is not a new concept. First used in mainframe hardware, timesharing enabled a single massive piece of computing iron to be sliced up and used by multiple applications on a single hardware instance. This simplified both the hardware landscape as well as the skills and resources needed to manage environments. With the rise of software-as-a-service (SaaS) in today’s modern private and public cloud environments, multi-tenancy is now a common and critical function. It allows cloud companies to scale and customers to save money by essentially pooling resources. As more applications demand more and more data, multi-tenancy is now a core function at the database layer.

    But when you have a lot of different applications running on the same infrastructure and accessing the same database, how do you prevent one application from impacting the data performance of another? And how do you determine from the underlying database infrastructure which tenants get what infrastructure and at what performance level?

    Here’s a look at the driving forces behind today’s real-time data needs and how they impact multi-tenant data architecture management.

    SLAs Are at the Heart of Multi-Tenant Data Management

    Everyone wants faster performance and more resilient systems. We’re impatient, so reliable, real-time performance delivers large, competitive business benefits. However, every tenant does not require the same level of performance. Actual performance needs may be plenty fast for a user or machine without requiring the fastest execution path possible with the available resources.

    Service-level agreements (SLAs) ultimately govern the resources delivered to a tenant. Real-time may need to be measured in sub-milliseconds in one instance (i.e., for a stock trade) but in sub-seconds for others (i.e., in a Zelle funds transfer). This is where developers, architects and operators partner with a business to understand its actual user or system needs and to build a system that can deliver the SLA using the minimum amount of resources possible.

    Mixed Workloads and the ‘Noisy Neighbor’ Problem

    As soon as workloads are shared, there’s the potential for one application to quickly grab and use resources in a way that impacts other applications. Often called a “noisy neighbor,” the impacting applications can dramatically interfere with real-time response time SLAs when multi-tenant apps share an infrastructure base. For example, one node in a database cluster becomes heavily loaded, and the performance of other nodes begins to be impacted, causing SLAs to be missed.

    Typically, multiple virtual machines (VMs) are hosted on a single physical machine or node but share system resources (i.e., processor, network, storage, etc.) with the other VMs. It’s possible for applications to overuse those shared resources, impacting other applications and sometimes even the database itself. Such overusage of the infrastructure layer (in this case, sharing compute, storage, network queues, etc.) leads to poor database and end-to-end system performance. It’s like plugging a 100-amp appliance into a 50-amp circuit. All the circuit breakers will trip, and the entire system comes to a screeching halt!

    Strategies to Maximize Performance for Database Multi-Tenancy

    Configuring and managing the database itself (not just infrastructure) with multi-tenancy in mind can maximize performance and avoid problems like noisy neighbors. Isolating databases in your multi-tenant architecture can help minimize non-database problems that can be the root cause of poor performance and missed SLAs.

    Start by picking databases that allow applications to isolate their data access based on sets (or tables in relational parlance) that can have storage limits attached to them. When combined with rate limits for transactions on a per-user basis, applications can be guaranteed their quota of runtime resources. Using the per-app quota configurations for items like storage and TPS (transactions per second), the database can evaluate the total capacity available in the system and ensure that enough resources are reserved and available to ensure that the SLAs of all applications are met all the time.

    Note that unexpected events that cause rebalancing of data within the distributed database, such as node and network failures, require additional resources. The database system will subtract that resource capacity from what is available for apps so that the system is not overcommitted during failure situations.

    Enterprises also need very good automation and observability to fix immediate problems quickly and have the requisite visibility into long-term trends that impact performance and SLA compliance. Otherwise, by the time a human notices a problem, the SLA may already have been missed and the business impacted.

    Good observability can help you understand how data is used and create those automations. As more applications access and use the same data set in different ways, you need modern observability tools integrated into (or interoperable with) the underlying database.

    With a good understanding of how your data-rich applications interoperate with other tenants in mixed-load environments, you can begin to deploy these strategies to make sure you conform to all required SLAs and keep all that data working hard for the business.

  • Consider Performance, Growth & Budget When Buying Data Analytics

    Consider Performance, Growth & Budget When Buying Data Analytics

    So, your business needs to invest in data analytics technology to improve efficiencies, competitive advantage or business outcomes. There are a ton of things to consider, but underlying each decision are three driving factors:

    • Performance – The need to meet service level agreements.
    • Growth – The need to accommodate success, as well as deal with inevitable industry changes.
    • Budget – The need to do both of the above while keeping costs low enough to not erode the profit gain from analytics.

    These three forces underpin nearly every technology selection decision, but what frequently isn’t noticed is how much these three drivers interact.

    1. Concurrency and Growth

    Test both for current levels of concurrency and the level you hope to achieve in the next few years as the organization grows.

    When evaluating benchmarks between vendors, consider that benchmarks are designed to make that company’s software look good. For example, if they are good at concurrency, you will see various levels of concurrency represented in the benchmark. If the benchmark shows only one workload at a time, or maybe ten, you can safely assume that concurrency is probably not something that technology is good at. Similarly, if the benchmark is done at 10 TB scale, the vendor probably isn’t that good at high-scale analytics.

    When bringing vendors in for a proof-of-concept, one mistake I’ve seen (more times than I can count) is testing the capabilities of the software with only one user at a time. People do not line up politely to use analytics one at a time.

    In addition to current levels of concurrency, over time, hopefully, the organization will grow, and the number of both internal analytics users and external customers will increase. Testing for the number of users you have now may not uncover problems that will invariably reveal themselves when a lot of people hit the system at once. The last thing your organization needs is to turn away customers because they’re overloading your systems.

    2. Performance & Price

    Evaluate price/performance, not performance alone.

    Everyone understands the importance of performance. Not one user has ever gone to a data architect or engineer and requested for their analytics to “please execute more slowly.” But performance isn’t simply how fast the analytical software can process queries, how fast the data pipelines can move data or how fast the response is between data input and reaction output. There’s also a monetary aspect of performance to bear in mind.

    In a recent benchmark, three data management and analytics stacks were compared with 60 simulated concurrent users/workloads. The difference in queries per hour on 250 TB of data was minor, perhaps 16 queries per hour for the slowest and 23 queries per hour for the fastest.

    (Mcknight Benchmark – SaaS Data Analytics Platform Comparison – vendor names removed)

    It looks like performance is just not that big a consideration for this technology. But there is no such thing as an unlimited budget. Let’s look at how much it would cost to get this level of performance out of each stack.

    (Mcknight Benchmark – SaaS Data Analytics Platform Comparison – vendor names removed)

    Price-performance is a far more useful metric of value to the organization. How much the software can do is irrelevant if the budget limits how much the business can afford to do with it.

    3. Software Efficiency

    Consider how much efficiency and control you’re willing to trade for ease-of-use.

    When you analyze data in the cloud, you’re not just paying for software, you’re also paying for hardware, a.k.a. cloud compute instances. Bundling the two costs in one bill makes things easier. But it also means that cloud providers make more money the more instances, or compute nodes, you use. You trade ease-of-use in billing for a higher price. And they no longer have a financial incentive to provide efficient software.

    Similarly, to make a cloud-based analytics software scale up in performance and concurrency with zero effort on your part, the software has to use only defaults for all settings internally. It can scale only by throwing more compute at the problem; compute that you’re paying for every time you use it. You’re trading control of the software–tuning, scaling guidelines, shut down conditions, etc.–for ease-of-use, again for a higher price.

    Throwing more compute at a problem isn’t always the best solution. Recently, I saw a POC where the customer implementation included 278 EC2 nodes on Amazon cloud. The organization didn’t initiate the POC because of inefficiency. They were having performance problems, even with that much compute power. A competing product was able to run on only nine EC2 nodes with the same workload and better performance. The company was paying 25 times as much for the compute hardware because it was convenient.

    Be aware of the tradeoffs.

    4. Affordable Scalability

    Paying for only what you use can become impractical when you use more and more.

    The volume of data that is being created, consumed and captured is growing exponentially. By 2025, it is estimated that nearly 30% of data being generated will be real-time. Instead of pulling data in batches, data will be pulled in continuously. Queries will be responded to in seconds, not hours, or in milliseconds, not seconds.

    Going to the cloud can make tremendous sense when a company or a workload in that company is new. But studies show that as businesses scale and growth slows, the pressure the cloud puts on business margins eventually outweighs its benefits. This is why most enterprises today are considering repatriating some or all of their workloads off the cloud.

    5. Flexibility for Change

    Make certain your stack will continue to work even as conditions change, and you won’t have to rebuild from scratch to do the same thing in a different location.

    When you’re building a long-term architectural strategy, you want to ensure you cover as many bases as possible because, the fact is, you really don’t know what the future holds. Ten years ago, maybe you had a Hadoop on-premises strategy or a data warehouse appliance. Today, you may have a cloud-focused strategy. Tomorrow, maybe you’ll move some or all of those workloads back on-premises due to performance, security, costs, regulations or other reasons.

    This is why you want flexibility that doesn’t mandate a “cloud-only” or an “on-premises-only” or a “this-cloud-only” architecture and which doesn’t integrate with only the same vendors’ software. You want an architecture that can work equally well both on the cloud as well as on-premises, hybrid or multi-cloud, providing the flexibility of going where business needs are best met.

    Containerization offers possibly the highest level of flexibility in deployment, but it can also add to deployment complexity, so again, be aware of what you’re trading. If the software you choose is platform-agnostic—not tied to any particular location, ecosystem or deployment model—then that will make future choices a lot more flexible.

    Regardless of what analytics you choose, what you decide to build or what use cases you have to meet, striving for balance in performance, growth and budget will create a strong data architecture, now and for the future:

      1. The analytics architecture is high-performance: it should meet SLAs, get things done on time and meet or exceed expectations and requirements.
      2. The analytics architecture is resilient to growth and change; even if there is rapid growth or sudden or extreme changes, it’s future-ready.
      3. The analytics architecture should control costs and live within an allocated budget.

    Keep these guidelines in mind when choosing an analytics provider. The decisions you make today will have direct ramifications on the growth, profitability and success of the business tomorrow.

  • The Benefits of a Streaming Database

    The Benefits of a Streaming Database

    Streaming databases offer an innovative approach to data processing that combines the familiarity of traditional models with a powerful data transformation engine. This hybrid system provides DevOps and DataOps teams with extremely fast results for complex SQL queries, aggregations and transformations that are impossible or take hours to process in the traditional batch computation model.

    By pre-specifying query results upfront (in materialized views) and then incrementally updating them as new data comes in, streaming databases offer a valuable alternative to traditional databases that require complex ETL pipelines or overnight batch processing.

    As the demand for faster results continues to rise, streaming databases are emerging as a fundamental new building block for data-driven organizations.

    streaming database

    History

    Streaming database concepts originated in capital markets, where fast computation over continuous data is highly valued. The first products were basic event processing frameworks that addressed specific needs within hedge funds and trading desks. However, their creators quickly recognized that SQL works as the declarative language for streaming data just as well as it does for traditional static databases.

    Today, modern streaming databases are most often used downstream of primary transactional databases and/or message brokers, similar to how a Redis cache or a data warehouse might be used.

    Trade-Offs

    Experienced engineers understand that no software stack or tooling is perfect and comes with a series of trade-offs for each specific use case. With that in mind, let’s examine the particular trade-offs inherent to streaming databases to understand better the use cases they align best with.

    Unique Features

    • Incrementally updated materialized views – Streaming databases build on different dataflow paradigms that shift limitations elsewhere and efficiently handle incremental view maintenance on a broader SQL vocabulary. Other databases like Oracle, SQLServer and Redshift have varying levels of support for incrementally updating a materialized view. They could expand support, but will hit walls on fundamental issues of consistency and throughput.
    • True streaming inputs – Because they are built on stream processors, streaming databases are optimized to individually process continuous streams of input data (e.g., messages from Kafka). Scaling streaming inputs involves batching them into larger transactions, slowing down data and losing granularity. In traditional databases (especially OLAP data warehouses), larger, less frequent batch updates are more performant.
    • Streaming outputs on queries – Many databases have some form of streaming output (e.g., the Postgres WAL), but what’s missing is output streams involving any kind of data transformation. Streaming databases allow for streaming output of complex joins, aggregations and computations expressed in SQL.
    • Subscribe to changes in a query – As a side effect of streaming outputs, streaming databases can efficiently support subscriptions to complex queries: Updates can be pushed to connected clients instead of forcing inefficient polling. This is a key building block for pure event-driven architectures.

    Missing Features

    • Columnar optimization – OLAP databases have advanced optimization techniques to speed up batch computation across millions of rows of data. Streaming databases have no equivalent because the focus is on fast incremental updates to results prompted by a change to a single row.
    • Non-deterministic SQL functions – Non-deterministic functions like “RANDOM()” are common and straightforward in traditional databases. But imagine running a non-deterministic function continuously, resulting in chaotic noise. Therefore, streaming databases don’t support non-deterministic SQL functions like “RANDOM().”

    Performance Gained

    • Minimized time from input update to output update – The time between when data first arrives in the streaming database (input) and when changes to results reflect the change (output) is sub-second. Additionally, it keeps up as the dataset scales because results are incrementally updated.
    • Repeated read query response times – When a query or query pattern is known and pre-computed as a persistent transformation, reads are fast because they require no computation: You’re just doing key-value lookups in memory, similar to a cache like Redis.
    • Aggregations – The resources needed to handle persistent transformations are often proportionate to the number of rows in the output, not the scale of the input. This can lead to dramatic improvements in performance in aggregations in a streaming DB versus a traditional DB.

    Performance Sacrifices

    • Ad-hoc query response times – While running ad-hoc queries in a streaming database is possible, response times will be much slower because the computation plan is optimized for continuously maintaining results, not answering point-in-time results.
    • Window functions – A window function performs calculations across table rows that are related to the current row. They are less performant in streaming databases because updating a single input row requires updating every output row. Consider a “RANK()” window function that ranks output by a computation. A single update can force an update to every row in the output.

    Factors Impacting Scalability

    • Throughput of changes – The changes or updates to input data are what triggers work in the system, so changing data will often require more CPU than data that changes rarely.
    • Cardinality of the dataset – The total number of unique keys will slow down read queries in traditional databases. In streaming databases, high cardinality increases the initial “cold-start” time when a persistent SQL transformation is first created and requires more memory on an ongoing basis.
    • Complexity of transformations – Unlike the on-request model in a traditional DB, SQL transformations are always running in a streaming database in scale in two ways:
      • Memory required to maintain intermediate state – Imagine how you would incrementally maintain a join between two datasets: You never know what new keys will appear on each side, so you must keep the entirety of each dataset in memory. This means that joins over large datasets can take a significant amount of memory.
      • Quantity and complexity of transformations – When a single change in inputs needs to trigger a change in outputs in many views, or when many layers of views depend on each other, more CPU is required for each update.

    Use Cases

    Like any software primitive, there are many use cases. Here are some categories of products and services particularly well-suited to streaming databases:

    • Real-time analytics – Use the same ANSI SQL from data warehouses to build real-time views that serve internal and customer-facing dashboards, APIs and apps.
    • Automation and alerting – Build user-facing notifications, fraud and risk models, and automated services using event-driven SQL primitives in a streaming database.
    • Segmentation and personalization – Build engaging experiences with customer data aggregations that are always up-to-date: personalization, recommendations, dynamic pricing and more.
    • Machine learning in production – Power online feature stores with continually updated data, monitor and react to changes in ML effectiveness – all in standard SQL.

    Conclusion

    Streaming databases offer a powerful yet accessible way for data and software teams to leverage stream processing capabilities. By using familiar SQL DB concepts and a stream processor to compute SQL transformations, data and software teams can focus on shipping complex data-driven products quickly, with improved performance and scalability. With a streaming database, organizations have the power to transform data in real-time and build the applications of tomorrow.

  • How Database DevOps Fuels Digital Transformation

    How Database DevOps Fuels Digital Transformation

    In today’s real-time, hyperconnected reality, data is a core driver behind most digital strategies. User data is constantly gathered and shared across various technology backends and platforms—as is system performance data to inform engineers. Data is pivotal for everything, from generating machine learning models, refining user experiences and even supporting new data-centric services. But although data is a clear backbone to many enterprises, there is no clear winner regarding which database to use.

    Large organizations often find themselves supporting different database styles, from relational databases to NoSQL and distributed SQL. And as organizations continue to adopt multiple databases, safely managing the processing and storage of data becomes more challenging to enact at scale.

    I recently met with Jakub Lamik, CEO, Redgate Software, to understand the role DevOps, specifically database DevOps, is playing in advancing data-driven digital transformations. According to Lamik, applying DevOps practices for database management can improve performance, increase productivity and help organizations stay compliant. Below, we’ll consider the state of today’s data-driven digital transformations and highlight ways to improve how DevOps is applied to database management.

    State of Data-Driven Digital Transformations

    The 2022 StackOverflow Developer Survey found the most popular databases to be MySQL, PostgreSQL, SQLite, MongoDB and Microsoft SQL Server. But most often than not, businesses deploy more than one of these in production. Redgate’s 2021 State of Database DevOps report found that 70% of businesses had more than one DBMS in use, and 48% had teams working on three or more different databases.

    Engineers operate in increasingly hybrid and distributed technology environments. This isn’t for no reason—in fact, 90% of organizations said multi-cloud is helping them achieve their business goals. Lamik also notes a diverse array of software technologies in practice today, from multiple clouds to decoupled microservices architectures and database types. But the desire to work with diverse toolsets causes fragmentation, which could hinder software delivery efforts. For example, organizations want to deploy the same framework and similar methodologies across different databases, said Lamik, and have it be database- and cloud-agnostic.

    Tips to Improve Database DevOps Operations

    To increase the pace of software delivery, infrastructure automation is often needed. And like other roles, database professionals are increasingly trying to automate operations and free themselves from mundane tasks. So, what are some ways to streamline database DevOps? First off, Lamik encourages adopting a common approach across multi-database environments, using standard toolsets that are flexible and portable. “The organizations that standardize deployments across different database environments can increase the frequency and reliability of deployments,” he said.

    Catching and resolving errors quickly is also becoming increasingly essential to meet objectives. And although DevOps brings many advantages, the CI/CD pipeline has gaps. “Observability supported by intelligent automation is key to closing those security gaps,” said Lamik.

    Thus, he encouraged additional oversight and tracking of changes to ensure stability throughout the database deployment pipeline. Also helpful is encouraging collaboration with security professionals (also known as DevSecOps) and integrating security principles into the DevOps workflow to ensure the process is secure by design.

    Another element of database security involves the proper observability and classification of sensitive data. As Lamik describes, organizations must be careful to distinguish between data used for production purposes and data used for development purposes. Ensuring sensitive data is cataloged correctly is imperative to avoid production data from being breached or accidentally publicly exposed.

    Final Thoughts

    In summary, applying DevOps ideals to database management could be a boon for development efforts. For those who haven’t yet considered it, a DevOps transformation will pay dividends, says Lamik. Furthermore, a common management layer could help wrangle multiple database types and help adopt new emerging database types, such as Snowflake and Postgres. Yet, when acquiring new database toolsets, Lamik admitted it may not be a quick transition. He recommends a land-and-expand strategy rather than a top-down approach (which typically isn’t as effective in empowering individual developers).

    Commenting on the industry’s recent move toward platform engineering, Lamik sees it as the next evolution of DevOps. Whereas DevOps brought helpful toolsets to streamline deployment and digitization, platform engineering is taking these practices and productizing them to be reusable at scale across teams and departments, he explained.

    Also, in the new cloud-native stack, containers are designed to be ephemeral, causing engineers to craft workaround solutions to enable stateful storage on platforms like Kubernetes. Regardless of deployment, data must have proper backup and recovery controls as well. In this new cloud-native paradigm, the role of database professionals is changing, explained Lamik, and “APIs are becoming the focal point of communication.”

  • Cribl Simplifies Telemetry Data Analytics

    Cribl Simplifies Telemetry Data Analytics

    Cribl today announced general availability of a search capability that makes it simpler to query observability data where it resides without having to first collect and centrally store it.

    Nick Heudecker, senior director for market strategy for Cribl, said Cribl Search makes it possible to analyze telemetry data at its point of origin or when it is stored in a low-cost object storage platform to minimize costs.

    As the volume of telemetry data being collected continues to grow, DevOps teams are struggling with ingesting it all. Cribl takes advantage of a platform it created to stream and normalize the collection of telemetry data. Cribl Search adds the ability to also query that data at the endpoint where it resides, said Heudecker.

    While observability has always been a core tenet of DevOps, achieving it has been a major challenge. In addition to deploying and managing agent software to instrument IT environments, there’s a significant amount of time and effort required to set up the pipelines necessary to transfer data to a platform for storage and analysis. The more data is collected, the greater storage costs become. Cribl is making a case for automating those tasks using a tier of software that abstracts data into a format that can be more easily collected and analyzed.

    In general, observability is emerging as the next logical evolution of continuous monitoring. Existing monitoring tools enable DevOps teams to track specific pre-defined metrics. Observability platforms are intended to make it easier to query telemetry data to discover the root cause of an issue that is not being tracked by a set of pre-defined metrics.

    It’s not clear yet whether IT organizations perceive observability to be an evolution of monitoring or a distinctly separate but complementary capability, but it’s apparent IT teams need a lot more context than they have historically been able to attain using legacy monitoring tools. Each platform used within an enterprise typically comes with its own monitoring tools that are used by the IT teams tasked with managing it. Whenever there is an issue, it’s not uncommon for DevOps teams to spend hours correlating data from multiple tools to ascertain the root cause of a problem. Observability platforms promise to reduce that time and effort by applying analytics to data collected from multiple platforms. The challenge is finding a way to efficiently collect all that data, which in the past has often required a data engineer to build a pipeline spanning multiple sources of telemetry data.

    One way or another, observability platforms should make it easier for DevOps teams to manage IT issues more proactively. The goal, as always, is to identify issues before they cause a major disruption. In that sense, it may be difficult to prove the ROI of deploying an observability platform. However, the best IT incident is always going to be the one that never happened or, at the very least, never escalated to a level of noticeable disruption.

  • Study: Real-Time Data Has A Transformative Effect

    Study: Real-Time Data Has A Transformative Effect

    We all know that data is the heart of the modern digital business. And real-time data is arguably the next evolution—having up-to-date data can empower energy savings and optimizations to reduce cost. Real-time data processing also aids fraud detection and improves health care responsiveness.

    Especially in DevOps, access to real-time data is necessary to reduce incident response time and inform ongoing SRE objectives. Leveraging performance data can also empower future decision-making and open new revenue opportunities. In essence, real-time data can have a transformational effect on the operations of a digital business.

    DataStax and ClearPath Technologies recently released the 2022 State of the Data Race Report. The study found that the majority of data leaders are using real-time data to create additional value. The report also suggested that working with cutting-edge real-time data technologies increases developer experience, which is good to maintain amid ongoing talent shortages.

    Below, I’ll review the key findings from the study to infer how organizations can effectively use these technologies and how they can mitigate roadblocks that arise throughout the process.

    The State of the Data Race

    Simply put, customers expect real-time experiences. And it’s not only the end consumers—developers are also increasingly accustomed to building in this modality. This is evidenced by the recent rise of asynchronous, event-driven architectures. The survey depicted how fundamental real-time data is to modern software, finding that 78% of respondents agreed that it is a “must-have,” not just “nice-to-have.”

    The investment appears to be paying off. A full 71% of all respondents said they can directly tie revenue growth to real-time data. Not only does it seem to have a transformative impact on revenue growth, but it enhances productivity, too—66% of real-time data-focused companies agreed that developer productivity has improved. Naturally, the report found that developers are the ones most likely to be working with real-time data.

    Although modern databases and real-time data processing tools are more readily available these days (and open sourced), the maturity of data handling is not homogenous. Based on the classifications of this report, “data leaders” made up only 10% of respondents.

    Roadblocks to Value

    The road to data maturity has a few obstacles that could limit adoption. First is data complexity—33% said this is one of the biggest barriers to leveraging real-time data. This was closely followed by controlling data costs (32%), data accessibility (30%) and available skillsets (28%).

    Most notably, a skills gap continues to be a commonly cited hindrance throughout industry reports and across technology niches. Interestingly, a lack of in-house talent rises slightly higher in organizations with a high real-time data maturity. Those surveyed in this category said that their top barrier is finding the appropriate skills in their business unit.

    Cybersecurity is also a top concern—51% say security challenges are a big concern when it comes to the use of real-time data. As the report surmised, the business-critical nature of real-time data raises its worth as a commodity and area for attack. Furthermore, amid new data sharing regulations, IT departments are now under more pressure than ever to uphold data security and avoid the leakage of sensitive personal information.

    Unlocking Real-Time Data

    This area has a lot of potential; 86% of developers at organizations with a strategic focus on real-time data said that the “technology is more exciting than ever,” the report found. Real-time data is beginning to be applied at the edge, where AI/ML can be trained to give a more responsive advantage to modern applications. Yet, organizations must prioritize its adoption and mitigate the roadblocks mentioned above to unlock the benefits of real-time data.

    Prioritizing real-time data will take more than tools—it will take vision. According to the study, those who created more value with real-time data initiatives were found to:

    • Have clear product owners
    • Have business unit accountability of data
    • And have department representatives working together in a cross-functional team.

    As you can see, these foundational goalposts have less to do with technology and more with lucid leadership and collaboration. That being said, an investment in the appropriate tooling is undoubtedly required to achieve real-time data.

    For example, this tooling might encompass NoSQL databases optimized for real-time or Apache Cassandra for high throughput. The transition to real-time will likely require streaming and messaging technologies as well. Less obvious, however, is an API layer to sit between the applications and the database. This element could greatly help open data accessibility in a standardized fashion.

    The State of the Data Race: Final Thoughts

    Working with real-time data can be a differentiator and have a transformative impact. As a result, most developers and leaders see real-time data as an exciting area. Yet, organizations must reduce data siloes to open up data access to realize its benefits. It will also take collaboration between data scientists and business divisions to construct meaningful outcomes.

    The State of the Data Race report surveyed 500 leaders across various industries. Above, we’ve summarized the key points from the survey. Readers are encouraged to pick up a copy for more nuanced data points and further commentary.

  • Where DataOps and Opportunities Converge

    Where DataOps and Opportunities Converge

    In today’s data age, getting data analytics right is more essential than ever. A robust data analytics implementation enables businesses to hit key performance metrics, build data and AI-driven customer experiences (think ‘personalize my feed’) and capture operational issues before they spiral out of control. The list of competitive advantages goes on, but the bottom line is that many organizations successfully compete based on how effectively their data-driven insights inform their decision-making.

    Unfortunately, implementing an effective data analytics platform is challenging due to orchestration (DAG alert!), modeling (more DAGs!), cost control (Who left this instance running all weekend?!) and fast-moving data landscapes (data mesh, data fabric, data lakehouses …). Enter DataOps. Recognizing modern data challenges, organizations are adopting DataOps to help them handle enterprise-level datasets, improve data quality, build more trust in their data and exercise greater control over their data storage processes.

    What is DataOps?

    DataOps is an integrated and agile process-oriented methodology that helps businesses develop and deliver effective analytics deployments. It aims to improve the management of data throughout the organization. 

    While there are multiple definitions of DataOps, below are common attributes that encompass the concept while going beyond data engineering. Here’s how we define it:

    We broadly define DataOps as a culmination of processes (e.g., data ingestion), practices (e.g., automation of data processes), frameworks (e.g., enabling technologies like AI) and technologies (e.g., a data pipeline tool) that help organizations to plan, build and manage distributed and complex data architectures. DataOps includes management, communication, integration and development of data analytics solutions, such as dashboards, reports, machine learning models and self-service analytics.

    Why DataOps?

    DataOps is attractive because it eliminates the silos between data, software development and DevOps teams. The very promise of DataOps encourages line-of-business stakeholders to coordinate with data analysts, data scientists and data engineers. Via traditional agile and DevOps methodologies, DataOps ensures that data management aligns with business goals. Consider an organization endeavoring to increase the conversion rate of their sales leads. In this example, DataOps can make a difference by creating an infrastructure that provides real-time insights to the marketing team, which can help the team to convert more leads. Additionally, an Agile methodology can be employed for data governance, where you can use iterative development to develop a data warehouse. Lastly, it can help data science teams use continuous integration and continuous delivery (CI/CD) to build environments for the analysis and deployment of models.

    DataOps can Handle High Data Volume and Flexibility

    The amount of data created today is mind-boggling and will only increase. It is reported that 79 zettabytes of data were generated in 2021 and that number is estimated to reach 180 zettabytes by 2025. In addition to the increasing volume of data, organizations today need to be able to process it in a wide range of formats (e.g., graphs, tables, images) and with varying frequencies. For example, some reports might be required daily, while others are needed weekly, monthly or on demand. DataOps can handle these different types of data and tackle varying big data challenges. Add in the internet of things (IoT), such as wearable health monitors, connected appliances and smart home security systems, and that introduces another variable for organizations that also have to tackle the complexities of heterogeneous data as well. 

    OK, so, how can we make this a reality? First, to manage the incoming data from different sources, DataOps can use data analytics pipelines to consolidate data into a data warehouse or any other storage medium and perform complex data transformations to provide analytics via graphs and charts.

    Second, DataOps can use statistical process control (SPC)—a lean manufacturing method—to improve data quality. This includes testing data coming from data pipelines, verifying its status as valid and complete, and meeting the defined statistical limits. This enforces the continuous testing of data from sources to users by running tests to monitor inputs and outputs and ensure business logic remains consistent. In case something goes wrong, SPC notifies data teams with automated alerts. This saves them time as they don’t have to manually check data throughout the data life cycle.

    DataOps can Automate Repetitive and Menial Tasks

    Around 18% of a data engineer’s time is spent on troubleshooting. DataOps enables automation to help data professionals save time and focus on more valuable high-priority tasks.

    Consider one of the most common tasks in the data management life cycle: Data cleaning. Some data professionals have to manually modify and remove data that is incomplete, duplicate, incorrect or flawed in any number of ways. This process is repetitive and doesn’t require any critical thinking. You can automate it by either setting customized scripts or installing a built-in data cleaning software tool.

    Additional processes that can be automated via DataOps include:

    • Simplifying data maintenance tasks like tuning a data warehouse
    • Streamlining data preparation tasks with a tool like KNIME
    • Improving data validation to identify flags and typos, such as types and range

    Building Your Own DataOps Architecture 

    To develop your own DataOps architecture, you need a reliable set of tools that can help you improve your data flows, especially when it comes to crucial aspects of DataOps, like data ingestion, data pipelines, data integration and the use of AI in analytics. There are a number of companies that provide a DataOps platform for real-time data integration and streaming that ensures the continuous flow of data with intelligent data pipelines that span public and private clouds. Looking to increase the likelihood of success of data and analytics initiatives? Take a closer look at DataOps and harness the power of your data.

  • Riverbed Unveils Unified Observability Platform

    Riverbed Unveils Unified Observability Platform

    Riverbed today launched a beta of a unified observability platform it will make available under the Alluvio by Riverbed brand name.

    Jim Hansen, senior vice president for product management at Riverbed, said the Alluvio by Riverbed observability platform combines several existing network and application performance management platforms into a single offering that is now infused with machine learning algorithms to provide deeper insights into IT environments.

    The goal is to provide an open observability platform that can not only aggregate data collected from both Riverbed and third-party offerings but also identify the best ways to resolve an IT issue, added Hansen.

    Scheduled to be generally available later this year, the Alluvio by Riverbed observability platform will differ from existing observability platforms in that it will enable IT teams to not just identify the root cause of an IT issue, but also note which processes should be employed to resolve it, he said.

    With this launch, Riverbed is attempting to reposition itself. Previously, the company offered network and application performance management platforms alongside a wide range of network software and hardware. Each of those offerings is now a source of data that can be fed into the Alluvio by Riverbed observability platform, noted Hansen.

    Observability, of course, has always been a core tenet of any DevOps best practice. Achieving that goal, however, has proven to be elusive. Most of the IT monitoring tools used today are designed to track a set of pre-defined metrics. Whenever there is an issue, IT teams typically convene in a war room where they try to uncover the root cause of, for example, a performance issue by comparing reports and the various dashboards generated by monitoring tools. It can take weeks to discover an issue that often only takes a few minutes to fix once the source of the problem is identified.

    In contrast, observability platforms promise to reduce the time required to discover IT issues by aggregating all the data generated by the IT environment within a single platform that correlates events and makes it easier to identify anomalous behavior. Armed with those insights, it becomes a lot simpler for IT teams to launch queries that enable them to resolve issues faster.

    Those insights are also increasingly being used to drive business decisions as organizations become more dependent on software to drive digital business transformation initiatives. However, a recent survey conducted by Riverbed finds three-quarters of business decision-makers (75%) said their organizations struggle to glean actionable insights from data generated by their technology infrastructure.

    The challenge IT teams inevitably encounter is they don’t always know the right questions to ask, which is an issue that can be addressed by machine learning algorithms capable of identifying anomalies indicative of an IT issue that needs to be resolved.

    There already is no shortage of observability platforms. The issue now is determining which observability platform to employ across a distributed IT environment that is becoming more instrumented with each passing day.

  • How Developers Can Handle the FinServ Data Deluge

    How Developers Can Handle the FinServ Data Deluge

    A decade ago, if you were to ask developers what the biggest challenges in commercial banking were, they would likely cite the difficulty of accessing and collating high-quality data on the day-to-day operations of banking institutions. Today, that’s completely changed.

    Or it has to some extent, at least. Developers working in the banking industry certainly have access to more data than ever before. But this has created challenges of its own. With so much data being collected, stored and at least partially processed, some developers have started to talk of a “data deluge”—an unstoppable torrent that overwhelms the developer’s ability to process and use it.

    In this article, we’ll try and make sense of this. We’ll also explore what it is being used for and what this means for the future of the banking industry.

    Promises and Challenges

    Before we get too deeply into the specifics, however, let’s take a moment to note that the data deluge is not entirely new and that it will be with us for quite some time to come.

    In fact, most guides to technology in the banking sector note that many of our current issues are caused by the sheer amount of data being collected. It has become acutely apparent that our data harvesting capabilities have grown far more rapidly than our ability to ensure accuracy or to label it in accordance with legally-mandated compliance frameworks.

    And, of course, this is only going to get worse. No one knows yet how the global COVID-19 pandemic and ensuing economic downturn will affect the global payments market, but it was previously predicted to reach $2 trillion by the end of 2025, with a compound annual growth rate of 7.83%. Even realizing a portion of that growth will create real challenges for developers in the banking sector.

    Using the Data Deluge

    With all that said, however, let’s also recognize that this is a ‘good problem.’ It’s not necessarily a bad thing that we have access to so much data—the issue is making sure that it’s accurate, safe and useful.

    And when it comes to doing that, three opportunities are already apparent. Here they are.

    1. Increased Velocity

    Many of the new approaches to leveraging data in the banking sector are focused on increasing speed. Indeed, the ability to access real-time intelligence in banking operations was one of the original promises of the big data revolution, albeit one that has taken some years to come to fruition.

    There are several reasons for that, but one of the most important is the fact that banks have lagged when it comes to updating their hardware and software to correlate with the amount of information they are collecting. Most banks now undertake significant levels of quantitative analysis on business and customer data, for instance, but few do so in real-time.

    Instead, many banking systems are still run on monthly data-processing schedules, meaning that it can take even the most technologically advanced bank 30 days to spot consumer or business trends. Implementing new technologies such as Apache Spark is one potential solution to this problem. Like Hadoop, it’s an open source big data analytics engine, but it’s faster, more scalable and easier to use.

    2. More Reliable Governance

    Another challenge when it comes to managing the deluge is that of governance. With so much data flowing in, it can be a challenge for developers and operations staff alike to keep track of its source and how it should be managed within data compliance frameworks.

    This is one reason, as we’ve long argued, that banking should consider DevOps. By uniting developers and operations staff, both can come to a deeper understanding of data compliance and how this must be integrated into every aspect of the contemporary banking industry.

    This is one area, however, in which more data is not necessarily a bad idea. Rather than pausing data harvesting because of concerns that it will create compliance issues, developers should aim to build systems that can carefully and reliably label this data throughout the custody chain.

    3. Focus on the Customer

    Finally, developers shouldn’t lose sight of the ultimate aim of collecting data—improving the customer experience. Data can be used to do this in two primary ways.

    The first is to put data to work in your customer-facing systems. This is the approach recently pioneered by Western Union, which offers an omnichannel approach that tailors and personalizes customer experiences by processing more than 29 transactions a second and integrating all that data into a single platform for statistical modeling and predictive analysis.

    Secondly, data can be used to inform strategic decisions. Just as consumers are now more savvy about the way that their data is used, leadership needs to understand that this type of data-based decision-making is the future of the industry.

    In other words, banks must be able to move faster to transform their data into intelligent insights and then put those insights into action to improve customer service, connect customers to information and products when and where it’s needed most and protect sensitive data and customer accounts from threats.

    The Bottom Line

    The data deluge creates challenges for developers and leaders alike. These challenges are only going to grow as our ability to harvest consumer and industry data continues to expand. If we are going to realize the promises of technology-driven banking, we need to improve the way in which data is managed and processed and ultimately improve our ability to extract meaningful insights from that data.

    And just as this challenge is not new, neither is the outcome—a better banking experience for the customer.

    If you want to learn more about financial services and FinTech development, operations, architecture and leadership through expert-led talks, panel discussions and keynotes, check out FinCon DX—now available on demand!