devopsy.kinsta.cloud

Category: DataOps

  • Why Data Silos Are Killing Your Developer Productivity

    Why Data Silos Are Killing Your Developer Productivity

    Every company is at risk of fostering data silos. In fact, a study conducted by Forrester Consulting for Airtable across employees in the U.S. and the UK revealed that, on average, knowledge workers spend 30% of their week looking for the information they require to do their job and the difficulty in finding and accessing data is the top reason why they feel disengaged from work. What causes essential data to be inaccessible in the workplace? We will explore the root of modern data silos, why they drain developers’ productivity and how to break away from these silos with better observability in your organization.

    What are Data Silos?

    A data silo is a collection of data held by a group that is not easily or fully accessible by other groups in the same organization.

    Data silos can be a byproduct of organizational structure. When different departments or business units in an organization operate independently, they develop their own way of collecting, storing and managing their data. The processes to access that data from other business units can be cumbersome or even nonexistent. Sometimes, the individuals responsible for the data will even be against sharing it for fear that it would be misused or misunderstood outside of their particular unit. The difficulty in accessing information then leads to independent data collection from other business units and ultimately creates other data silos and multiple sources of truth.

    Data silos can also happen during times of growth for an organization. New tools are necessary to support the growing needs of the business and are added to the list of tools already in use. Over time, this list grows, and systems that were designed to operate in isolation can create data silos by limiting the ability to integrate with other systems. A recent survey of marketing organizations by Emarsys showed that the surveyed organizations are using an average of 91 marketing cloud services. These 91 tools undoubtedly create data silos unless they are fully integrated together or into a central system.

    Why are Data Silos Killing Your Developer Productivity?

    Data silos aren’t only slowing your developers down by limiting their access to the data needed for development. They also put a lot of extra work on their plates. And it goes without saying that this additional time spent on data management reduces the time available for improving your product.

    Management of Redundant Data
    Multiple teams in the organization storing their data separately means increased workloads for developers, who need to update and maintain separate copies of the same data.

    Inconsistent Data Quality
    Data gaps or quality issues happen more frequently since different parts of the organization have their own standards for collecting, cleaning and validating data. This can lead to development delays or errors in the product.

    Time-Consuming Integrations
    Developers have to code a lot of integrations to connect all the different tools your organization uses and keep the data flowing between them.

    Inefficient Collaboration
    When different teams work in silos, the chances of misunderstanding are greater, leading to development delays and misaligned priorities.

    Difficulty in Data Governance
    Maintaining high standards of data privacy, security and compliance is time-consuming when the data is spread out in different silos and can create costly regulatory risks.

    Steps Leaders Can Take to Break Down Data Silos

    Fortunately, there are steps that you can take to break down the data silos in your organization, relieving your developers at the same time. Each step may have its own set of advantages and challenges, so the solutions you choose ultimately depend on your organization’s needs and objectives.

    Create a Unified Data Strategy
    Set clear goals and standards for data collection, storage, management and analysis at the company level. You should also implement data governance and security policies.

    Adjust the Culture
    Foster a culture of collaboration and data sharing across the company by encouraging teams to share data, insights, and best practices across different departments. You can achieve this through regular communication, training and incentives for cross-functional collaboration.

    Establish Cross-Functional Teams
    Bringing together members of different departments to collaborate on data-related projects can prove invaluable. These teams should have a shared understanding of the organization’s data strategy and should be empowered to make decisions about data collection, analysis and sharing.

    Invest in Data Integration and Interoperability
    Invest in tools and technologies that promote data integration and interoperability across different systems and departments. Adopt standard data formats, implement APIs and data connectors or use data integration platforms to make observability collaborative. Make sure that the tools you choose to add to your company have various native integration capabilities to avoid custom integrations as much as possible.

    Congrats, you now know all about data silos and how to break them down! The next step for you is to identify the data silos in your company and put a plan in place to get rid of them. Your employees and coworkers will thank you!

  • There’s No Value in Observability Bloat. Let’s Focus on the Essentials

    There’s No Value in Observability Bloat. Let’s Focus on the Essentials

    The observability market paired with cloud security has been predicted to reach $70B by 2026, but that market prediction is a myth, and I’ll tell you why.

    If observability was a grand buffet and every company needed a feast of features and capabilities of this proportion, maybe this prediction would be valid. But most companies don’t need an over-the-top feast. Legacy APM and observability solutions are serving up an all-you-can-eat buffet when the reality is that most companies only need the essential ingredients to satisfy their observability strategy. And the cost for that can be much more modest. The $70B observability market prediction is, therefore, suspect, considering the true needs of most organizations.

    Complex cloud ecosystems, the rise of applications built on microservices and Kubernetes, and the overall scale and complexity of applications have led to exponentially more data being created in the pursuit of observability. Storing and processing all that data is expensive, and as the volume of data balloons, so does the cost.

    And while the complexity of business-critical systems and the volume of data have scaled, so have the overall cost of those legacy tools—in direct proportion. Partially, this is driven by the features and instrumentation added that only contribute to the bloat and valueless telemetry.

    Customers aren’t seeing value in any of this bloat, and for the most part, all that’s been accomplished is increased complexity in deploying and operating the system, with a commensurate lengthening of time to value. Plus, pricing models haven’t significantly changed despite the advancements in the market. The fact is, organizations can’t really say that they have gotten more insight into the overall performance of their mission-critical systems. What they are saying, however, is that these tools are becoming overwhelming to manage and overwhelmingly expensive, and developer productivity is not increasing.

    What organizations need and want are real-time insights into the health and performance of their cloud infrastructure and applications–specifically the confluence of logs, metrics and traces within the context of modern observability. There is little tolerance for software bugs, slow web experiences, crashed apps and other service interruptions, and the responsibility lies on developers, engineers and ITOps teams to quickly resolve production issues before they impact customer experience.

    There is a call for observability, but rather than an explosion of features and noise, what’s being called for is a narrowing of focus.

    Observability Has an Efficiency Problem

    Telemetry data gathered from the distributed components of modern cloud architectures needs to be centralized and correlated for engineers to gain a complete picture of their environments.

    Engineers need a solution with critical capabilities such as dashboarding, querying and alerting, and AI-based analysis and response, and they need the operation and management of the solution to be streamlined.

    What’s important for them to know is that it’s not necessary to spend more to ensure peak performance and visibility as their environmental complexity grows.

    Legacy players experiencing some maturity in the market may have conditioned them to believe this is true—they have set a standard for higher costs correlating with incremental amounts of data, no matter the quality of that data. Companies are being fed the lie that all this data may be relevant and they’ll be doing themselves a disservice by neglecting even one byte of it.

    Data Does Not Inherently Equal Insights

    No doubt, more data is being generated, but most of it is not relevant or valuable to an organization. Observability can be optimized to bring greater value to customers, and that’s where the market is headed.

    Call it “essential observability.” It’s a disruptive vision to propose a re-architected approach to observability, but what engineers need is a new approach making it easier to surface insights from their telemetry data while deprioritizing low-value data. Costs can be reduced by consuming only the data that enables teams to maintain performance and drive smart business decisions.

    Essential observability is cutting out the fluff and only focusing on—and paying for—the impactful data. It’s about arming engineers with meaningful insights so they can quickly resolve production issues and business disruptions. Organizational performance increases without inflating data complexity and resource consumption.

    The Essential Observability Revolution

    Observability tools have evolved beyond the model of linear price increases as complexity and data scales. For companies (especially growth-stage organizations) that want a partner to help them make more informed decisions and keep their costs more reasonable (i.e., stop recklessly spending for useless data), they need to focus on tools offering them the essentials.

    Solutions are available, offering targeted features and automated capabilities out of the box, delivering the insights needed to understand and fine-tune data. These solutions provide continuous support, reduce complexity and accelerate time to value.

    The next revolution in observability isn’t features and functionality but providing insights to make data-driven decisions that keep essential observability cost-effective while not compromising the value derived.

  • Why Enterprises Should Embrace Data-Driven Software Management

    Why Enterprises Should Embrace Data-Driven Software Management

    Many Fortune 500 companies have recently undergone extensive layoffs and now find themselves having to achieve more with fewer people—while grappling with how to integrate generative AI into their products. They must build while economizing, streamlining and increasing efficiency in every part of their company. Yet one of the most vital cogs of the machinery has struggled to evolve over the past 20 years.

    In fact, until now, managing high-tech engineering teams at the world’s biggest companies has not been all that high-tech.

    Back in the 1970s, the “waterfall” model was introduced: An inflexible process in which each phase of software development depended on the previous one’s completion. Then, around the 2000s, companies started moving towards “agile” models that tracked progress in two-week sprints.

    Now, a third era of software development management is emerging. It’s not waterfall or agile; it’s data-driven.

    Current management systems cannot give managers real-time, granular insights so they can make decisions on the spot and run projects in the most efficient way possible. However, data-driven models generate live data on how teams are progressing which has far-reaching effects on businesses’ productivity and future survival.

    The emergence of DORA, a framework for tracking development metrics makes this possible. For the first time, new tools enable us to capture the software development life cycle automatically, without any manual input from the engineer, leading to fewer roadblocks, a 20% faster cycle time and greater ROI.

    Despite this, only a handful of Fortune 500 companies have embraced data-driven metrics. Here are the reasons that shift should happen now.

    Real-time Insights are Game-Changers

    Until now, companies have been using inaccurate methods to track their software development teams. Project management reports on engineers’ output are reviewed every two weeks, monthly, or even quarterly. This means that impactful changes are made at a delayed point in time rather than when they can make the greatest difference to cycle time.

    In contrast, DORA measures specific real-time performance indicators: Deployment frequency (DF), lead time for changes (LT), mean time to recovery (MTTR) and change failure rate (CFR) (For more information, see here). These far more data-driven metrics measure efficiency and velocity at every key point of day-to-day software development, with feedback to management happening continually.

    Often, individual engineers won’t have enough capacity or oversight to inform managers of specific issues during a cycle. Using DORA metrics is like being able to detect that a team has an abnormally high BPM though they are in a rest phase. It’s a signal of illness; that more or different resources are needed. That then requires you to go in and inspect the micro factors that can be adapted.

    For any Fortune 500 company, this quick turnaround is vital. Any small issue in a product cycle can easily escalate and become a critical problem in a matter of days. Instead, using DORA can accelerate software development and, ultimately, time to market.

    DORA Gives you Internal Benchmarks

    But, how can you tell if something is wrong while using DORA? In the first few months of adoption, it’s likely that companies will compare their internal data to industry benchmarks. Unprecedented data has recently been published that offers general performance benchmarks for each DORA metric based on industry research.

    But the end game is to be able to compare your DORA metrics to your own internal benchmarks. With time, each company will develop its own personal baselines that better represent internal capabilities and limitations based on innumerable variables.

    The purpose of DORA benchmarks, however, is not to draw a fixed red line. Think of it more like a speed limit that ensures that people are working at a steady pace while allowing for flexibility.

    A clear sign that your team may be burnt out is a really low cycle time. At times it can be a technical bottleneck, but the beauty of DORA is that it can also point you towards well-being issues.

    DORA Needs to be Coupled With Granular Insights

    DORA is unparalleled when it comes to flagging managers that something is wrong. DORA metrics can flag that a team has a high cycle time – for example, an abnormally long deployment frequency suggests that engineers are working more slowly or that there may be a build-up of code that isn’t being uploaded to the repository quickly enough during the review process. At that point, you need to zoom in to get more information. There are multiple metrics that provide insights into the exact pain points in a team’s development, and you can develop these internally or via tools or engineering management software that provides more holistic oversight.

    In this particular case, you can use a granular metric called “time to merge from first review” to see how fast engineers incorporate feedback from their peers during code reviews. If it is taking a week for the first review to be carried out, integrated into the code and finally uploaded to the general platform, then perhaps there’s a bottleneck occurring during the review process.

    Perhaps the most important benefit of Engineering Leadership 3.0 is that it no longer has to be a guessing game. If they sense something’s wrong, they can check their DORA dashboard, receive confirmation of the problem, hone in with granular metrics, and take corrective actions immediately.

    Teams that adapt their behavior as a result of these data-driven metrics will add incredible ROI to the company. But these changes can’t happen overnight. It will start with getting buy-in from engineering managers and the executive team. For Fortune 500 companies, that often requires elaborating action plans, integrating external tools to operate the tracking process and training engineering managers (in our case, this takes up to five weeks). But once they’re a part of the business, they will become indispensable in running smoothly without stacking up issues.

  • Oracle Bill is 5x Client’s Budget ¦ Toyota Out of Space

    Oracle Bill is 5x Client’s Budget ¦ Toyota Out of Space

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Birmingham looks like the Detroit of the UK—is it Oracle’s fault?
    Plus: Was Toyota’s factory failure caused by running out of disk space? (more…)

  • 80% of Bosses ‘Regret’ Stopping WFH ¦ PSA: Disable STS!

    80% of Bosses ‘Regret’ Stopping WFH ¦ PSA: Disable STS!

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Rethinking return-to-office mandates and a ridiculous, ancient Windows bug. (more…)

  • Why Your Data Lake Strategy is Broken for Observability

    Why Your Data Lake Strategy is Broken for Observability

    Someone once said, “Life is better at the lake.” That may be true in the physical world, but when it comes to using data lakes for observability, that’s not typically the case.

    Just about every organization in every industry has a digital presence, whether it’s a website, an app or another digital service. This makes observability a universal need; it helps teams monitor software performance, optimize performance, detect potential security issues and quickly monitor for bugs if and when they occur. These responsibilities typically fall on engineering teams and require as close to real-time response as possible.

    Some organizations also have business intelligence (BI) needs, aimed at generating insights from multiple data sources for business decision-making. These responsibilities typically fall on business decision-makers; they are not real-time decisions, but rather made by looking back over long time periods.

    All-In on a Data Lake Strategy

    Companies that go all-in on a data lake strategy may ignore the needs of the engineering teams working on the real-time mission-critical aspects of the business and force all of the company’s data, queries, dashboards, etc. into a data lake solution. That doesn’t actually meet the needs of the engineering teams. The result? Engineering teams use unsanctioned solutions, fracturing the company’s data strategy from the ground up.

    By using a standards-based observability solution built on open source InfluxDB 3.0, companies can give their engineers the observability tools they need while preserving their data lake strategy.

    Here are three main challenges that teams come across when using data lakes for time-series, and how to solve the problem with InfluxDB 3.0.

    Problem One: Inefficient and Slow Ingest
    Data sources generate time-series data at an incredibly high rate, easily producing hundreds or thousands of data points every second; some sources even do so as quickly as every nanosecond. Data lakes can store massive volumes of data, but they aren’t optimized to quickly ingest and make it readable on a millisecond time scale.

    Data lake ingest jobs can be measured in minutes or even hours, which is far too late for any meaningful interventions that can prevent catastrophes such as downtime. Customers will notice the outage before the data is even ingested into the data lake, much less acted upon.

    Problem Two: Time-Series Life Cycle
    Full-fidelity data is necessary for real-time analytics and observability, but the value of the full-fidelity data fades over time. Storing the full-fidelity data in a data lake may be convenient, but it is costly and not necessary for typical BI purposes.

    InfluxDB is designed for the time-series data life cycle. This life cycle entails:

    1. The leading edge of data. That is, data written in the last two hours or so. This is the data that is most likely to be queried, so the full fidelity data is kept in memory ready to query instantly.
    2. Compacted data. That is, data older than the leading edge that is still likely to be queried and requires full fidelity is compacted off to disk in object store, which still permits fast querying but does not incur the costs of storing data in memory.
    3. Retention policies. That is, data that has lost its value in its full fidelity form. For example, for observability data, data older than a few weeks can be aggregated into a much smaller data footprint. Then, this aggregated data is ingested into the company’s data lake while the full fidelity data is automatically dropped from InfluxDB 3.0.

    Without such a system, it is necessary to store the full fidelity data in the data lake and incur those massive costs without a reasonable way to manage the life cycle of the data. With such a system, the correct amount of observability data can be ingested into your data lake to support your BI needs.

    Problem Three: Slower Query Performance
    Engineers can experience a high degree of frustration when they are troubleshooting an issue in real-time, but their dashboards won’t load or their queries won’t return in a reasonable amount of time. Data lakes simply are not designed to produce fast results for observability, and so query performance is a primary complaint of engineers trying to use a data lake for observability. Waiting minutes for a query to return when you are running a BI report may be annoying, but it could spell extended downtime for an engineering team.

    Fortunately, InfluxDB is built on the Apache Arrow standards, which means that it can be directly queried by your engineering team. Via protocols such as Delta Sharing and Iceberg, anyone can query into InfluxDB 3.0 from their data lake and join the real-time observability data stored in InfluxDB 3.0 with the data lake data. This is akin to having your cake and eating it too.

    The Bottom Line

    If you are trying to use a data lake strategy to fulfill your engineering team’s observability needs, you are going to frustrate those teams because the data lake is not designed for data ingestion, the data life cycle, or the query needs of those teams. This will cause those teams to create rogue solutions that are inconsistent with your company’s data lake strategy.

    By providing those teams with open source InfluxDB 3.0, engineering teams will have a first-class observability platform that is designed to interoperate sensibly with your company’s data lake. This enables the critical mission of your observability platform while maintaining your overall data lake strategy for your BI needs.

  • US DoJ Makes PyPI Give Up User Data ¦ Tape Storage: Not Dead

    US DoJ Makes PyPI Give Up User Data ¦ Tape Storage: Not Dead

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: PyPI complies with a “string of subpoenas,” and LTO continues to grow, despite predictions of its demise. (more…)

  • Linux 6.3: What’s New ¦ AWS Layoffs are a Worry

    Linux 6.3: What’s New ¦ AWS Layoffs are a Worry

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: A new Linux kernel drops and layoffs at Amazon Web Services point to trouble. (more…)

  • Linux Tweak Brings Big Speedup ¦ DCs in SPAAACE (Redux) ¦ Atlassian Fires 500

    Linux Tweak Brings Big Speedup ¦ DCs in SPAAACE (Redux) ¦ Atlassian Fires 500

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Intel optimizes Linux multithreaded networking, data centers in space (again), and more DevOps toolmaker layoffs. (more…)

  • How Embedded Analytics Can Transform Future Businesses

    How Embedded Analytics Can Transform Future Businesses

    It’s no secret that big tech companies like Google, Amazon and Netflix are not only leveraging data but also delivering embedded analytics and insights as a standard functionality of their software, services, applications and toolkits. The analytics appear to consumers as recommendations, usage habits and other helpful information. Such insights not only help consumers extract the most value out of the product they are using but also help providers understand how they can provide better services. Businesses that embed analytics to provide a differentiated user experience also have a direct impact on engagement and revenue.

    Have you considered how embedded analytics could differentiate your company’s own software and hardware products? Let’s understand how embedded analytics can help transform technology businesses:

    1. Analytics are expected and welcomed in your products

    By embedding analytics into existing enterprise software and applications, users can drill down into real-time and context-aware data (for example: How busy a location will be at various times of the day) and can benefit from faster data-driven decision-making without worrying about application compatibility. Commonly, software applications provide in-app analytics to delight and serve their customers, while hardware companies can provide helpful stats on a web portal or app.

    2. Embedded analytics is easier than ever today

    Cloud technology has completely disrupted tech-enabled industries. It’s easier than ever for software, apps and hardware to connect to a data repository to save logs that are a rich source of analytics. The invention of the S3 bucket and object store has made it fairly cheap to keep log data. Finally, the transition to platform-as-a-service (PaaS) and software-as-a-service (SaaS) has fundamentally added an “easy button” to tracking consumption patterns and resource utilization.

    3. Embedded analytics helps design better products and services

    Embedded analytics can give businesses more insight into application telemetry (how customers use the data, how they engage with the product) and provide insights into the challenges they face from a usability standpoint. Providers can use advanced analytics to quickly iterate, draw patterns and make predictions, troubleshoot customer challenges, discover new avenues to monetize, detect and mitigate security threats or create entirely new features and applications that put customers on a faster path to improved application experience and better decision-making.

    4. Embedded analytics increases operational efficiencies

    The tech industry is transitioning from traditional monolithic architectures where applications are self-contained to microservices architectures where different components are spun up as needed via API calls. By embedding analytics directly into developer tools, businesses can manage resources without sacrificing availability or performance. Companies can allocate and optimize resources dynamically and understand peaks and valleys in consumption. Businesses must be able to leverage historical and real-time data to build predictive models to proactively identify and mitigate issues with individual components of the application before the customer experiences a problem.

    Embedding Analytics in Your Application

    Before you start embedding analytics in your application, it’s important to choose the right database foundation, as this will support current needs and future goals. Below are the top capabilities that should be standard features in analytics databases:

    1. Handles massive data volumes: The database must scale easily and efficiently with minimal cost using massive parallel processing. Today’s scale might be terabytes, tomorrow it might be petabytes and so on.

    2. Enables speedy data analytics: Speed is a key factor in a positive user experience. Users want instant insights because they don’t like waiting for results. The database should be capable of optimizing load and query performance so that analytics is both timely and relevant.

    3. Embeds machine learning: Machine learning algorithms are critical for predictive analytics. A platform that can embed ML and iteratively learn from new data can make a big difference and add a whole new level of intelligence.

    4. Features native integrations: The more time developers spend figuring out tools and integrations, the less time is spent innovating. The platform must ideally have out-of-the-box integrations with popular applications, data sources (on-premises, cloud, multi-cloud or hybrid), ETL, and visualization tools (e.g., Microstrategy, Looker, Power BI, Qlik, Tableau). It should be flexible enough to support user-defined functions for things that don’t have native integration.

    5. Encourages self-service: The database should make it easy for data scientists to leverage tools and programming languages they already know such as R, Java, C/C++, Python and SQL. Data scientists will also appreciate a platform that has built-in functions for a wide range of analytics (such as geospatial, time-series, event-series, real-time, machine learning, text analytics, pattern matching and regression) and delivers the tools to spot trends, uncover anomalies and anticipate the unexpected.

    6. Supports highly advanced analytics: Many platforms can analyze single tables or do a simple look-up but few have the power to analyze dozens of tables, hundreds of data types, dimensions and attributes from numerous sources over several years of history.

    7. Unified: The database should be able to streamline everything right from data ingestion to exploration to production until delivery. Users should be able to analyze data in its place to always have a single source of truth.

    It’s not hard to visualize that the most disruptive companies of the future will be the ones that harness data as a core asset and embed analytics in their products and services. Embedding analytics requires the right database foundation; to get to the right foundation, IT teams need to cut through the noise and study the fine print. With the proper database, the future is yours to create.