devopsy.kinsta.cloud

Tag: data

  • Why You Need to Break the Observability Data Silo

    Why You Need to Break the Observability Data Silo

    In my role as a developer advocate for Coralogix, I speak with organizations across almost every conceivable industry, and there is a pattern across almost all of them. In the world of observability, the challenge is battling scale. At scale, observability becomes a complex problem to solve. Surprisingly, the most common pattern for handling observability data is also holding people back from unlocking the true power of their data.

    What is the Data Silo in the World of Observability?

    To understand the data silo, we first need to understand the typical architecture of an observability solution, whether it’s homegrown or SaaS. Observability data can be divided into three types: Logs, metrics and traces. These three types come together to give engineers a complete picture of the health and performance of their system.

    These three data types are typically held in different storage mechanisms. A popular solution for logs is open search, while Prometheus holds the mindshare for metrics, and Jaeger is a very popular tracing solution. This means three solutions, three interfaces and, subsequently, three problem-solving workflows.

    What’s wrong with separating data like this? Even though there are three types of data, there is still only one goal: To understand the health and performance of the system. This is the only use case. The data silo is the dominant trend for observability solutions to provide a fragmented view of the data. It requires engineers to piece together stories as they go, rather than correlating across all of this data at once, to provide a complete picture of system performance.

    This slows down troubleshooting. If there is an issue, an engineer will first review one of the data types. Then, they will run different queries against other data types to confirm. So they may detect a 500 error in the logs, then move to the traces to see the impact on traffic, and finally, review the application metrics. This means three minimum queries when attempting to assess only one problem.

    Breaking the Data Silo

    Breaking the data silo can be a daunting challenge. There are some key points to focus on when a company starts out and some finishing touches that make for an excellent developer experience.

    Unify data storage as much as possible. For example, traces can be stored in Prometheus rather than in a separate solution. This will enable engineers to query, for example, trace latencies alongside CPU metrics. It will also minimize the UI work necessary to join together data. If an organization has the expertise, it’s possible to store everything inside an OpenSearch cluster. This means a single interface, with a single query language, to query all available observability data. Anyone assessing this option should simply decide what their company can maintain and what skills they have in-house.

    Present observability data as one. If storage unification isn’t an option, then focus on the presentation layer. Tools like Grafana can consume most observability solutions as external data sources and render them all on the same dashboard. While an engineer will still have to query them using the specific language of the tool, it means that they can render things alongside one another, making for an insightful dashboard.

    Correlate data in one user interface. This means metrics are rendered in a single dashboard alongside logging and tracing data. This will give users a static view of the data at once, even if there is a limit to how queryable the data is.

    Create universal values. If it’s impossible to make any of the previous recommendations, then this is an easy, quick win that will at least make querying observability data more consistent. An easy way to do this without making your code messy is to use a library like MDC (or your tech stack’s equivalent). Add some tags or values that will exist in all logs, metrics, and traces. For example, the name of the application and the feature it belongs to, i.e. application is credit-card-service and feature is payments. This means that if something goes wrong, there is already a set of key-value pairs that translate across every data type.

    At the very least, this will mean an engineer can issue consistent queries across the three solutions, albeit in different syntaxes. This represents the least value, but also the least rework and effort, and will lower the cognitive load of engineers who are already trying to debug a potentially serious outage.

    The Data Silo Can be Beaten

    A siloed observability strategy drives waste and inefficiency in engineering teams, precisely when they need efficiency the most. By breaking the data silo, observability data can be queried, analyzed, and correlated together to form a single, cohesive and complete picture of your system health.

  • Stream Big, Think Bigger: Analyze Streaming Data at Scale

    Stream Big, Think Bigger: Analyze Streaming Data at Scale

    Data streams are all the rage. Once a niche element of data engineering, streaming data is the new normal—more than 80% of Fortune 100 companies have adopted Apache Kafka, the most common streaming platform, and every major cloud provider (AWS, Google Cloud Platform and Microsoft Azure) has launched its own streaming service.

    This shift is fueled by the growing demand to deliver events quickly, reliably and at scale to support a wide range of internal and external applications. From always-on, customer-facing applications to mission-critical operations, a growing number of use cases depend on subsecond updates—further upping the ante for real-time data processing and dissemination.

    There’s no doubt 2023 will be a big year for streaming technology.

    Meanwhile, as streaming moves toward ubiquity, we’re seeing another shift in the way businesses use the data: Analyzing events as they’re created for real-time insights. With the right tools in place, people can instantly compare what’s happening now to what happened in the past, triage problems as they occur and make time-sensitive decisions in the moment.

    With the proliferation of data streams, a new set of use cases and requirements for real-time analytics has emerged. To unlock the power of streaming in 2023, data teams will need to look beyond the traditional batch-oriented stack and adopt a streaming-native approach to the analytics architecture.

    The Streaming Paradigm Shift

    Data has always been at the center of how businesses operate. Until recently, we lived in a very batch-dominant world of business and data. The goal of data infrastructure was to identify data at a fixed point in time and store it for eventual use. The evolution from daily batch operations on mainframes to the internet-driven always-on world replaces fixed, static “data at rest” with fast-moving data in motion, as information flows freely between applications and data systems within and between organizations.

    Data at rest still exists, and batch systems continue to serve a variety of reporting purposes. But reality isn’t fixed or static, and to meet the demands for seamless, authentic data experiences, the systems we build must be designed for data in motion.

    Hence, the rise of streaming technology—and with it, a new mindset for how we think about data. As streaming increases, streaming data platforms increasingly act as an organization’s central nervous system, connecting the different functions and driving mission-critical operations. Evolving technologies are purpose-built to support data in motion systems: Stream processors and event databases.

    Apache Druid fits into the purpose-built category; as a real-time analytics database, it is designed to enable users to query every event as it joins the data stream and do it at a tremendous scale while enabling subsecond queries on a mix of stream and batch data.

    Thousands of businesses are already using streaming processors like Kafka and Amazon Kinesis with Druid to build cutting-edge systems that make terabytes to petabytes of streaming data accessible to applications and humans in milliseconds. Some of those businesses—including Reddit, Citrix and Expedia—were highlighted at Current 2022, the annual streaming event organized by Confluent.

    The next evolution of data intelligence is here, and it’s based on the ability to react to events as they are happening. But on the growth chart of streaming adoption, we are only at the beginning of an upward curve—a critical inflection point where streaming (and the technology that is built for it) starts to become the bedrock of everyone’s data architecture.

    So, when it comes to enabling scalable, subsecond analytics on streaming data, many developers and data innovators are wondering, ‘How?’

    We’re Streaming! Now What?

    At Current, we spoke with hundreds of Kafka users who were turning their attention to building out the next phase of their data streaming pipelines and embedding them more deeply into their organizations.

    While streaming adoption is becoming more widespread, most companies still have just one or two use cases they’re solving with a streaming platform. At Current, many people spoke about how Kafka effectively set their data in motion. But when it came time to analyze or put those streams to work in a user-facing application, their “data in motion” turned into “data in waiting” because their analytics system was designed for batch data, not streaming.

    To solve this problem, a new type of database is needed—and that’s why Druid was created. Turning billions or trillions of events into streams that can then be immediately queried by thousands of users simultaneously is where stream processors like Kafka, in combination with Druid, have unlocked a new set of capabilities that are enabling new developer-built analytics applications.

    For example, Reddit generates tens of gigabytes of event data per hour from advertisements on its platform. To let advertisers understand their impact and decide how to target their spending, Reddit needs to enable interactive queries across the last six months of data—hundreds of billions of raw events to slice and dice. But they also needed to empower advertisers to see user groups and sizes in real-time, adjusting based on interests (sports, art, movies, technology…) and locations to find how many Reddit users fit their target. To achieve this, they built a Druid-powered application that ingests data from Kafka and enables Reddit’s advertising partners to make real-time decisions and get the best possible ROI on their campaigns.

    Stream-to-Batch Vs. Stream-to-Stream

    Reddit chose Druid as the database layer of their application in large part because of its close integration with Kafka, as Druid was designed to ingest and analyze streaming data. This sets Druid apart from virtually every other analytics database, the rest of which were built for batch ingestion.

    Batch ingestion works exactly the way it sounds—it takes chunks of stream data, dumps them into a file, processes the file, then loads it into the database. Using a batch-based system to analyze streams introduces a bottleneck to your streaming pipeline. By contrast, Druid provides a connector-free integration with the top streaming platforms and handles the latency, consistency and scale requirements for high-performance stream analytics cost-effectively.

    In addition to out-of-the-box integrations, Druid has built-in indexing services that provide event-by-event ingestion. This means streaming data is ingested into memory and made immediately available for use. No waiting for events to be processed through batch processes before they can be queried. This capability, coupled with exactly-once semantics, guarantees data is always fresh and consistent. If there’s a failure during stream ingestion, Druid will automatically continue to ingest every event exactly once (including events that enter the stream during the outage) to prevent any duplicates or data loss.

    However, perhaps the most powerful reason to use Druid for streaming data analytics is its near-infinite scalability. For even the largest, most complex ingestion jobs, Druid can easily scale into the tens of millions of events per second. With variable ingestion patterns, Druid avoids resource lag (as well as overprovisioning) by enabling dynamic scaling. It combines the query and ingestion performance of shared-nothing architecture with a cloud data warehouse’s flexibility and non-stop reliability. This means you can add computing power and scale out without downtime or rebalancing, which is handled automatically.

    In essence, while using a cloud data warehouse to serve both batch-oriented and real-time use cases might sound efficient, doing so defeats the purpose of a streaming pipeline.

    “The weakest link in the chain sort of dominates everything,” Databricks CEO Ali Ghodsi told the audience at Current. “If you have one step of your pipeline that is batch-processed and really slow, then it doesn’t matter how fast you are in the rest of the pipeline.”

    That’s why when Confluent’s data engineering team needed to build a real-time observability platform, they chose to use Kafka and Druid together. Today, the platform ingests over five million events per second from Kafka and handles hundreds of concurrent queries from a large team of analysts.

    “We do a lot of things today that we didn’t imagine doing three or four years ago,” Matt Armstrong, who leads the observability and data platform team at Confluent, said in a Druid Summit 2022 interview. “From a [product] perspective, you could think of Druid as being able to power a great product [Confluent Cloud] with a great support experience—but perhaps more importantly—an unconstrained future.”

    It’s Time to Think Big

    We’re already moving to this entirely new way of thinking about moving, analyzing, and sharing data. Streaming technology has opened the door for a plethora of new use cases and products to come to fruition. We believe organizations that view streaming data as a force multiplier and implement a streaming-native approach to analytics will gain a competitive advantage in 2023 and beyond.

    Druid is purpose-built for these new streaming use cases, but it’s also built for a new type of application—one that draws characteristics from both the analytics and transactional database worlds. Industry-leading teams have built mission-critical applications that enable operational visibility, customer-facing analytics, drill-down exploration and real-time decision-making using streaming data and Druid. This trend is now starting to take hold and the next wave of analytics applications is already being built.

  • Why GraphQL Will Reign King

    Why GraphQL Will Reign King

    After years of development, GraphQL is set to reign as the go-to API for data fetching. But what is GraphQL? Simply put, it’s a query language that allows you to fetch data from any server, and it’s a noteworthy alternative to the likes of other web service development styles like representational state transfer (REST) and simple object access protocol (SOAP). There are a few reasons why GraphQL will dominate the DevOps world, including its overall efficiency, ability to work with any server and overall ease of use.

    Understanding GraphQL

    GraphQL is a query language for APIs and a runtime for fulfilling queries with existing data. GraphQL provides a complete and digestible description of the data in an API, gives clients the power to ask for exactly what they need, makes it easier to evolve APIs over time and enables powerful developer tools.

    Top Benefits of GraphQL

    GraphQL addresses many pain points that often slow down developers, who prioritize adaptability and working quickly and efficiently when building applications.

    For one, it is much more flexible than other data query languages, allowing users to specify exactly what data they want and in which format. Other data query languages require users to retrieve all data in a specific format, regardless of whether they actually need the data. That excess, unnecessary intake can cause delays and confusion.

    GraphQL’s ease of use is also invaluable, thanks to its simple and intuitive syntax. The language is strongly typed, which means developers can catch errors early on in the development process before code goes into production. This can save much-needed time and money in the long run.

    Another benefit is that it can be used to fetch data from multiple sources because it uses a schema that defines the different types of data available. This means if a developer has a field in a schema that could potentially return data from multiple sources, they can specify which source to fetch the data from. Developers don’t get this benefit from other data query languages, which often allow data fetching from only a single source.

    Advantages for DevOps Teams Over Other Data Query Languages

    As the world of software development continues maturing, so do the tools and technologies DevOps teams use to build and operate their applications.

    REST and SOAP are popular web service technologies, but they come with some major drawbacks. Notably, both REST and SOAP rely heavily on URL structures and verbose XML payloads, which makes it difficult to evolve the API without breaking existing clients. And because each endpoint is a unique URL, providing a consistent experience across different parts of the API can be challenging. REST and SOAP are also quite chatty and can make it hard to optimize performance.

    In contrast to REST, GraphQL provides a single endpoint for all data requests and is much faster than making many HTTP requests. GraphQL’s declarative query language specifies precisely the data needed – no more, no less – increasing bandwidth and improving performance. Applications can run faster and use fewer resources, which is especially important for mobile apps that need to be lightweight and fast.

    Understanding the data is crucial for DevOps teams, and GraphQL allows developers to visualize data in a graph. Teams can see the relationships between different points more easily, leading to better-informed decisions about the best ways to manipulate the data.

    The increased efficiency when conducting API calls is also a plus with GraphQL, which helps teams retrieve all information they need with a single API call. Traditional REST APIs often require multiple API calls, mainly because each resource in a REST API is represented by its own unique URL, along with its own set of permissions and data that needs to be retrieved. GraphQL can eliminate this bottleneck.

    Overall, dealing with complexity is significantly reduced when developers use GraphQL. When using traditional REST APIs, writing a lot of code to handle different edge cases and error conditions is the norm. But GraphQL streamlines this – much of the complexity can be handled by the server itself, making way for simpler and more maintainable codebases. DevOps teams are also able to decouple their front-end and back-end development efforts by using a schema-based approach. GraphQL enables frontend and backend components to be developed separately without worrying about breaking changes, which means faster development cycles and fewer errors.

    The amount of data that organizations need to understand, organize and query will only increase in the coming years. DevOps teams can get ahead of the ongoing data and information onslaught by taking advantage of GraphQL and its unmatched ability to provide an efficient, powerful and flexible API for data fetching and manipulation.

  • How Embedded Analytics Can Transform Future Businesses

    How Embedded Analytics Can Transform Future Businesses

    It’s no secret that big tech companies like Google, Amazon and Netflix are not only leveraging data but also delivering embedded analytics and insights as a standard functionality of their software, services, applications and toolkits. The analytics appear to consumers as recommendations, usage habits and other helpful information. Such insights not only help consumers extract the most value out of the product they are using but also help providers understand how they can provide better services. Businesses that embed analytics to provide a differentiated user experience also have a direct impact on engagement and revenue.

    Have you considered how embedded analytics could differentiate your company’s own software and hardware products? Let’s understand how embedded analytics can help transform technology businesses:

    1. Analytics are expected and welcomed in your products

    By embedding analytics into existing enterprise software and applications, users can drill down into real-time and context-aware data (for example: How busy a location will be at various times of the day) and can benefit from faster data-driven decision-making without worrying about application compatibility. Commonly, software applications provide in-app analytics to delight and serve their customers, while hardware companies can provide helpful stats on a web portal or app.

    2. Embedded analytics is easier than ever today

    Cloud technology has completely disrupted tech-enabled industries. It’s easier than ever for software, apps and hardware to connect to a data repository to save logs that are a rich source of analytics. The invention of the S3 bucket and object store has made it fairly cheap to keep log data. Finally, the transition to platform-as-a-service (PaaS) and software-as-a-service (SaaS) has fundamentally added an “easy button” to tracking consumption patterns and resource utilization.

    3. Embedded analytics helps design better products and services

    Embedded analytics can give businesses more insight into application telemetry (how customers use the data, how they engage with the product) and provide insights into the challenges they face from a usability standpoint. Providers can use advanced analytics to quickly iterate, draw patterns and make predictions, troubleshoot customer challenges, discover new avenues to monetize, detect and mitigate security threats or create entirely new features and applications that put customers on a faster path to improved application experience and better decision-making.

    4. Embedded analytics increases operational efficiencies

    The tech industry is transitioning from traditional monolithic architectures where applications are self-contained to microservices architectures where different components are spun up as needed via API calls. By embedding analytics directly into developer tools, businesses can manage resources without sacrificing availability or performance. Companies can allocate and optimize resources dynamically and understand peaks and valleys in consumption. Businesses must be able to leverage historical and real-time data to build predictive models to proactively identify and mitigate issues with individual components of the application before the customer experiences a problem.

    Embedding Analytics in Your Application

    Before you start embedding analytics in your application, it’s important to choose the right database foundation, as this will support current needs and future goals. Below are the top capabilities that should be standard features in analytics databases:

    1. Handles massive data volumes: The database must scale easily and efficiently with minimal cost using massive parallel processing. Today’s scale might be terabytes, tomorrow it might be petabytes and so on.

    2. Enables speedy data analytics: Speed is a key factor in a positive user experience. Users want instant insights because they don’t like waiting for results. The database should be capable of optimizing load and query performance so that analytics is both timely and relevant.

    3. Embeds machine learning: Machine learning algorithms are critical for predictive analytics. A platform that can embed ML and iteratively learn from new data can make a big difference and add a whole new level of intelligence.

    4. Features native integrations: The more time developers spend figuring out tools and integrations, the less time is spent innovating. The platform must ideally have out-of-the-box integrations with popular applications, data sources (on-premises, cloud, multi-cloud or hybrid), ETL, and visualization tools (e.g., Microstrategy, Looker, Power BI, Qlik, Tableau). It should be flexible enough to support user-defined functions for things that don’t have native integration.

    5. Encourages self-service: The database should make it easy for data scientists to leverage tools and programming languages they already know such as R, Java, C/C++, Python and SQL. Data scientists will also appreciate a platform that has built-in functions for a wide range of analytics (such as geospatial, time-series, event-series, real-time, machine learning, text analytics, pattern matching and regression) and delivers the tools to spot trends, uncover anomalies and anticipate the unexpected.

    6. Supports highly advanced analytics: Many platforms can analyze single tables or do a simple look-up but few have the power to analyze dozens of tables, hundreds of data types, dimensions and attributes from numerous sources over several years of history.

    7. Unified: The database should be able to streamline everything right from data ingestion to exploration to production until delivery. Users should be able to analyze data in its place to always have a single source of truth.

    It’s not hard to visualize that the most disruptive companies of the future will be the ones that harness data as a core asset and embed analytics in their products and services. Embedding analytics requires the right database foundation; to get to the right foundation, IT teams need to cut through the noise and study the fine print. With the proper database, the future is yours to create.

  • The Curious Connection Between Cloud Repatriation and SRE Ops

    The Curious Connection Between Cloud Repatriation and SRE Ops

    I have a fondness for philosophy. I’m about three classes short of a degree, and every few years I tell myself one day I’ll finish it. Thus, I am very familiar with what is known in statistics—and logic—as a post hoc fallacy, from which we get the saying “correlation is not causation.” This is the logical error of assuming that if event Y followed event X, event Y must have been caused by event X. The most famous callout of this fallacy came from Bobby Henderson, who illustrated the absurdity of assuming causation from correlation with his chart demonstrating that global warming was caused by the declining number of pirates in the world. Yeah, that doesn’t make sense, but neither do a lot of charts from which people draw causality. Just because two data points are mapped against each other does not mean one caused the other. In many cases, it doesn’t even make logical sense to correlate the two. After all, pirates and global warming? No one actually takes that seriously. But it’s an important point to make as we dive into the question of the relationship between site reliability engineering (SRE) operations and cloud repatriation.

    To be clear, I’m not suggesting that adopting SRE practices causes cloud repatriation. But I am suggesting that there is a close and meaningful relationship between the two. The fact that Google—a cloud provider—created SRE as a practice is not a mistake. The model, mindset and skillsets associated with SRE are integral to successfully operating cloud infrastructure and services.

    Cloud Repatriation Is Real

    Public cloud repatriation itself is a somewhat taboo topic in certain circles. Consider the controversy raised by Andreessen Horowitz when it published “The Cost of Cloud, a Trillion Dollar Paradox” and suggested companies were repatriating from the cloud and realizing significant cost savings as a result. Some would have you believe it’s not happening, but there’s enough data and anecdotal evidence to indicate that yes, it is.

    F5’s 2021 state of application strategy report asked the market about public cloud repatriation. A mere 13% had repatriated apps and another 14% were planning to. One year later, that combined total rose 40 percentage points to 37% and 30%, respectively. This is not an anomaly, as there are multiple credible analyst firms reporting similar results. Interestingly, the rate of repatriation is not globally universal. APCJ and LATAM are both far less likely to repatriate than EMEA and NA.

    I maintain that companies are repatriating apps from the public cloud—and the question isn’t ‘Are they?’ but rather ‘How many workloads are they pulling out—and where are they going?’ That’s a question for the State of Application Strategy 2023 research.

    For now, let’s consider a possible enabler of repatriation—SRE operations. Because even if the increasing cost of cloud is a driver of the desire to repatriate if you don’t have the skills to operate as efficiently elsewhere—and thus benefit from lower cost—then why would you repatriate?

    And we posit that it is SRE operational practices and skills that enable companies to repatriate and maintain the efficiency and cost savings needed to justify the decision—whether they’re moving those workloads to another public cloud, on-premises or to the edge.

    Digging Into the Data

    On the surface, there is a strong correlation between the adoption and application of SRE practices with cloud repatriation that seems to indicate that organizations with the ability to operate in a cloud-like manner, i.e., they’ve adopted SRE practices, effectively pick up their toys (apps) and go home (on-premises or elsewhere) because they can.

    Cut another way, only 4% of organizations that have not adopted SRE practices have repatriated apps from the public cloud. A whopping 73% of those who have adopted SRE practices have also repatriated apps.

    Of course, adopting practices does not necessarily mean applying practices. So, we looked at how organizations are actually operating applications, systems, and infrastructure. Specifically, we looked at the percentage of their operations that use SRE practices. Perhaps unsurprisingly, that generated similar results.

    Of those who operate 0% of their apps, systems, and infrastructure using SRE practices, 81% are not repatriating. Conversely, of those who use SRE practices for 76%–99% of apps, systems and infrastructure operations, 54% have repatriated. The point at which repatriation appears to begin picking up steam is when organizations surpass using SRE practices to operate more than one-quarter (25%) of their apps, systems and infrastructure.

    Remember I noted that APCJ and LATAM were far less likely to repatriate? They’re also far less likely to be leveraging SRE practices to operate their apps, systems, and infrastructure. In fact, over one-quarter (26%) of respondents in LATAM and APCJ (29%) were operating ZERO percent of apps, systems and infrastructure using SRE practices. In EMEA? That’s only 5%. And in NA, even lower at 2%.

    Meaningful Relationship or Curious Coincidence?

    There appears to be an inarguable correlation between organizations embracing SRE as an operational practice and public cloud repatriation rates. But is it a meaningful relationship or merely a curious coincidence?

    I’m going to argue that it’s a meaningful relationship.

    The practices and skillsets associated with SRE are wholly suited to operating a cloudy environment—at scale. As I said before, it’s no mistake that it was Google who created SRE and has literally written the book on it. The value of cloud is in its operational model, which can dramatically lower the cost per transaction, whether measured by HTTP exchanges or customer sessions. That enables a cost-efficient scale of applications and digital services.

    The use of automation and practices that tend to focus on meaningful incidents rather than non-disruptive occurrences provides the cost-efficient scale of the people (and thus their expertise) who are tasked with maintaining a high level of availability and performance.

    The adoption and use of SRE practices enable organizations to efficiently scale operations whether in the public cloud or on-premises or at the edge. And what the data tells us is that organizations appear to be using the capability to do just that.

  • A New Development Model: Bringing SaaS Apps to Data

    A New Development Model: Bringing SaaS Apps to Data

    The last few decades have seen multiple major shifts in how applications are built and deployed, and now we’re seeing the start of a new shift that opens up enormous opportunities for all kinds of developers. Instead of bringing data to applications, developers are bringing their applications to the data.

    Let’s back up. In the 1990s, as data grew too large for local machines, applications moved with the data to data centers—a model more like the mainframes that preceded them. Then, in the 2010s, those same applications migrated from data centers to the cloud, bringing the data with them again. Now we’re seeing a third major shift, where customers take back control of their data and centralize it in the cloud. And developers are picking up on this trend, building their applications to not control their own data, but instead to connect to the customers’ data, letting them have and control a single copy.

    In a traditional SaaS application, the SaaS vendor stores and processes data in their own platform, even if that platform is running on one of the major public clouds. A traditional CRM (customer relationship management) tool, for example, will use its own data platform to store all of the data that all of its customers are managing with the tool. When a customer wants access to their data, often to join with other data sets, they either have to export the data (via APIs or CSVs) or upload the other data set into the tool and use whatever analytics capabilities the tool has built. This is painful for customers and also painful for the developer who has to support use cases that are further afield from traditional relationship management.

    In this new world, the customer runs and the applications all connect to the same data platform, storing and processing their data there. This means that joining two data sets is a single query away, with no movement of the data required. It also means that the customer owns and controls the data, making security and governance hurdles much easier for developers to overcome. And, because the customer pays for the data platform, the developer doesn’t have to mark up the costs and is able to command significantly higher margins.

    While it’s early, this shift is already underway. Fast-growing startups like Supergrain, Anecdotes.ai and Rudderstack, to name a few examples, are taking full advantage of this shift and leading investors have picked up on it as well.

    Why Now? Access to the Best Data Platforms Available

    There are two key trends that are leading to the emergence of these connected applications: The wide adoption of cloud data platforms and the prioritization of control over data by customers.

    In the last ten years, a new class of highly scalable and cost-efficient data platforms has emerged, allowing enterprises of all sizes to adopt technology that was previously only available to the most sophisticated vendors for use inside their own products. Previously, SaaS vendors had to roll their own complex data platforms that were very specific to their use case and charge high prices to cover the cost of running those platforms. Today, we’re not only seeing SaaS vendors migrate to cloud data platforms but also enterprises adopt the platforms themselves. This means that, for the first time, every enterprise has access to the best data platforms available.

    We’re also seeing a huge change in how enterprises think about and secure their data. Partly because of regulations, and partly because of publicity around data breaches, large and small companies are paying significantly more attention to how secure, how controlled, and how well governed their data is. When your data is spread across dozens or more different SaaS applications, it’s nearly impossible to guarantee its security and access controls. Centralizing that data into a single platform, and connecting your applications to it, allows for a single point of control and security for every piece of data.

    What’s In It For Developers?

    It’s clear that connected applications are a big win for customers: A centralized source of truth with strong security and control. But this model has enormous benefits for application vendors as well.

    First, it changes the go-to-market dynamics by letting you tap into work that customers have already done to centralize their data. According to George Xing, co-founder and CEO of customer engagement platform Supergrain, CEPs built with traditional SaaS models suffer from several drawbacks including long implementation times, poor reporting, and higher costs. The connected app model makes Supergrain more competitive, he said.

    “Instead of moving your customer data to a third-party data platform, we bring the marketing software to your data warehouse, unifying source-of-truth customer data and marketing product experiences into one cohesive system,” Xing said.

    It also accelerates time to market. Developers no longer need to build and maintain a data platform for their customers and can focus instead on their core product differentiators. They also get to leave responsibility for data security and compliance to the customer and their data platform.

    With the customer managing and paying for storage and compute directly, vendors can significantly increase their margins, charging for the value they provide, rather than trying to mark up the compute they’re consuming.

    What’s Next for SaaS?

    The incredible growth in the amount and types of data available to companies is only accelerating this shift in how applications store and operate on their customers’ data. We’re going to see more and more SaaS vendors support connected application deployment models, but it won’t replace vendor-managed models. Instead, there will be opportunities for both models to flourish depending on the customer and specific application. We expect that many vendors will offer both models, just as we’ll see others go even further, bringing their entire application—including its UI, logic, authentication, etc.—fully into the customers’ data platform.

    The opportunity for companies that take advantage of these new models is enormous, and we’re in the very early days. But the applications that take full advantage of this shift will be the winners of the next generation of data apps.

  • Survey Shows Steady DevSecOps Progress

    Survey Shows Steady DevSecOps Progress

    A survey of 200 DevOps and IT/information security professionals published this week by Mezmo, a provider of an observability platform, conducted in collaboration with the market research firm Enterprise Strategy Group (ESG), finds only 22% report their organization has a formal DevSecOps strategy to integrates security into their software development lifecycle (SDLC) processes.

    Among those that have adopted best DevSecOps practices, however, 60% saw an acceleration in incident detection, while 53% saw a reduction of incidents in production environments.

    Overall, the survey finds 62% of respondents report their organization is actively evaluating use cases for, or has plans to implement DevSecOps.

    Mezmo CTO Rob Fry said the primary challenge organizations face when it comes to DevSecOps remains cultural. In fact, 99% of survey respondents said establishing a culture of collaboration between development and cybersecurity teams is either critical (51%) or important (48%). Building cross-functional DevOps teams that have the required level of security expertise, however, is a major hurdle when there is a general shortage of cybersecurity expertise, noted Fry.

    The survey finds that on average it takes 17.5 person hours to triage and understand security incidents, an amount the survey finds that 82% of respondents desire to reduce.

    However, organizations are also struggling with the sheer volume of data collected by DevOps teams. More than half the respondents (52%) said the variety of data to be captured, analyzed and be acted on throughout the software development lifecycle (SDLC) hinders security efficacy/efficiency. A total of 43% said the scale of data to be captured, analyzed and acted on throughout the SDLC also hinders security efficacy/efficiency.

    Organizations surveyed capture several (54%) or even hundreds (32%) of terabytes per month, with 6% capturing a petabyte or more per month. Well over two-thirds (69%) do not capture certain data sources because of the high cost of storage/retention, which Fry noted is going to be problematic if there is an incident. Log data (35%), followed by metrics (29%) and traces (22%) account for most of that data, the survey finds.

    In all, 84% of respondents said getting the right data and tools to developers is key to achieving success. A full 91% of respondents are using multiple tools to get the most value out of their data. Two-thirds (66%) have dedicated logging analysis tools, while 59% have a security information event management (SIEM) platform. Cloud to two-thirds 64% have a data lake hosted in the cloud, while 55% have one in an on-premises IT environment.

    A full 87% are using open source tools as part or all of their observability stack because they are more customizable. However, 84% noted that over time it will become challenging to manage, adapt and scale with these solutions. Nearly all survey respondents (98%) said they will likely investigate a managed observability solution over the next 12 months.

    The application security issues that plague organizations today have been around for decades, so it’s not likely they will be resolved in a few short weeks. On the plus side, at least, there’s now a lot more awareness of the root cause of the problem than ever.

  • A Guide to Sustainable Application Modernization

    A Guide to Sustainable Application Modernization

    Sustainable development is a blueprint for societal growth that is applied around the world to balance the needs of the present while optimizing for the future. Sustainability consists of three pillars — economic, environmental and social — that must work together to meet this goal. The complexities of digital transformation and application modernization can benefit from applying a sustainability lens to this challenge that not only includes the technical benefits of cloud computing but, more broadly, the three P’s of sustainability: Profit, people and planet. Application modernization will benefit the business in measurable ways, but can also affect unexpected positive cultural and green initiative benefits.

    Sustainable Application Modernization: What is it?

    Sustainability initiatives have always had to take a holistic approach that combines human, ecological and economic health and vitality. Sustainable application modernization is no different in that business leaders are dealing with finite resources (budget, staff and time) that must be applied now to align with longer-term priorities and their associated consequences. The main goal of application modernization is to transform an enterprise’s monolithic applications into modern architectures that can more fully take advantage of cloud-native services such as containers, microservices, serverless and a range of DevOps methodologies. These technical capabilities lead to a number of high-value cloud benefits including scalability, elasticity and engineering agility. Too many modernization strategies and business plans stop there and this is a mistake. By undervaluing the sustainability benefits that include economic, environmental and people upsides, technology business leaders are not accounting for some of the most important benefits of modernization.

    Breaking down the modernization problem along the three pillars of sustainability helps define sustainable application modernization:

    • Economic sustainability: The business benefits of modernization must account for releasing more features and fixes more frequently, less downtime, lower technical debt and higher rates of innovation.

    • Environmental sustainability: A modern architecture is more resource efficient, scales at a much finer level than a monolith, and is able to take advantage of cloud data centers that offer the “greater efficiency of aggregated compute resources.” These cloud-native architectures enable more intelligent application orchestration and resource allocation usage based on demand, need, cost, seasonality and performance.

    • Social sustainability: Besides the ongoing challenge of recruiting and retaining talent for maintaining a monolith, the resulting development team structure is a direct reflection of the architecture itself and this is most often monolithic, too. A more domain-centric approach using modern technologies creates leaner, more focused team topologies that can also operate effectively in remote work structures.

    The broader benefits and impacts of effective application modernization align directly to metrics associated with elite performers in DevOps research reports that track four key performance categories: Frequency of application deployments, application feature lead time, change failure rates and time to recovery. This long-standing and annual research report from DevOps research and assessment (DORA) metrics also now reaches beyond the business and technical dimensions into the performance, culture, people, burnout and team impacts of these modern best practices.

    Application Modernization is Not Just an IT Problem

    Most application modernization projects never fully get off the ground because technology leaders look at modernization as just an IT problem. The business cases for these projects focus on technical benefits or simple cloud migration and lift-and-shift infrastructure cost improvements. A sustainable application development methodology opens up this calculus to the three dimensions of sustainability: Economic, environmental and social.

    Much as everyone from developing countries to the richest nations in the world view sustainability initiatives along these lines—balancing economic growth, environmental health and the vitality of their citizenship—application modernization projects can be a catalyst to major improvements that drive profit, maximize efficiency while improving company culture, retention and team growth.

    Sustainable Application Modernization: Three Steps to Get Started

    The enthusiasm for application modernization is high, but organizational precendent and a history of failed efforts stop most projects before they get started. These three steps can set you on a sustainable path going forward.

    • Start with a data-driven plan: A monolithic application architecture requires a quantitative understanding of the technical debt you are carrying at an architectural level; the move to a cloud-native architecture unlocks the business, environmental, and cultural benefits discussed above. Thus, it’s critical to assess your application estate from an architectural perspective to calculate the complexity and risk associated with your applications, prioritize which applications to modernize and extract ROI benefits related to the modernization effort.
    • Build a broad coalition across the pillars: A sustainable application modernization plan works across three pillars that span different roles, functions, and initiatives in an organization. This strategy creates both an opportunity and a need to recruit outside the technology team to the business and product owners that can validate and then promote the economic upside. Next, pull in green initiatives and corporate sustainability representation that can quantify and promote the environmental benefits further. Finally, HR, training and project management experts will understand and align with the people-side of modernization to create more agile, happier and efficient teams.
    • Create a continuous modernization culture: This approach is designed to maximize short-term benefits while optimizing for the future. It’s critical to look at this as an ongoing, continuous best practice that becomes ingrained at both a technical and cultural level. Most modernization strategies have resulted in one-off projects that, even if completed, may quickly devolve back to bloated monoliths due to a lack of cultural and organizational commitment to take these practices forward and iterate across all applications in the future.

    Sustainable application modernization methodologies can change the course of application development inside an organization and provide a catalyst toward a thriving future that pushes the technology forward and can maximize the value to the business, environment and company culture.

  • Clinical Trials: Programming for Scale, Speed and Success

    Clinical Trials: Programming for Scale, Speed and Success

    According to the Journal of the American Medical Association (JAMA), the median cost of bringing a new drug to market is $985 million and the average cost is $1.3 billion. However, those in health care hoping to harness promising new treatments, and patients facing steep prescription costs, see this as progress. After all, previous studies estimated the average price for developing a drug to be as high as $2.8 billion, a cost usually recouped at the expense of consumers. While savings are being realized, trials still come with quite a price tag—but the reasons why are understandable. First, clinical trials are complex and, simply put, time is money. The final stage alone, the Phase 3 clinical trial, can involve tens of thousands of participants, each with hundreds or thousands of data points that need to be tracked over a number of years. This can produce hundreds of millions of data points that must be protected, secured and blinded, and all of which must be analyzed by numerous applications.

    Many cite the fast development of a COVID-19 vaccine as proof that the process can be expedited. While the vetting of these vaccines was thorough, the processes benefited from unprecedented resources, industry and government cooperation and global pressure to quickly find a way to slow a highly infectious disease. Even so, one can’t help but wonder why clinical trials typically take six to seven years? Haven’t we learned how to speed up trials without jeopardizing health or draining budgets?

    The short answer is yes—there is a way to reduce the time and resources needed for clinical trials. Pharmaceutical companies use dozens of software platforms from an array of vendors in their trials. As a result, crucial data gets trapped in unconnected silos, adding complexity and stalling progress. However, by unifying apps and data stores in one integrated system, trial efficiency can be raised and time-to-completion reduced by over 20%.

    Raising a Platform

    Deloitte noted that COVID-19 revealed health care’s need for structural and technology transformation, adding that its future will be “enabled by radically interoperable data, artificial intelligence (AI) and open, secure platforms.” The same holds true for clinical trial data software platforms; however, creating a unified system is an enormous undertaking for software developers and software engineers. Most importantly, any unified system must be able to provide the following four capabilities.

    ● Security: Data in platforms must not only be protected from external threats such as hackers and ransomware but it must also be blinded. The ideal is a double-blind clinical trial in which neither participants nor their physicians know who received the actual drug and who received the placebo.
    ● Modeling: A platform must be capable of extending a data model for each clinical trial to prevent the need to rebuild every time. The models, and the platform overall, must be flexible but still capable of managing hundreds of millions of records across multiple trials.
    ● Extensibility: Over the course of years, data collected—even in a single trial—may change. The ability to extend a scheme on the fly to fit evolving needs can save an enormous amount of time versus building a platform and transferring data. This is especially true when you have millions upon millions of data points.
    ● Ease: A platform must be easy to modify by non-programmers. If a trial designer can implement rules without having to bring in a programmer, the trial will move ahead much more quickly.

    Key to being able to raise an effective clinical trial platform are software engineers and developers. According to the Bureau of Labor Statistics, software developer jobs alone will grow more than 22% by 2029—faster than all other occupations combined. While this is good news for software pros, demand will create a shortage exceeding 1.2 million by 2026, which isn’t so great for the pharmaceutical companies that need talent.

    Then again, there are purpose-built platforms available that provide data, software, services and an ecosystem of partners to support functions from research and development through commercial availability. They’re designed to bring such products to market faster and more efficiently while maintaining complete compliance. The platforms can not only be comprehensive, but they can also mitigate reliance on developers and engineers, with regular provider updates future-proofing operations and decreasing investments.

    Programming for Success

    First and foremost, pharmaceutical companies are achieving faster trial goals by leveraging the cloud and DevOps. In doing so, all data can reside in a central repository that’s accessible from anywhere. There’s no hardware requiring time-consuming manual configuration. What’s more, modern cloud infrastructures are resilient, secure and robust. And all of this is largely being made possible by software-as-a-service (SaaS) platforms.

    Despite this progress, the need to technologically bolster trials continues. AI is now being applied to auto-classify documentation. Naturally, these types of developments require the implementation of programmability and extensibility at scale. The best minds in software engineering have made considerable gains. It’s now up to pharma to ensure their trials and operations continue to be programmed for success.

  • How Visibility Helps Devs Focus on Innovation

    How Visibility Helps Devs Focus on Innovation

    As software engineering leaders adjust to a future defined by remote and hybrid work models, they are shifting priorities in the organization of work to compensate for the lack of in-person collaboration. According to a recent study conducted by Jellyfish, the top priority for engineering leaders is ensuring that each member of their team is focused on the highest-priority work for as much time as possible. To achieve this goal, these leaders are using real-time visibility and data-driven decision-making to guide more effective management.

    Earlier this year, Jellyfish analyzed the work of more than 23,000 engineers and surveyed hundreds of leaders to better understand the current state of engineering management. The research revealed that engineering leaders are shifting resources away from reactive work—such as responding to customer-reported bugs—to the type of work that directly impacts top-line revenue. Additionally, engineering teams are emphasizing growth and innovation, developing the type of infrastructure that will enable their teams and their software to scale long-term.

    Prioritizing Visibility

    To shift from reactive to proactive work, engineering leaders must first have visibility into what their teams are doing. How can you allocate your time and resources more effectively if you don’t know how they’re being allocated today? Before work can be prioritized effectively, leaders must understand how much bandwidth their teams have to take on new projects. Additionally, they need data-backed insights in order to align the priorities of the engineering team with those of the company at large.

    Visibility is also key to avoiding work that wasn’t part of the plan. According to the study, teams spent 22% of their time on unplanned work in 2021, a 15% increase year-over-year. This trend indicates that engineering leaders may not realize when certain types of work are taking attention away from mission-critical initiatives. When asked to name their biggest challenge heading into 2022, 45% of leaders (by far the largest group) selected “making sure everyone is focused on the highest priority work.”

    To gain visibility into their operations, some engineering teams are doubling down on data aggregation and analysis. According to the report, while all organizations tend to devote similar amounts of time to “keeping the lights on” and customer support activities, those that invest in a more robust data organization are able to spend 29% more time on innovation and 48% less time on unplanned work.

    Building Through Data and Metrics

    The report also found that engineering leaders who’ve embraced data-driven decision-making are seeing more productive, healthy and focused teams. These organizations tend to release code more quickly and foster higher collaboration than their peers. Conversely, organizations that don’t prioritize data-driven decision-making are more likely to experience burnout, decreased productivity and less focus on innovative work.

     

    Here again, the difference between embracing data and ignoring data comes down to technology. Aggregating data and metrics manually can be tedious and challenging. However, emerging tools are able to automatically collect, measure and analyze data to offer insights on what teams are prioritizing. As a result, leaders can more easily align engineering work with business priorities, ensure more predictable development cycles, increase financial responsibility and continuously improve operations.

    For every engineering leader, nailing this approach to data will be essential for maintaining efficiency as teams expand. More than 79% of engineering leaders expect to grow the size of their teams in 2022, and respondents named employee onboarding ramp time, virtual onboarding and hiring among the most significant challenges they face.

    Despite the numerous challenges engineering teams have endured in the past two years, the industry has adapted and become more collaborative as remote and hybrid work models multiply. For leaders who seek to increase data-driven decision-making, new tools and approaches will result in more intentional engineering work and improved visibility and operations.