devopsy.kinsta.cloud

Tag: data analysis

  • Consider Performance, Growth & Budget When Buying Data Analytics

    Consider Performance, Growth & Budget When Buying Data Analytics

    So, your business needs to invest in data analytics technology to improve efficiencies, competitive advantage or business outcomes. There are a ton of things to consider, but underlying each decision are three driving factors:

    • Performance – The need to meet service level agreements.
    • Growth – The need to accommodate success, as well as deal with inevitable industry changes.
    • Budget – The need to do both of the above while keeping costs low enough to not erode the profit gain from analytics.

    These three forces underpin nearly every technology selection decision, but what frequently isn’t noticed is how much these three drivers interact.

    1. Concurrency and Growth

    Test both for current levels of concurrency and the level you hope to achieve in the next few years as the organization grows.

    When evaluating benchmarks between vendors, consider that benchmarks are designed to make that company’s software look good. For example, if they are good at concurrency, you will see various levels of concurrency represented in the benchmark. If the benchmark shows only one workload at a time, or maybe ten, you can safely assume that concurrency is probably not something that technology is good at. Similarly, if the benchmark is done at 10 TB scale, the vendor probably isn’t that good at high-scale analytics.

    When bringing vendors in for a proof-of-concept, one mistake I’ve seen (more times than I can count) is testing the capabilities of the software with only one user at a time. People do not line up politely to use analytics one at a time.

    In addition to current levels of concurrency, over time, hopefully, the organization will grow, and the number of both internal analytics users and external customers will increase. Testing for the number of users you have now may not uncover problems that will invariably reveal themselves when a lot of people hit the system at once. The last thing your organization needs is to turn away customers because they’re overloading your systems.

    2. Performance & Price

    Evaluate price/performance, not performance alone.

    Everyone understands the importance of performance. Not one user has ever gone to a data architect or engineer and requested for their analytics to “please execute more slowly.” But performance isn’t simply how fast the analytical software can process queries, how fast the data pipelines can move data or how fast the response is between data input and reaction output. There’s also a monetary aspect of performance to bear in mind.

    In a recent benchmark, three data management and analytics stacks were compared with 60 simulated concurrent users/workloads. The difference in queries per hour on 250 TB of data was minor, perhaps 16 queries per hour for the slowest and 23 queries per hour for the fastest.

    (Mcknight Benchmark – SaaS Data Analytics Platform Comparison – vendor names removed)

    It looks like performance is just not that big a consideration for this technology. But there is no such thing as an unlimited budget. Let’s look at how much it would cost to get this level of performance out of each stack.

    (Mcknight Benchmark – SaaS Data Analytics Platform Comparison – vendor names removed)

    Price-performance is a far more useful metric of value to the organization. How much the software can do is irrelevant if the budget limits how much the business can afford to do with it.

    3. Software Efficiency

    Consider how much efficiency and control you’re willing to trade for ease-of-use.

    When you analyze data in the cloud, you’re not just paying for software, you’re also paying for hardware, a.k.a. cloud compute instances. Bundling the two costs in one bill makes things easier. But it also means that cloud providers make more money the more instances, or compute nodes, you use. You trade ease-of-use in billing for a higher price. And they no longer have a financial incentive to provide efficient software.

    Similarly, to make a cloud-based analytics software scale up in performance and concurrency with zero effort on your part, the software has to use only defaults for all settings internally. It can scale only by throwing more compute at the problem; compute that you’re paying for every time you use it. You’re trading control of the software–tuning, scaling guidelines, shut down conditions, etc.–for ease-of-use, again for a higher price.

    Throwing more compute at a problem isn’t always the best solution. Recently, I saw a POC where the customer implementation included 278 EC2 nodes on Amazon cloud. The organization didn’t initiate the POC because of inefficiency. They were having performance problems, even with that much compute power. A competing product was able to run on only nine EC2 nodes with the same workload and better performance. The company was paying 25 times as much for the compute hardware because it was convenient.

    Be aware of the tradeoffs.

    4. Affordable Scalability

    Paying for only what you use can become impractical when you use more and more.

    The volume of data that is being created, consumed and captured is growing exponentially. By 2025, it is estimated that nearly 30% of data being generated will be real-time. Instead of pulling data in batches, data will be pulled in continuously. Queries will be responded to in seconds, not hours, or in milliseconds, not seconds.

    Going to the cloud can make tremendous sense when a company or a workload in that company is new. But studies show that as businesses scale and growth slows, the pressure the cloud puts on business margins eventually outweighs its benefits. This is why most enterprises today are considering repatriating some or all of their workloads off the cloud.

    5. Flexibility for Change

    Make certain your stack will continue to work even as conditions change, and you won’t have to rebuild from scratch to do the same thing in a different location.

    When you’re building a long-term architectural strategy, you want to ensure you cover as many bases as possible because, the fact is, you really don’t know what the future holds. Ten years ago, maybe you had a Hadoop on-premises strategy or a data warehouse appliance. Today, you may have a cloud-focused strategy. Tomorrow, maybe you’ll move some or all of those workloads back on-premises due to performance, security, costs, regulations or other reasons.

    This is why you want flexibility that doesn’t mandate a “cloud-only” or an “on-premises-only” or a “this-cloud-only” architecture and which doesn’t integrate with only the same vendors’ software. You want an architecture that can work equally well both on the cloud as well as on-premises, hybrid or multi-cloud, providing the flexibility of going where business needs are best met.

    Containerization offers possibly the highest level of flexibility in deployment, but it can also add to deployment complexity, so again, be aware of what you’re trading. If the software you choose is platform-agnostic—not tied to any particular location, ecosystem or deployment model—then that will make future choices a lot more flexible.

    Regardless of what analytics you choose, what you decide to build or what use cases you have to meet, striving for balance in performance, growth and budget will create a strong data architecture, now and for the future:

      1. The analytics architecture is high-performance: it should meet SLAs, get things done on time and meet or exceed expectations and requirements.
      2. The analytics architecture is resilient to growth and change; even if there is rapid growth or sudden or extreme changes, it’s future-ready.
      3. The analytics architecture should control costs and live within an allocated budget.

    Keep these guidelines in mind when choosing an analytics provider. The decisions you make today will have direct ramifications on the growth, profitability and success of the business tomorrow.

  • New-Age Tech and Culture Driving Data Democratization

    New-Age Tech and Culture Driving Data Democratization

    Data democratization has become critical for organizations to leverage data’s true value and fully realize its benefits. Making data available across the organization helps companies to serve customers better and make data-driven decisions that align with their goals and objectives. It essentially means enabling access to a company’s data resources to employees, bound by reasonable limitations on legal confidentiality and security.

    As a means to remove data silos in organizations, the indispensability of data democratization is clear. It allows more widespread access to and use of data within an organization. This can lead to more informed decision-making, increased collaboration and innovation and more efficient use of resources.

    Breaking down data silos can help reduce the risk of data breaches and improve data security. While the jury may be out on whether data democratization is beneficial in preventing security breaches, silos between networks or security systems make it difficult to identify large-scale attacks as they tend to represent disjointed security protocols. There is less likelihood of robust security protocols between non-communicating teams.

    New-Age Tech as an Enabler of Data Democracy

    New-age technology such as cloud computing, blockchain, and advanced analytics are aiding data democratization. For example, 5G enables more significant volumes of data to be moved faster and more securely in the form of big data. Blockchain provides higher-quality data and improves the operation of cyber and physical systems. In addition, IoT allows for exchanging data between devices, where processes can be established between devices without intermediaries.

    A hybrid cloud adoption survey conducted in March 2022 of IT professionals primarily based in North America and Europe revealed 93% of the IT industry would adopt a hybrid of cloud and on-premises solutions or migrate fully to the cloud within five years, using it to store and manage data. Gartner estimated that 85% of enterprises would embrace a cloud-first principle by 2025 and that it will be essential to execute digital strategies in business.

    This suggests that new-age technologies play a significant role in breaking down data silos and enabling organization-wide access to data. By leveraging these technologies, organizations can easily store, process and share data, leading to better decision-making, increased collaboration and improved business outcomes.

    Data creation, development, and utilization directly relate to Industry 4.0 applications. In practice, data democratization is seamlessly transferable to Industry 4.0 applications. The question arises: Do industrial companies have the capabilities today to apply the concept of data democratization? And to what extent?

    As new-age tech drives the emergence of new business models, companies that will lead a cultural shift as well as a technology evolution will be successful. This is possible via a business-centric data strategy, along with the democratization of analytical tools and platforms to enable stakeholders to help build better insights into their business processes.

    Culture as a Prerequisite for Data Democracy

    A Google Cloud and Harvard Business Review survey of industry leaders revealed 97% believe organization-wide access to data and analytics is critical to the success of their business. However, only 60% of respondents believed that their organizations effectively distribute that access today.

    What are the prerequisites for data democracy? First, companies should inculcate a culture change to create a pull from inside the organization rather than first developing the technology. Second, decision-making in companies is based on experience and intuition than on data; they should compulsorily begin to use data as a baseline for business decisions. Third, leaders should encourage a data-driven culture to grow from within.

    Formalizing and sharing knowledge can remove roadblocks to the democratization of data. Therefore, one form of action is to set specific incentives to foster a mindset of knowledge sharing. For instance, existing practices to incentivize participation in the suggestion or lessons-learned programs can be used as a reference.

    Many companies have procedures to involve employees in the continuous improvement process. But while employees can use the existing IT systems to perform their day-to-day work, they need more capabilities to cover their individual information needs. Thus, as a way forward, capability building within is critical. Another strategy is decentralizing decision-making; if employees can gain insights independently, they should be able to act on these insights independently.

    Organizational culture is vital to data democratization, as it determines the attitudes, behaviors and values that guide how data is shared and used. On the other hand, a culture that values secrecy or hierarchical control over data can lead to resistance to data democratization efforts and perpetuate data silos. A culture that lacks trust and transparency can also lead to reluctance to share data, which can impede collaboration and hinder data-driven decision-making.

    Data Democratization – The Way Ahead

    Organizations need to assess and work on their tech stack and internal culture to promote a data-friendly culture supporting data democratization. To foster a data-friendly culture, organizations should encourage open communication and collaboration among employees and provide training and resources for employees to access, understand, and analyze data.

    Open communication, collaboration, innovation and trust are essential attributes for breaking data silos. A corporate culture that values data-driven decision-making will lead to more widespread access to and use of data and support the implementation of new-age technologies. Finally, there is a need to establish clear data governance policies and procedures that support data democratization and build a continuous improvement process by encouraging experimentation and learning from failures.

    Image Source: Joshua Sortino, Unsplash

  • Implementing Data-Driven DevSecOps

    Implementing Data-Driven DevSecOps

    Right now, the way DevSecOps is typically implemented doesn’t fit with the rapid and agile DevOps CI/CD pipeline at all. It’s like applying 19th-century firefighting methods to a modern forest fire.

    Back then, firefighters employed a “bucket brigade,” where they would form a queue and pass buckets from one hand to another to put out a blaze. No doubt, it’s a lot of work, but much of the effort is wasted. Water inevitably spills out of the buckets as they are handed from one person to the next and by the time the bucket is emptied into the fire, half the water is gone. And not only is so much effort wasted, it’s far too slow and ineffectual to combat the kind of wildfires we face today.

    Likewise, the largely manual methods of current DevSecOps initiatives are ineffectual in fighting the fires of digital threats and cyberattacks modern mobile apps face. Like a modern megafire, these threats and attacks are growing and changing every second, seeking new vectors to spark new attacks elsewhere. Traditional DevSecOps tools like code scanning and penetration testing identify vulnerabilities and then the security teams start the manual “bucket brigade” to add as much protection as they can against them before the app has to be released. But neither the threats nor the CI/CD process has paused. New features have been added that have created new vulnerabilities, and the threat landscape has evolved. It’s inefficient because pentesting can’t provide the kind of real-time data about attacks and threats that developers need to provide protection against current threats. The result is the release of vulnerable mobile apps. 

    A Data-Driven Process

    In business, the C-suite is working hard to transform their companies into data-driven organizations where decisions are made not on the basis of gut feelings or expert opinion, but rather on analysis of hard data. That same approach needs to be brought to security implementation for mobile applications.

    Additionally, the implementation of security into a mobile app needs to be automated. With so much of the CI/CD process already automated, security cannot lag behind.

    These two elements combined create data-driven DevSecOps. In this method, the development and security teams have a system of record that provides real-time cyberattack and cyberthreat information about apps in the field, which drives decisions the team makes about the protections that are most urgent and that must be included in the next build. 

    It’s now well within the realm of possibility for mobile app developers to gather near-real-time information on exactly the kinds of threats and vulnerabilities that their mobile apps are facing in the field. When combined with location, network and other kinds of data, developers can gain a granular understanding not only of the most common threats their apps experience, but also which threats are most prevalent in specific geographic regions. They can also proactively identify the rapidly mounting threats that will become an enormous problem in the near future.

    With this data in hand, development teams can make informed decisions about which protections they should prioritize in the next build to make the best use of their time and resources to provide maximum protection to their end users. 

    The Need for Automation

    But having data isn’t enough. Development teams need the ability to act quickly on the insights provided, and manual implementation methods are too slow and cumbersome to keep up with the rapidly changing threat landscape for mobile apps. Once the decision is made, a system must also exist to automate the incorporation of mobile app security, anti-fraud and anti-cheat protections from within the CI/CD pipeline, so that security implementation runs as smoothly as feature creation.

    The advantages of data-driven DevSecOps are many. First, publishers, developers and security teams can understand exactly what cybercriminals are doing against their own apps with real users. With this data and the wealth of other data that organizations can collect from their apps, they can determine which threats are the most pressing, which are rapidly emerging and even how they are distributed among different geographic regions—all of which is extremely valuable information when deciding which protections to build into the next release.

    Finally, once the app is released, teams can use the data they receive about threats and attacks to prove the effectiveness and value of each protection that was included. It’s important information not only for continuous improvement, but also to justify the value of data-driven DevSecOps to management and the C-suite.

    With data-driven DevSecOps, organizations can provide data, transparency and visibility around security to all stakeholders in the CI/CD process—all while making the incorporation of security far more efficient and effective, along with the real-time data to prove it.

  • Filter the Firehose

    Filter the Firehose

    We are tired. Information overload is a problem in the modern world. We hear instantly about events we never would have known about otherwise, or that we would have learned about months after the fact. Today, moments after an event, we have thousands of “professionals” analyzing it for us, a millions-strong army of amateurs telling us they know everything about it and a legion of bots telling us what to think about the event. No matter what the event is. Often “the event” isn’t even newsworthy, and yet the process feeds on itself.

    We in IT have it worse. We have a flood of security and application events constantly flooding in and so much data that we call it a ‘data lake.’ More like a data ocean at some organizations. We’re tired. They call it by various names—alert fatigue, data overload, event flood, “the firehose.”

    No matter what it’s called, it’s a big problem. And if it is not effectively managed, we will make mistakes. Particularly in security—but also across IT—we cannot afford to allow the flood to distract us or burn us out.

    Look into AI. The amount of data, alerts, information, metrics, logs, traces, etc. that we are accumulating—and the rate at which we are doing so—is growing faster than humans can reasonably keep up with—even if there were no staffing constraints. Don’t hesitate any longer; don’t assume you or a vendor NOC are managing this well enough with only eyeballs. AI solutions have matured enough to reliably sift through at least the top layer(s) of noise, reducing the burden without increasing risk. In fact, given the increasing possibility of human error as volumes increase, probably with less risk.

    Ask vendors about it; talk to your security vendors about their offerings. A huge number of security vendors have NOCs now that they would love to have you subscribe to. Many of them are highly automated and use AI. Data vendors are behind a bit in this regard (for a lot of reasons) but they are catching up. Not too long ago, the vast majority of work done to normalize disparate datasets was done via trial and error. AI and scripts written from massive experience are lightening that load, at least.

    So check it out. Find ways to make things manageable again. We all know things are slipping through the cracks as the volume of items reported and the number of locations they are reported from increases. Don’t wait until you have another emergency; look into the options and get something running.

    For those of you already using AI/ML, unless you implemented it very recently, you’ll probably want to validate what you’re using, at what level and how much the AI/ML solution is helping, and tweak or replace. This space is moving too fast for “set it and forget it” today. Maybe in a few years, we can.

    With the rare exception of something I use every day and think you might benefit from, I adhere to a policy of not recommending products here, so I can’t point you in the right direction other than to say that your vendors (data, SIEM and security) can offer solutions and/or suggestions. That, and “You shouldn’t wait” are my suggestions.

    And keep rocking it. Part of that data/log/alerting growth is all the apps and infrastructure you have helped put in and that are critical to the organization. So protect it with every tool available—and use that free time to work more miracles.

  • Opportunities and Challenges of Observability

    Opportunities and Challenges of Observability

    Observability platforms that unify metrics, logs and traces across applications and IT infrastructure are transforming how IT is managed. Most DevOps teams today are able to continuously monitor IT environments using tools that track a set of pre-defined metrics; observability platforms make it possible to aggregate data so that DevOps teams can launch queries that help them uncover the root cause of IT issues. DevOps teams leverage observability to allow them to understand what is happening within their applications more effectively and efficiently.

    Historically, applications were black boxes—meaning it was (and still is, in many cases) extremely difficult to know what is happening under the hood of the application.

    With the advent and subsequent growth of observability tools and techniques in the DevOps space, teams went from struggling to collect and understand the available data (which could vary from robust to almost non-existent) to easily aggregating data from numerous sources and being able to understand what that data means to being completely deluged with data—so much data it’s hard to keep up.

    Gather Data is No Longer Enough

    Josh Chessman, vice president of strategy and innovation at IT infrastructure monitoring specialist Netreo, explained that as applications become more and more complex, (e.g., running across multiple disparate environments with different technologies and in different locations) being able to collect data from these different environments is no longer enough.

    “Instead, DevOps engineers need to be able to collect the data and efficiently analyze it, too, to understand what is happening within the application,” he said. “Observability is the concept that drives this.”

    Chessman pointed out there are numerous observability issues DevOps teams face, but that they vary from organization to organization (and sometimes from team to team within an organization). 

    “Part of the problem is that many organizations simply lack tools to provide the data collection and analysis needed to move to an observability paradigm,” he said.

    This could be for numerous reasons, including budget (if you already have an existing tool, can you afford to acquire a new tool?), functionality (not all tools that claim to provide observability really do), implementation (this stuff is not easy and a poorly implemented tool could actually make things worse) and more.

    “One of the larger challenges I’ve seen is that organizations will go out and acquire an ‘observability’ tool and assume that is all they have to do,” Chessman said. “Unfortunately, that is not the case. Acquiring the tool gets you the technology, but the team must also be structured and ready to take advantage of the concepts and functionality provided by the tool.”

    GitLab’s senior director of product management Kenny Johnston explained that, on average, DevOps teams use five different observability tools which tend to inadequately connect developers to observability.

    In turn, developers struggle to respond to incidents swiftly while jumping across multiple tools and sources of truth and struggle to assign service level objectives (SLOs) and error budget definitions to the responsible product development teams.

    “To remedy this, we recommend adopting a unified software development solution that replaces the DIY ‘duct tape’ of products constraining developers and helps them deliver software more securely, at speed and scale,” he said. 

    Johnston said the most flexible and future-proof solutions today are open source tools like Prometheus and its ecosystem, as well as the OpenTelemetry toolset for more advanced use cases such as tracing.

    Prometheus has become very common in the developer space and, unlike other open source solutions or libraries, its format automatically emits metrics.

    “This allows for simple and future-proof interoperability, as one of the hardest things to change later down the line is code running,” he said. “Using open standards reduces the need for developers to rewrite and reintegrate the same type of code over and over again.”

    Observability Requires Understanding Your Goals

    From Chessman’s perspective, understanding what your goals are is critical for observability. 

    “Observability tools can do many things, but their primary function is to collect and analyze data,” he said. “While that is all well and good, if you do not define your KPIs from the beginning, all the collected and analyzed data is meaningless since you have no concept of what good and bad are.”

    Trying to define good and bad from the data collected is likely to result in failure because you are looking at where you are to figure out where you want to go.

    KPIs vary from application to application and, if you do not have them identified and well-defined before beginning down the observability path, it is that much harder, he said.

    Most tools (observability or otherwise) are capable of collecting hundreds if not thousands of metrics, but not all—not even most, in many circumstances—will be relevant to an organization. 

    “Instead of having to sift through the cruft to find the valuable data, it is better to start with identifying the valuable data and then figure out how to get there,” Chessman said. “If you don’t identify your KPIs in advance you could end up using the wrong KPIs for the wrong scenarios.”

    Johnston added that KPIs, like DevOps, are an iterative process and starting with KPIs helps developers create systems that can meet performance and availability expectations.

    “It’s important that DevOps teams continuously review their KPIs to allow for regular feedback on successes and failures,” he said.

    From his perspective, the best place to start for maintaining and improving observability is by storing observability definitions, instrumentation, SLOs, thresholds, incident response templates and dashboards as code.

    “This allows for rapid improvement via code changes by any team member, whether that’s a site reliability engineer (SRE), developer or platform team,” he said. 

     

  • LogDNA Survey Surfaces Observability ROI Concerns

    LogDNA Survey Surfaces Observability ROI Concerns

    A Harris Poll of 200 senior engineering professionals suggests it may be a while before most DevOps teams are able to achieve true observability in their IT environments.

    The survey was conducted on behalf of LogDNA, a provider of an observability platform, and found nearly three-quarters of respondents (74%) are struggling to achieve their observability goals despite investing hundreds of thousands of dollars. In fact, well over a third (38%) said they are already investing $300,000 annually on observability tools.

    Despite those investments, less than half of respondents are very satisfied with their ability to use log data. Key areas of frustration include the challenges of collaboration with colleagues spanning multiple teams (67%), the fact that the tools are not easy to use (66%) and the routing of security events (58%).

    Nevertheless, the bulk of survey respondents said they are optimistic about observability. A full 85% of participants said they believe true observability is possible.

    LogDNA CEO Tucker Callaway said the core issue that many organizations fail to recognize is that observability is really a data literacy and management challenge. Organizations are collecting logs, metrics and traces from a wide range of tools and platforms without thinking through optimal data pipeline construction. The result is an aggregation of data that has not been normalized to simplify surfacing actionable intelligence, he said. In fact, many organizations quickly discover that all they’ve really accomplished is increase their data storage costs, said Callaway.

    In the absence of that data normalization, Callaway noted it becomes exceedingly difficult for cross-functional teams to query all the data being collected within a repository.

    At its core, observability promised to make it easier to identify IT issues before they cause a disruption. Observability in one form or another has always been a core tenet of any DevOps best practice. Initially, DevOps teams focused on continuous monitoring as the most effective way to proactively manage application environments. However, it can still take days, sometimes weeks, to discover the root cause of an issue. Observability platforms make it possible to correlate events in a way that makes it easier for analytics tools to identify anomalous behavior that might be indicative of the root cause of an IT issue.

    In contrast, traditional IT monitoring focuses on predefined metrics to identify when a specific platform or application is performing within expectations. Observability platforms combine metrics, logs and traces—a specialized form of logging—to make it easier for IT professionals to interrogate data being generated by a wide range of DevOps tools and platforms.

    Of course, the assumption is that DevOps teams actually know what queries to launch to determine what the root cause of an IT issue might be. In the longer term, machine learning algorithms should make it easier to identify anomalies within IT environments. In the meantime, however, achieving observability still requires a fair amount of DevOps expertise that is not always readily available.

    Eventually, observability will become much more automated than it generally is today. As that transition occurs, IT teams should find the ROI of observability is steadily improving.

  • How AI is Changing DevOps for the Better

    How AI is Changing DevOps for the Better

    Managing a DevOps environment involves a high degree of complexity. The rapid proliferation of data has made it difficult for DevOps teams to effectively onboard, ingest and unlock data and implement information to make good business decisions and deliver delightful experiences. Imagine a team poring over troves of information to identify critical factors that triggered an event—they would end up investing long hours to interpret and analyze the information.

    The future of DevOps will be AI-driven. It’s fair to say that it’s challenging for human intelligence to harness large volumes of highly complex data. Consequently, AI-powered solutions will become the number one choice for integrating and analyzing data and transforming how teams develop, deliver, deploy and manage applications.

    By 2023, 40% of DevOps teams will leverage application and infrastructure monitoring solutions with built-in AI capabilities, Gartner noted. Currently, in times of disruption, Gartner predicts the AIOps market will grow at breakneck speed, between $300 million and $500 million per year.

    Let us explore how AI and DevOps are intertwined before we explore how AI is transforming DevOps.

    DevOps and AI are Intertwined

    AI and DevOps are intertwined. Integrating AI in a DevOps environment boosts the overall performance and efficiency of an organization.

    AI and ML-powered solutions enable DevOps teams to onboard, map and integrate large volumes of customer or partner data with accuracy and speed. That is to say, business users find it easier to drive processes like data onboarding, data mapping and data integration, thus improving ease of doing business and delivering improved customer experience (CX).

    Organizations that rely on AI-powered solutions no longer need to devote long hours to custom coding and EDI mapping to analyze large volumes of data. The time saved can be used to focus on more important, innovation-driven tasks.

    AI also boosts automation, allows users to address and resolve issues in both data and applications and facilitates better cooperation among teams.

    4 Ways AI is Transforming DevOps

    The efficacy of DevOps correlates with an upward shift in AI. It can empower even non-technical business users to create connections and drive transactions at the speed of business. As a result, it brings non-technical business users to the forefront of business, thus freeing IT and developer teams to focus on more important tasks. Business users can leverage machine learning algorithms to collect data from different parts of DevOps systems and then map and integrate it in minutes through predictions.

    Here are four ways AI is transforming DevOps:

    1. Improved data access: The lack of free access to complex data streams is a challenge for DevOps teams. AI enables business users to access all the siloed data and harness its true potential for better insights delivery and decision-making. AI-powered tools can gather data from various sources for consistent and repeatable analysis.
    1. Faster data integration: AI and ML-enabled technologies enable non-technical business users to execute data mapping and data integration in minutes instead of months. As non-technical users drive operations, IT becomes free to take up governance and instead focus on more high-value tasks. Thus, not only can AI help business users integrate data faster to deliver experiences and value but also enables IT teams to drive innovation and growth.
    1. Higher implementation efficiency: With the support of AI, businesses can perform more tasks with less human intervention and minimal coding. This reduces the burden laid on IT or developer teams, enabling them to concentrate more on innovation and creativity.
    1. Better security: As security is essential to facilitate a successful software implementation, DevSecOps is a fundamental aspect of software development. Organizations must bolster their security systems to guard against an increase in threats. AI can play a vital role here. It can enhance DevSecOps and boost security by recording threats and executing ML-based anomaly detection through a central logging architecture. By combining AI and DevOps, business users can maximize performance and prevent breaches and thefts.

    Accelerating development cycles while making sure data is interpreted and analyzed properly is a challenge faced by DevOps teams today. Artificial intelligence-powered solutions are helping organizations transform every stage of DevOps building cycles without burdening IT teams or adding costs. Ergo, AI can enable users to transform the DevOps environment, deliver delightful experiences and grow revenue.

  • How to Optimize Data in Software Development

    How to Optimize Data in Software Development

    Data can drive significant business value for every company. Globally, we are collecting more of it than ever before, and it keeps growing—from 45 zettabytes in 2019 to a projected 175 zettabytes by 2025, according to IDC. However, companies are struggling to monetize all the personal, behavioral, engagement and transactional data from customers and operations and turn it into actionable intelligence to drive their digital strategies. Leveraging insights from your data is critical for success and the development of new digital solutions to better serve customers, grow sales, improve operations and more.

    Data helps organizations develop the right software for the right customer at the right time. Data-driven organizations are 23 times more likely to acquire and retain customers than their peers, and are growing at an average of more than 30% annually, according to Forrester.

    Organizations with a deep and dedicated focus on data and analytics tend to follow three common paths to successfully drive their software development initiatives.

    Discovery Matters

    It is nearly impossible to draw value, understanding, and insights from the massive amount of data businesses collect without a proper discovery system in place. Having a comprehensive process of collecting data from various databases and silos and consolidating it into a single source that can be easily and instantly evaluated is critical for any software development project.

    During client engagements, data scientists perform deep-dive research into clients’ data pools and workflows to support client roadmaps. We recently partnered with a global e-commerce company to create a recommendation system for their customers. The discovery process allowed us to verify the feasibility of recommended products using their existing data before moving ahead with the project. Using machine learning, the team was able to clean and analyze information from multiple angles. With that knowledge and confidence, the rest of the development project proceeded quickly knowing the data was in place and the project was viable.

    Align KPIs

    Software development metrics help organizations and their partners track and measure how an application is performing and how effective the development team is in its work. IT organizations rely on a range of KPIs to fully understand software engineers’ progress, as well as gauge software quality including performance, engagement and user satisfaction. While each engagement is different and requires a different set of KPIs, ensuring that the organization and development team are on the same page is crucial.

    During the e-commerce project, success was measured through conversion and user retention rates. Other KPIs that are regularly used include developer productivity and performance metrics like cycle time and team velocity, and software developer metrics like response time, usability and UX data, among others.

    Deploy Agile

    By using agile methodologies at all stages of development, developers are able to focus on business results, not just architecture and new technologies. A similar agile approach can also be applied to data management. When it comes to data science, it’s all about extracting useful information from raw, structured and unstructured data and implementing machine learning models to analyze vast amounts of information. This process requires more creativity and is a key driver for understanding potential risks and failures over the course of any given project.

    With agile, data models and analyses are broken down into manageable bite-sized chunks that can be built and tested quickly—the same way software is developed. If the analytics do not yield the expected results, data scientists see the failures immediately and can correct their course. They could ask the team to analyze different data sets, adjust the models, tweak algorithms or make an educated decision about the project’s viability.

    A multinational credit company took this approach to the next level when it used agile to develop a new, big data hybrid cloud platform. After anonymizing, aggregating and storing information in a data lake, the team quickly realized the database it was planning to use was limited to a certain number of partitions and entities. By running agile sprints and developing and testing the minimum viable product (MVP) continuously, the team was able to seamlessly switch to a new database that had more scale and functionality.

    The value of data is only going to increase with time. Data-driven organizations embrace the use of data and analytics to gain unique business insights to support their software development projects and overall business. By focusing on the data in each and every development project, companies have a tremendous opportunity to gain a competitive advantage and grow their businesses.

  • Security Observability: Why Tracing?

    Security Observability: Why Tracing?

    Detecting a cyberattack can take more than 200 days. During this time, attackers might be exploiting your system while you are completely unaware. Cyberattacks can cost affected organizations as much as $13 million per year.

    What if we could reduce the time to detect a cyberattack all the way down to zero? How much money could you save by detecting and stopping an attack while it is still happening?

    And what if we could do even better than that? What if we could detect security vulnerabilities before an attack even occurred?

    Security observability is critical to achieving these goals. Let’s take a look at what security observability means and how distributed tracing, which is one part of observability, can help to defend your business from cyber security threats.

    What Is Security Observability?

    Knowing what is happening inside software is the best way to learn how to defend it. Observability refers to the ability to see the state of a system at any time. It’s like opening up a watch and watching the gears turn.

    Applications and services constructed using APIs and microservices have an advantage over other systems because you can more easily observe what’s happening inside of them. You add hooks to the APIs and microservices and use additional tools to collect the details.

    This presents us with an opportunity to better secure applications and services against cyberattacks by allowing us to see exactly what is happening inside the system.

    The Three Pillars of Observability

    Three types of data are needed to make a system observable. These are commonly referred to as the “Three Pillars of Observability:

    • Logs: Raw data, recording every activity that happens in the system
    • Metrics: Aggregated performance metrics, such as CPU load and memory usage, collected as min, max, average, number of counts, and so on
    • Distributed traces: The complete path a request follows as it flows through the microservices
    • Data analysis/visualization tools (sometimes referred to as the “fourth” pillar).

    Yes, that makes four, but this item isn’t data. The above data is too complex to digest directly, making these tools paramount to achieving observability.

    Figure 1. Sending observability data from the microservices to their respective collectors.

    The three pillars are shown in the diagram in Figure 1. Each microservice in the diagram contains additional code that sends logs, metrics, and traces to their respective collectors. These collectors gather the data, store it, filter it, and pass it forward to analysis and visualization tools that make the data usable by humans.

    Traditional security tools, such as SIEMs, make use of logs and metrics to secure a company’s assets. However, these tools fall far short of protecting applications.

    Limitations of Logs and Metrics for Protecting Apps

    Logs and metrics begin to paint a picture of what is happening inside your APIs/microservices. But the picture is incomplete. Logs and metrics can help developers ensure that the system is working properly and they can help with solving problems during debugging. But they do not provide enough information to protect the system from cyber attacks.

    Logs: Just the Facts

    Every time a microservice is called or a request is generated, a message describing that activity is sent to the log collector.

    Code Snippet 2. Example of a log message, indicating that Service A has received a GET request.

    This message contains information about what has happened, but it is in a very raw form.  You can think of a log as a fact. It doesn’t provide any real understanding of what happened in the system.

    If this GET request was the first step in an attack, we can’t tell that from looking at the logs alone.

    Metrics: Adding Insights – What Happened

    Metrics are aggregated data collected from microservice activity. Metrics may include things like CPU usage, memory usage, execution time, number of calls or requests per unit of time, and so on. Each metric is associated with the microservice that recorded it.

    Figure 2. Metrics example, showing the number of calls received for the Login service. What do the spikes reveal? Are they attacks, or are they normal user behavior?

    Figure 2. shows activity over time for a login service. The login activity is usually uniform and consistent, but there are very large spikes that occur occasionally.

    The spikes show that there are bursts of login attempts that are occurring occasionally on a system.  They could indicate that a hacker is sending a brute force attack, attempting to guess usernames and passwords for administrators in the system.  Or they could indicate a hacker is trying to overload the server with a Denial of Service (DoS) attack.

    But these spikes might indicate normal user activity.  Perhaps many users are running an app simultaneously because they just saw a compelling television commercial that was aired during the Super Bowl halftime. Or maybe everybody is contacting Uber to request a ride at the same time because a Rolling Stones concert just ended.

    Another possibility is that these bursts of activity indicate a bug in the system.  Maybe there is some sort of unforeseen infinite loop that is causing an unexpected burst of activity.

    We just don’t know what is causing these bursts. To gain the insight we need to secure APIs, we need to know what happened before this metric was generated. We need the ability to work backward.

    Distributed Tracing: Context and Causality – Why Things Happen

    Distributed tracing can help us understand the reason things happen.  When we know why things happen, we can determine if an attack is underway.

    Figure3. A “span”, or trace, showing the path a request follows through the API/microservices system.

    Distributed tracing is a form of tracing that is specially adapted to microservice architectures.  It allows us to observe each request traveling through the system from its beginning to its end. We can use traces to improve performance and understand what typical behavior looks like.

    But traces also give us the power to identify attacks on our microservices and stop them.

    Traces show the entire path a request followed through the edge API and microservices. They can tell us:

    • If the request went through all of the required services, including authentication and authorization checks
    • If the request called any unexpected services or granted access to unauthorized data
    • Where the request entered the system. (If a request entered without going through an edge API, also known as an API gateway, there may be a problem)
    • The IP address that originated the request
    • And if they are working especially well, traces might even help identify an attacker.

    Data Analysis and Visualization Tools: Seeing What Humans Can’t

    In a real system, there may be hundreds of different paths requests can take, just like the one shown in figure 3. And each request could touch tens or even hundreds of services. While distributed tracing gives us the ability to observe all of these, the complexity makes it impossible for humans to gain much insight.

    Machine learning and data visualization tools can identify patterns in the data, in order to identify normal behavior and abnormal behavior, and can even create alerts when abnormal behavior is detected.

    These patterns of normal and abnormal behavior help us to identify security vulnerabilities and attacks within APIs and microservices.

    Tracing to Reveal Vulnerabilities

    The diagram in Figure 4 shows the combination of traces through a system. We can use this diagram to identify possible security vulnerabilities.

    Figure 4. Possible security vulnerability, indicated by the red-colored trace. The diagram shows that this group of requests may inadvertently grant unauthorized access to the user’s account-info.

    Figure 4 shows a pattern that contains an “Excessive Data Exposure” vulnerability, which is vulnerability #3 in the OWASP API Security Top 10. The pattern shows that when a user views a shopping cart, the API will also return the user’s account information.  This unnecessary information may include personal information, account information, or financial information a hacker can exploit.

    These insights help developers recognize potential security vulnerabilities and correct the code to eliminate them.

    Tracing to Detect Attacks in Real-Time

    Tracing can monitor each request and compare it to established norms. The system can generate an alert if a trace deviates from this expected behavior.

    Figure 5. Possible attack in progress. The diagram shows that a possible attacker may have found a way to bypass a required authentication or authorization step.

    Figure 5 illustrates a pattern that could indicate either a “Broken Object Level Authorization” attack or a “Broken User Authentication” attack, which are #1 and #2 in the OWASP API Security Top 10 vulnerabilities.

    This figure indicates that an attacker has found a way to enter another user’s shopping cart and possibly obtain information for that user, by bypassing either an authorization or an authentication check.

    Typically, tracing allows us to go back in time to find attacks such as the one above. Can tracing help to stop attacks in real-time?

    Yes, tracing provides the information we need to detect and stop attacks in real-time. However, more is needed. A sophisticated AI on top of a tracing mechanism would be able to notice abnormalities humans may miss and can stop abnormal behavior before any damage is done.

    Tracing plus AI gives us real-time security observability. In a future blog post, we’ll discuss how AI takes what tracing gives us and uses it to create real-time protection against attacks.

    Use Distributed Tracing to Defend Your APIs

    Cybercrime is expensive, and it can take a long time to detect an attack. Attackers are creative, and older techniques using logging and monitoring do not give enough insight to recognize an attack when it happens.

    Security observability uses internal states to thoroughly comprehend what is happening inside your API/microservices system with the goal of protecting them from attack.

    Observing your software, using logs, metrics, and most importantly, distributed traces gives you the opportunity to identify attack patterns, eliminate security vulnerabilities, and block attacks before they can cause serious damage. View a recorded Traceable demo to see for yourself.

  • Getting Started With Continuous Monitoring

    Getting Started With Continuous Monitoring

    DevOps continues to gain traction among organizations as demand grows for digital product and platform development. According to Gartner, 87% of business leaders believe digitalization is a priority. Every DevOps transformation requires a dedicated, continuous learning process and effective implementation to reach maturity. If a practice or pattern is passed over or ignored, it can put a damper on DevOps success. Continuous monitoring, for example, is a very important part of every DevOps life cycle that is often overlooked.

    Through effective continuous monitoring efforts, organizations can scale and deliver digital products and platforms faster and more securely. In fact, these continuous monitoring advantages tie directly to modern customer needs. If they haven’t already, business leaders must embrace continuous monitoring along every point of the DevOps life cycle. These best practices can help you get started.

    Understanding Continuous Monitoring

    The practice of continuous monitoring helps to collect and analyze outcomes, statuses, exceptions and key metrics within each step of the DevOps process – from development to deployment and production. Additionally, it offers deep insight for DevSecOps teams by enabling compliance triggers and security alerts configuration. It allows the organization to detect issues or security concerns throughout every phase of the DevOps lifecycle.

    Understanding what continuous monitoring does is a good start. However, to effectively adopt continuous monitoring, the organization must know what to monitor.

    What Should Be Monitored

    An effective continuous monitoring process collects enough data to gain sufficient understanding of both successes and failures of the organization’s systems and processes. Some of the events collected by the continuous monitoring systems will require immediate attention, while others may lead to longer-term strategy discussions.

    Collecting the right information is always tricky considering the number of endpoints that generate logs and events. One should rank the significance of each source in relation to the type of data it produces, the functional area it supports and the impact it has on the system’s performance, as well as on organization’s goals and commitments (requirements posed by SLA’s, regulations, contracts, etc.).

    A combination of technology and strategy helps ensure that the right data is collected at the right time. The next step, and perhaps one of the most significant challenges, is finding the balance between monitoring and analysis.

    How to Achieve Balance

    Balance is, by far, one of the most difficult decisions in log monitoring and analysis. Gather too little data and risk missing early trouble indicators or important alerts. Collect too much and get buried in analysis paralysis – essentially, leading to the same situation of overlooking items of significance. Striking the right balance requires understanding the system’s continuous monitoring processing capacity and using its full potential. This may also involve data preparation steps during intake to ensure optimal processing and storage of the collected material.

    Offloading mundane tasks and parts of the analysis to machine learning and artificial intelligence (ML and AI) and fuzzy logic systems will enable big data processing gains. With this approach, the continuous monitoring capacity can be significantly increased, along with the semantic quality of the alerts and notifications produced by the system. As a result, security operation center (SOC) staff can devote their attention to the analysis of events which are more meaningful to the system and operations, leading to a more efficient continuous monitoring process.

    A DevOps transformation will likely not reach full maturity unless the organization applies continuous monitoring to the DevOps life cycle. Although often overlooked, it remains central to DevOps success. With this in mind, a basic understanding of continuous monitoring, what to monitor and finding the balance between log monitoring and analysis, serve as a foundation for getting started with continuous monitoring.