At a GrafanaCon event today, Grafana Labs unveiled an update to its core Grafana visualization platform. The company is offering an Explore Metric module which it says can identify an issue’s root cause — and to do so without anyone having to launch a query against the open source Prometheus platform that is at the core of the Grafana Cloud observability platform.
That’s not the only enhancement. Version 11 of Grafana also makes it easier to edit and share dashboards, in addition to streamlining the number of alerts that a single incident might generate. In addition, Grafana Labs is now making available a curated edition of the OpenTelemetry software dubbed Grafana Alloy that it will extend and support alongside the rest of its open source software portfolio.
Finally, the company is updating Loki log management to make it simpler to streamline the storage of log data.
Grafana Labs CTO Thomas Wilkie said that collectively these updates will make it easier for a wider range of IT professionals, including application developers, to invoke the capabilities of a Grafana Cloud platform based on open source software.
As IT environments become more complex, the need for observability platforms becomes more pronounced. IT teams typically have access to a raft of monitoring tools for tracking pre-defined metrics but their ability to launch queries to surface the root cause of an issue has been limited. As observability platforms continue to evolve, the line between observability and monitoring continues to blur, said Wilkie.
Less clear is to what degree observability platforms might enable IT teams to rationalize existing monitoring tools. In the meantime, the pace at which IT teams are embracing observability varies widely from one organization to another. IT teams need enough expertise to launch relevant queries to get the full return on their investment in an observability platform. In the absence of that expertise, acquiring an observability platform becomes less compelling, in the short term.
In theory, there soon should come a day when machine learning algorithms and other forms of artificial intelligence (AI) make it easier for organizations to identify issues without having to craft queries. But it’s not likely AI will eliminate the need for site reliability engineers (SREs), said Wilkie. Instead, AI will make it simpler to onboard new members to a DevOps team while deploying more applications than ever, he added.
It’s still early days so far as adoption of AI and observability is concerned but there will be an inevitable convergence. The overall goal should be to make it easier to troubleshoot highly distributed application environments that today span cloud, on-premises and edge computing applications. That’s critical because as application environments become more distributed the overall size of most of the IT teams tasked with managing them is not increasing. As such, that pace at which DevOps workflows can be automated will prove crucial.
In the meantime, IT teams should expect it will become more difficult to triage complex IT environments before it hopefully becomes simpler thanks to the rise of observability, automation and AI.
Observability has burst onto the scene across all types of operational and security-focused activities. Its need is being driven by increased demands for businesses to be more responsive to changes and more proactive when dealing with potential problems. In particular, security observability holds the promise to help reduce the time to detect a cyberattack. And going a step further, it can help detect security vulnerabilities before an attack even occurs.
What is security observability?
In the past, most operations relied on visibility into their systems and applications to manage security. Visibility would be achieved via monitoring applications, systems, and their logs. Monitoring solutions would collect and analyze information to detect suspicious behavior or unauthorized system changes. And having defined which types of behavior should trigger alerts, solutions would take action as needed.
Perhaps more to the point, monitoring lets you detect a known set of problems or conditions. Thus, it is essential for alerting and analyzing long-term trends. Simply put, monitoring lets users know how their systems and applications are functioning and being used.
However, the problem with security monitoring of complex distributed applications is that it is a passive approach to highly dynamic operations. And it often does not provide any insights into root cause problems or anomalies that might be precursors to problems in the making.
Security observability expands on monitoring by enabling correlation and inspection of the data to provide much deeper insights. Observability typically requires logs, metrics, and deep tracing. All data is used for modeling and analytics. Companies can mine the data, look for patterns, use artificial intelligence and machine learning to remediate problems and proactively defend against problems.
Why security observability is needed now
The need for security observability is growing due to the way applications are developed.
The general industry trend is a move to cloud-native architectures based on microservices. Such an approach brings many benefits. Applications can scale quickly. Companies can easily change or update a small aspect of a larger application without impacting the rest of the app. And businesses can make use of new technologies. For example, a business can add a new front-end or implement more sophisticated AI or machine learning modeling.
Such a development approach can introduce potential blind spots and complexity. In particular, modern applications built using cloud-native architectures and microservices introduce new challenges. The loose coupling of such applications, their distributed nature, and increased complexity makes it harder to understand vulnerabilities. As a result, traditional cybersecurity approaches break down. They miss the interactions and inter-dependencies of the many elements that comprise modern applications.
Furthermore, traditional security monitoring solutions can be limited. For example, visibility often must be split across tools resulting in insecure platforms or situations where relevant data is not even used in monitoring.
Adding to the security challenges is the widescale embracement of low-code/no-code development methodologies. Such technologies open development up to business units and citizen developers. These groups can build sophisticated applications, often with little oversight or guidance. As a result, apps may be created by assembling building blocks that may be outdated, unpatched, or simply have vulnerabilities.
DevSecOps needs security observability
With all these trends, it is not surprising that businesses are exploring new approaches to security. Just as quality assurance was melded into development processes using automated testing, security ops are being meshed with development. Hence, the rapid implementation of DevSecOps methodologies in organizations today.
While security issues can grow when using cloud-native development methodologies, there is a lesser-known security benefit. Applications and services constructed using APIs and microservices have an advantage over other systems because developers and security staff can more easily observe what’s happening inside them. In addition, businesses can add links to the APIs and microservices and use tools to collect the details of such operations.
Given this wealth of data, security observability expands on monitoring. Rather than just detecting vulnerabilities and incidents that raise compliance issues in the development process, DevSecOps using security observability, seeks to identify issues and automatically take corrective actions.
Moving forward, businesses should expect more adoption of DevSecOps and more automation based on security observability. This will only serve to strengthen security and compliance across systems and applications.
Modern applications are mobile first and are built around cloud-native distributed microservices architectures. These architectures have become the basic building blocks for complex and reliable distributed web and mobile applications. Many of these distributed APIs expose the business logic directly over the web; hence the attack surface and attack vectors are very different for these next generation applications. These next-generation applications demand next-generation application security solutions that need to stitch together a distributed application’s activity using techniques such as distributed tracing and then use next generation ML techniques to solve important application security problems.
Proactive anomaly detection
Anomaly detection approaches are a practical way to discover unseen strategies of attackers. However, these approaches are prone to very high false positive rates, thereby defeating their very purpose. Most anomaly detection systems flood analysts with false alarms resulting in a cognitive fatigue. Traditionally, the pitfalls of anomaly detection approaches have been addressed via false-positive workflows, better user interfaces and user experience tools. Analyst feedback is taken as a reinforcement on the algorithm and the algorithm tunes itself. However, the engineering cost of such an approach is very high and the product gets very complex leading to reliability and flexibility issues. Secondly, it still requires analysts in the loop to tune the back-end algorithms. Most practitioners think the issue lies with the anomaly detection algorithms, but in reality there is nothing much that the underlying mathematics powering these algorithms can really do in a cyber security setting. These very algorithms have been proven in domains other than cyber security. At Traceable, we believe that the problem lies elsewhere. Our understanding of the nature of data shows that we need to support anomaly detection algorithms with something else. We need to marry specific application security nuances with the mathematics powering these anomaly detection algorithms. This understanding led us to build TraceAI.
TraceAI and API
TraceAI is the core engine that powers the application security functions of the Traceable Security Portfolio. The foundational principles behind TraceAI are:
To be able to dissect the AppSec requirement into the constituent entities that are root causes for application vulnerability
To be able to holistically observe the interaction behavior among these entities
To be able to discover anomalous patterns in this behavior
Using these guiding principles TraceAI implements a complex behavioral anomaly detector. To power this behavioral anomaly detector, we have identified five entities that interact with one another to result in security vulnerabilities. They are:
APIs – the basic building blocks of the distributed applications
API interaction – how these APIs are wired in order to provide the intended functionality
Actors – users, IP addresses or client-side applications such as mobile apps that use APIs to benefit from the intended functionality or in a malicious case, subvert the intended functionality. The actors may be assigned different roles such as admins, clients, managers etc…
API Data – inputs from actors required for the APIs to come to life. Outputs from APIs in response to the input.
Resources – back-end data or compute functionality requested by actors by calling APIs
TraceAI combines APIs, API interaction, API data, resources and actors holistically to build an actor-centric and API-centric behavioral anomaly product. As a result, common behaviors, outlier behaviors, automaton behaviors (bots), and attack behaviors are discovered from trace data. These behaviors are API-centric as well as Actor-centric.
False positives are eliminated by introducing additional security concepts commonly referred to as Trust-and-Risk models. TraceAI automatically identifies API risk and Actor-trust by observing data relative to a baseline which is again automatically built. The baseline in itself is not a piecemeal integration of various unconnected baselines, but is a holistic combination of various observations modeled using modern machine learning tools. We do not need teams of data scientists and security engineers to support the system as the underlying algorithms automatically learn from data. Once in production, the algorithms run in a self-drive mode, automatically building baselines and generating ML models.
Machine learning algorithms under the hood
The foundation of any machine learning is the depth and breadth of the dataset relative to the number of features used for detection purposes. Traceable agents, deployed inside your application, API gateways, and Kubernetes cluster, automatically collect all the necessary data and build distributed traces that form the data-layer foundation for TraceAI. Traces are continuously collected to become a learning set on logic and data flow from Edge APIs, to internal APIs to backends and third party services and thus generate the features that represent the application context. Having addressed the data challenge via the tracing platform our next challenge was to choose appropriate machine learning technology.
The biggest challenge for machine learning in cybersecurity is having to deal with high-dimensional categorical data of sequential nature, combined with the need for explainability, causality, and attribution. Because the attackers’ strategy constantly changes, supervised learning has its limitations. Most popular ML algorithms fail miserably while simultaneously dealing with these two requirements. Taking these factors into account, we have engineered ML-based solutions by building algorithms ground-up (instead of fully relying on libraries) and adopting a self-supervised learning framework. Our unique way of representing multidimensional categorical information allows us to perform ML-based detection even on certain forms of encrypted data as well.
A key component of the ML-based solution is the ability to discover the API endpoints, and to automatically generate the API spec via data analysis. As a result the expected request/response headers, body parameters and query parameters are not only identified, but are also characterized. This helps build the API spec, violations to which are flagged as anomalies. This completely automated process without the need for manually annotated data is feasible owing to the richness of data captured by the core distributed tracing platform.
Complementing the ability to detect API data anomalies, TraceAI also detects anomalies in API interaction, and anomalies in users accessing these APIs. The User Behavior and Business Logic representation modules capture, represent and build baselines for regular user behavior and expected business logic. Unsupervised ML techniques are employed because these techniques are the only scalable alternatives. Once the baseline models are built, business logic abuse and behavior deviations are captured and surfaced as alerts.
Extending behavioral anomaly detection to predictive threat detection, TraceAI introduces a novel exploratory learning paradigm for API security. This approach harnesses the fundamental principles behind reinforcement learning, adapted uniquely to application and user behavior representation. Using such an approach, we are able to generate potential anomalous patterns even before they occur. By comparing users’ activities against these generated patterns, we can identify users who are beginning to tread on a dangerous path. Hence, we can flag these users even before a full-blown attack is launched. As a byproduct, we are also able to quantify known and risky behaviors without manual intervention. Some of the machine learning techniques that we have used in order to power TraceAI are MinHash based embeddings, hierarchical user behavior clustering using LSH and OLAP-style sub-clustering, Bi-Directional RNN variants for time-series anomaly detection, and use of random walks with evolutionary algorithms for explorative learning.
Anomalies in user behavior in the context of business logic
Understanding application behavior and user behavior helps in:
Identifying violations to known user behaviors
Discovering novel behaviors
Grouping users based on their behaviors to create user cohorts
Determining how exactly APIs are being used in order to build a web application
One can represent user cohort behavior as a summarized flow-graph as shown below.
Sample application flow.
Each node in the graph represents an API that was called during an active user’s session. Two nodes are connected if there is a transition from one API to another, and the flow between two nodes represents various statistics such as how many users exhibit such a transition or how often such a transition occurs across user sessions and so on.
In the above figure, there are around 26 different nodes, each representing an API, and the flows connecting these nodes indicate paths of API access based on monitoring users’ API call graphs. For a human analyst, this map is very difficult to comprehend and more so to detect that a particular user’s behavior has changed compared to other users’ behavior or to detect that a particular user’s behavior has changed compared to a baseline. Only machine learning algorithms can automatically interpret these kinds of representations and that too at the scale that we have to deal with.
Behavioral clusters
The figure above shows how TraceAI was able to create user behavior groups based on the pattern of API access. In essence TraceAI has peered into the complex flow graph shown above and then identified groups of users based on their respective API call flow behavior. The gray circles represent users who fall under commonly occurring behavior groups. These groups are low priority for an analyst as they are population characteristics. The red circle-A represents a user whose behavior is very similar to the gray circle-B. This means the user in circle-A is accessing the same APIs as users in circle-B but managed to attack one or more APIs. This further implies that the user in circle-A is following the same business logic as many other users but has managed to compromise one or more APIs. This is interesting information for an analyst, because one can now understand what led to the compromise in addition to knowing what was compromised.
Tracking user behavior to detect API attacks
Here, the figure above shows the user’s behavior and where exactly the attacks took place. There were authorization bypasses detected in two of the APIs (nodes) colored in red. The analyst now knows where the attack took place and how they got to this stage. The APIs leading up to the attacked API can now be analyzed for security vulnerabilities and code design flaws. In essence TraceAI has isolated and surfaced interesting behaviors from many possible behaviors so that security analysts can investigate further. By cutting down on possible candidates, TraceAI has reduced cognitive load and investigation fatigue by many orders of magnitude. TraceAI can also detect outlier behavior to flag groups of users who are not explicitly responsible for a known attacks, but are exhibiting suspicious behavior and not being like the rest of the users. Analysts can also dig into the outlier groups shown as “blue” circles, analyze few of these users, and then as a batch put all these users under a watch list. By working at the level of a group, multiple users can be tagged/flagged/watched without having to deal with each user separately.
Summary
Traceable combines the power of Distributed Tracing and TraceAI technologies to protect next generation attacks on Web Applications and APIs. By choosing the right machine learning technology, and focusing on addressing API security issues Traceable is pushing the boundaries of API security.
We would love to talk to you further and demonstrate how our technology addresses API security issues in your environments.
The need for modern applications to support continuous integration, delivery and deployment spawned the need for DevOps. Increasingly, as cyberthreats evolve and grow and application development methods and time frames change, security now needs to be an integral part of today’s application life cycle. As a result, businesses should move from DevOps to DevSecOps.
DevSecOps is needed to address security implications introduced by changes in the way applications are now developed, deployed and updated. Some of the main drivers for moving to DevSecOps include:
The speed and frequency of new apps creation and updates: It is no secret that developers are under great pressure to create new applications and features in faster time frames than ever before. Some industries such as banking and financial services are being challenged by nimbler FinTech organizations offering innovative services and better customer engagement. The same is true in retail versus online competitors. And most other businesses are in the midst of a digital transformation. All of this necessitates the development of more apps in shorter time frames.
As the industry knows from DevOps, app development is just one part of the more extensive application lifecycle work that needs to be done. Here again, there is a change from tradition. Beyond the customary needs to update, patch and maintain applications, today’s users are accustomed to quick actions on their demands. They have an app store mentality. When they point out a flaw or suggest an enhancement, they expect these things to be addressed immediately.
They want the same responsiveness in their business applications. It is no longer enough to put items in a Jira or some other project tracking solution for later consideration; actions must be taken immediately.
In both the building and enhancing of the apps, the need for speed can overlook security issues.
Cloud-native architectures: Many modern apps are built using cloud-native architectures based on loosely coupled microservices and containers. The architecture lets developers quickly build and update applications. The approach typically makes use of third-party and open source code to build a larger application. For instance, a loan processing application might include an open source decision engine, an outside financial institution’s credit rating service and a commonly used mobile app user interface.
Another advantage of using a cloud-native architecture is that changes can be made quickly. Rather than writing a monolithic app from scratch or recoding an entire app when small changes need to be made, changes only need to be made to impacted elements. If a new feature needs to be added to a front-end mobile user interface, that’s the only element of the app that must be changed.
These conveniences that make cloud-native so well-liked in the development community, again, can mean that security issues are overlooked.
Low-code/no-code and composable applications: New technology such as low-code and no-code platforms abstract details about the code from developers. And they enable citizen developers, who certainly do not have an eye for security, to build their own apps. Composable techniques further simplify application development via the reuse of components.
Bringing Security Into DevOps
The implication of these industry changes is that security needs to be entrenched in the process throughout an application’s life cycle. It is no longer enough to test for vulnerabilities after the fact.
Specifically, security operations must be integrated at every step of application development, modification, and deployment. It must be real-time instead of reactive. Security can no longer wait for a breach or attack; it must constantly examine the elements of an application as they operate and are modified.
Additionally, DevSecOps is increasingly needed to find weaknesses and vulnerabilities due to hidden dependencies in today’s cobbled-together applications. The bottom line is that implementing cybercrime protection at the code-level and secure coding practices are lower cost, more efficient and more effective than trying to patch security holes in a finished application.
Detecting a cyberattack can take more than 200 days. During this time, attackers might be exploiting your system while you are completely unaware. Cyberattacks can cost affected organizations as much as $13 million per year.
What if we could reduce the time to detect a cyberattack all the way down to zero? How much money could you save by detecting and stopping an attack while it is still happening?
And what if we could do even better than that? What if we could detect security vulnerabilities before an attack even occurred?
Security observability is critical to achieving these goals. Let’s take a look at what security observability means and how distributed tracing, which is one part of observability, can help to defend your business from cyber security threats.
What Is Security Observability?
Knowing what is happening inside software is the best way to learn how to defend it. Observability refers to the ability to see the state of a system at any time. It’s like opening up a watch and watching the gears turn.
Applications and services constructed using APIs and microservices have an advantage over other systems because you can more easily observe what’s happening inside of them. You add hooks to the APIs and microservices and use additional tools to collect the details.
This presents us with an opportunity to better secure applications and services against cyberattacks by allowing us to see exactly what is happening inside the system.
The Three Pillars of Observability
Three types of data are needed to make a system observable. These are commonly referred to as the “Three Pillars of Observability:
Logs: Raw data, recording every activity that happens in the system
Metrics: Aggregated performance metrics, such as CPU load and memory usage, collected as min, max, average, number of counts, and so on
Distributed traces: The complete path a request follows as it flows through the microservices
Data analysis/visualization tools (sometimes referred to as the “fourth” pillar).
Yes, that makes four, but this item isn’t data. The above data is too complex to digest directly, making these tools paramount to achieving observability.
Figure 1. Sending observability data from the microservices to their respective collectors.
The three pillars are shown in the diagram in Figure 1. Each microservice in the diagram contains additional code that sends logs, metrics, and traces to their respective collectors. These collectors gather the data, store it, filter it, and pass it forward to analysis and visualization tools that make the data usable by humans.
Traditional security tools, such as SIEMs, make use of logs and metrics to secure a company’s assets. However, these tools fall far short of protecting applications.
Limitations of Logs and Metrics for Protecting Apps
Logs and metrics begin to paint a picture of what is happening inside your APIs/microservices. But the picture is incomplete. Logs and metrics can help developers ensure that the system is working properly and they can help with solving problems during debugging. But they do not provide enough information to protect the system from cyber attacks.
Logs: Just the Facts
Every time a microservice is called or a request is generated, a message describing that activity is sent to the log collector.
Code Snippet 2. Example of a log message, indicating that Service A has received a GET request.
This message contains information about what has happened, but it is in a very raw form. You can think of a log as a fact. It doesn’t provide any real understanding of what happened in the system.
If this GET request was the first step in an attack, we can’t tell that from looking at the logs alone.
Metrics: Adding Insights – What Happened
Metrics are aggregated data collected from microservice activity. Metrics may include things like CPU usage, memory usage, execution time, number of calls or requests per unit of time, and so on. Each metric is associated with the microservice that recorded it.
Figure 2. Metrics example, showing the number of calls received for the Login service. What do the spikes reveal? Are they attacks, or are they normal user behavior?
Figure 2. shows activity over time for a login service. The login activity is usually uniform and consistent, but there are very large spikes that occur occasionally.
The spikes show that there are bursts of login attempts that are occurring occasionally on a system. They could indicate that a hacker is sending a brute force attack, attempting to guess usernames and passwords for administrators in the system. Or they could indicate a hacker is trying to overload the server with a Denial of Service (DoS) attack.
But these spikes might indicate normal user activity. Perhaps many users are running an app simultaneously because they just saw a compelling television commercial that was aired during the Super Bowl halftime. Or maybe everybody is contacting Uber to request a ride at the same time because a Rolling Stones concert just ended.
Another possibility is that these bursts of activity indicate a bug in the system. Maybe there is some sort of unforeseen infinite loop that is causing an unexpected burst of activity.
We just don’t know what is causing these bursts. To gain the insight we need to secure APIs, we need to know what happened before this metric was generated. We need the ability to work backward.
Distributed Tracing: Context and Causality – Why Things Happen
Distributed tracing can help us understand the reason things happen. When we know why things happen, we can determine if an attack is underway.
Figure3. A “span”, or trace, showing the path a request follows through the API/microservices system.
Distributed tracing is a form of tracing that is specially adapted to microservice architectures. It allows us to observe each request traveling through the system from its beginning to its end. We can use traces to improve performance and understand what typical behavior looks like.
But traces also give us the power to identify attacks on our microservices and stop them.
Traces show the entire path a request followed through the edge API and microservices. They can tell us:
If the request went through all of the required services, including authentication and authorization checks
If the request called any unexpected services or granted access to unauthorized data
Where the request entered the system. (If a request entered without going through an edge API, also known as an API gateway, there may be a problem)
The IP address that originated the request
And if they are working especially well, traces might even help identify an attacker.
Data Analysis and Visualization Tools: Seeing What Humans Can’t
In a real system, there may be hundreds of different paths requests can take, just like the one shown in figure 3. And each request could touch tens or even hundreds of services. While distributed tracing gives us the ability to observe all of these, the complexity makes it impossible for humans to gain much insight.
Machine learning and data visualization tools can identify patterns in the data, in order to identify normal behavior and abnormal behavior, and can even create alerts when abnormal behavior is detected.
These patterns of normal and abnormal behavior help us to identify security vulnerabilities and attacks within APIs and microservices.
Tracing to Reveal Vulnerabilities
The diagram in Figure 4 shows the combination of traces through a system. We can use this diagram to identify possible security vulnerabilities.
Figure 4. Possible security vulnerability, indicated by the red-colored trace. The diagram shows that this group of requests may inadvertently grant unauthorized access to the user’s account-info.
Figure 4 shows a pattern that contains an “Excessive Data Exposure” vulnerability, which is vulnerability #3 in the OWASP API Security Top 10. The pattern shows that when a user views a shopping cart, the API will also return the user’s account information. This unnecessary information may include personal information, account information, or financial information a hacker can exploit.
These insights help developers recognize potential security vulnerabilities and correct the code to eliminate them.
Tracing to Detect Attacks in Real-Time
Tracing can monitor each request and compare it to established norms. The system can generate an alert if a trace deviates from this expected behavior.
Figure 5. Possible attack in progress. The diagram shows that a possible attacker may have found a way to bypass a required authentication or authorization step.
Figure 5 illustrates a pattern that could indicate either a “Broken Object Level Authorization” attack or a “Broken User Authentication” attack, which are #1 and #2 in the OWASP API Security Top 10 vulnerabilities.
This figure indicates that an attacker has found a way to enter another user’s shopping cart and possibly obtain information for that user, by bypassing either an authorization or an authentication check.
Typically, tracing allows us to go back in time to find attacks such as the one above. Can tracing help to stop attacks in real-time?
Yes, tracing provides the information we need to detect and stop attacks in real-time. However, more is needed. A sophisticated AI on top of a tracing mechanism would be able to notice abnormalities humans may miss and can stop abnormal behavior before any damage is done.
Tracing plus AI gives us real-time security observability. In a future blog post, we’ll discuss how AI takes what tracing gives us and uses it to create real-time protection against attacks.
Use Distributed Tracing to Defend Your APIs
Cybercrime is expensive, and it can take a long time to detect an attack. Attackers are creative, and older techniques using logging and monitoring do not give enough insight to recognize an attack when it happens.
Security observability uses internal states to thoroughly comprehend what is happening inside your API/microservices system with the goal of protecting them from attack.
Observing your software, using logs, metrics, and most importantly, distributed traces gives you the opportunity to identify attack patterns, eliminate security vulnerabilities, and block attacks before they can cause serious damage. View a recorded Traceable demo to see for yourself.
