devopsy.kinsta.cloud

Tag: APIs

  • 7 API Observability Anti-Patterns to Avoid

    7 API Observability Anti-Patterns to Avoid

    Observability has become a buzzword, and the API world is slowly catching up. But be careful when looking for content about API observability- there are many outdated best practices and irrelevant pieces of content.

    I was recently invited as a guest to Stephen Townshend’s SlightReliability podcast, and his brilliant blog post on “Bad Observability” inspired me to start ranting about the topic.

    Here is a follow-up to our discussion: The observability anti-pattern to avoid when working with APIs. Thanks, Stephen, for the inspiration!

    Anti-Pattern: You Forgot Your Users

    It’s easy to lose sight of the fact that APIs are built for people to use. APIs are not just a technical interface; they are a bridge that connects digital experiences and enables people to accomplish tasks efficiently.

    There are two different kinds of users for your API:

    • The developers who are integrating your API into their system.
    • Your customers’ users, who will ultimately interact with the applications or services that rely on your API.

    If you can’t understand how they interact with your APIs, you’re missing half the picture.

    When developers integrate with your APIs, how long does it take them to make their first successful API call? What are typical errors slowing them down? Observability plays an essential role in surfacing insights to improve developer experience.

    Once your customers’ applications or services relying on your APIs are in production, can you track how your releases impacted their requests? Which version of your APIs are they using? What are the most popular endpoints? Are all customers getting the same performance? Same error rates?

    Anti-Pattern: Relying Only on API Monitoring

    API monitoring typically involves sending automated test requests to API endpoints and verifying the responses. It helps you keep track of the health of your APIs even when there is no customer activity at all, and can be used to report uptime.

    But API monitoring is nothing more than black box testing; you don’t see what is happening inside. You just know that your requests reach the API and receive responses, but you can only create and maintain tests for some possible use cases. You will miss something.

    Anti-Pattern: Using API Access Logs for Troubleshooting

    API access logs are records of requests and responses made to an API, capturing details such as the requestor’s IP address, HTTP method, requested endpoint, status codes and other relevant information for monitoring, debugging and security purposes.

    Using access logs to troubleshoot API issues is like searching for a needle in your microservices haystack. One of our users had a performance issue with their APIs; they spent days searching for the root cause using access logs. Is it the gateway? Does it need more CPU? Is it a configuration issue? They enabled distributed tracing and, within seconds, they could see what was happening: All the time was spent in the authentication server issuing JWT tokens.

    To really understand what is happening in your system, you need to be able to track a single user request from the API gateway to the upstream services and their dependencies for effective troubleshooting.

    As Martin Thwaites puts it: “Logs are a poor man’s observability tool.” You need distributed tracing.

    Anti-Pattern: Different Teams, Different Tools, Different Data

    Different teams have different needs. The tool that works for your DevOps team might not fit your product team best. By forcing everyone to use the same tool, you’re not promoting unity but inefficiency.

    However, while promoting tool diversity might be the right move for your organization, there should be some coordination and integration between different tools and data sources. Open standards like OpenTelemetry help ensure that information can be shared and leveraged across teams and tools.

    The DevOps and the product teams should rely on the same error rate when assessing the performance and reliability of the newest API product launched.

    Anti-Pattern: One Size Fits All

    Different API architecture styles have different needs when it comes to observability.

    REST APIs might be today’s default, but newer styles like GraphQL and gRPC are gaining popularity. Each of these styles has distinct characteristics and requirements regarding observability. Just think about GraphQL’s way of handling errors, which differs significantly from REST APIs. Web Socket and event-driven APIs have their own unique observability challenges, as well.

    Different API architectures have different needs, and your observability strategy needs to reflect that to improve the overall reliability of your application.

    Anti-Pattern: Observability for Production Only

    Why aren’t you using observability in pre-production?

    By using distributed tracing during the API development life cycles, developers can trace the flow of requests and responses across different services, providing a comprehensive view of how the API behaves under various scenarios.

    With observability, they can identify performance bottlenecks and even detect architectural problems like N+1 problems in GraphQL queries.

    For teams leveraging APIOps practices, observability data can be used as an additional validation before promoting an API to the production stage.

    Anti-Pattern: Starting the Trace at the API Gateway is Overrated

    Tutorials about observability usually start the instrumentation process at a microservice level. It is great to have detailed insights into your microservices. But a lot will happen at the API gateway level when dealing with APIs.

    You want to capture all your user transactions, including those that never reach your microservices because of rate-limiting rules, an authentication problem or a caching mechanism.

    Starting the trace at the API gateway gives you a clear entrance point and a complete picture of the journey of all your users. That’s why it’s important to use a modern API gateway with built-in support for OpenTelemetry.

    So there you have it: Seven API observability pitfalls to avoid. Keep your APIs observable and your users happy!

    To hear more about cloud-native topics, join the Cloud Native Computing Foundation, Techstrong Group and the entire cloud-native community in Paris, France at KubeCon+CloudNativeCon EU 2024 – March 19-22, 2024.

  • Squaring the Circle: How to Make Public APIs Private

    Squaring the Circle: How to Make Public APIs Private

    APIs have become an integral part of enabling seamless data exchange and functionality between systems and businesses. In fact, APIs are trebling every year. Public APIs are, by their nature, designed to be open and discoverable. By definition, this also means that APIs are open to attacks from the networks. This is why APIs have become the number one attack vector.

    APIs becoming the top attack vector makes sense. All our other business apps are hidden behind private networks–usually MPLS (AWS Direct Connect, Azure ExpressRoute, etc.), VPN or zero-trust network access (ZTNA). But APIs are exposed to the public Internet.

    Day 2 API Security Solutions and Snowflakes

    This network exposure is why there are so many ‘Day 2′ API security solutions – WAFs, API security platforms, IDS, etc. These solutions do a great job of blocking threats that have already been identified, yet there are successful API attacks each day. Why?

    Each API is unique and evolves rapidly–constant changes in business logic, authentication, authorization and configurations. APIs are snowflakes. The result is that many API attacks are effectively zero-day attacks–novel attacks that exploit recent and unique changes to specific APIs.

    A Difficult Race to Win

    Network exposure and snowflake characteristics are why API-related vulnerabilities multiply each day.

    The ‘snowflake’ issue means many attacks are unknown to the Day 2 solutions – they attack vulnerabilities that didn’t previously exist. So, after the novel attack, we race to apply a fix. Who are we racing against? Every attacker on the internet. The API is public–exposed to the networks–not like our other business apps, which are private behind an MPLS, VPN or zero-trust solution. Racing against attackers on the internet is difficult–and coming second may result in a service outage, data breach or customer compromise.

    Why Not Avoid the Race?

    So why not make the APIs private, avoid exposing them to the internet, put them behind MPLS, VPN or ZTNA, like we do for all our other business apps? API clients are too distributed and dynamic; there is no good way to manage agents or dedicated circuits across all those API clients. Those solutions were not made for APIs–it is like fitting a square peg in a round hole.

    Zero-Trust API Networking Needs to be Built for APIs

    To make our APIs private like the rest of our business apps, we must first meet the business need. API providers need to achieve the business velocity and agility they require. On the technical side, we need a secure networking solution which is built to manage the distribution of the API clients, and assume that there will always be snowflake-like vulnerabilities.

    Let’s look at an open source solution for zero trust APIs which is purpose-built for those business, technical and operational requirements:

    There are usually three reactions to the diagram above:

    1. OK, tell me how it works. See below.
    2. Looks complex. Nope. All software, spun up in minutes, centrally managed.
    3. Seems like a proprietary solution for users accessing their applications. Yes! It uses principles similar to the user-to-app zero-trust world but secures APIs.

    How Zero-Trust API Networking Works

    Here are the four pillars of zero-trust API networking:

    1. Eliminate public access. Your API clients will use private IPs or domains to reach your APIs. If your company name is Acme, then your clients might now use api7.acme.unreachable (you make up any domain you want). Envision an attacker trying to interrogate *.acme.unreachable to find snowflake vulnerabilities. Not possible because it is not a public domain—your API server is no longer reachable.
    2. Identity, authentication, authorization, mutual TLS. Authorized API clients do need to access your APIs from the internet. Your zero-trust API overlay network is accessed from the internet only with X.509-certificate cryptographically authenticated identities, continuous authentication and least privileged access authorization via mTLS links. You move the policy enforcement point (PEP) from your DMZ to your API clients. Your API clients add a few lines of zero-trust networking code to enable this model. This code replaces VPNs, MPLS, etc. (there are non-code options too). For example, some of your API clients likely use Resty, a popular open source REST client library for Golang. Their existing Resty code adds a few lines of zero-trust API networking code: 
    3. Outbound-only. You eliminated the inbound access. Your server will instead open identified, authenticated and authorized sockets to your zero-trust API overlay network (you’ll close all your inbound ports and only enable outbound ports to specific overlay network router IPs or to your x.acme.unreachable domain).
    4. Management. API networking security is not very valuable without simple management. The zero-trust API overlay cloud is software-only and cloud-native. You spin up your private API clouds in minutes, anywhere, without deploying hardware. If an API client is compromised, one click or automated action removes its access. Network usage for each client/API is in your web console or management dashboard.

    The above is based on the open source OpenZiti zero-trust networking platform, but any zero-trust API platform should provide similar functionality. Specifically, it should enable:

    • Speed. Securing APIs but slowing down business or complicating operations is not the trade-off most organizations want. Security and speed is the goal.
    • Minimize the threat surface. Close all inbound ports in front of your API servers. Use private, unreachable addresses or domains to minimize your attack surface and stop racing against network-based attackers.
    • Extensibility. Software-only, cloud-native architectures. Preferably open source based.
    • Operations excellence. Centralized visibility, controls, policies and identities. Rich telemetry data. Performance-optimized overlay networks.
    • Zero-trust security. Identity-based networking, continuous authentication, strong authorization, mutual TLS (mTLS), encrypted delivery, microsegmentation.
    • Flexibility. API clients add a few lines of code to get zero-trust API networking. Other API clients use API-specific agents or DNS to access the overlay, if desired, while your API server/gateway is still private.
    • Recovery. Assume breaches. Implement microsegmentation, prevent data exfiltration and provide simple controls to immediately remove threats.

    Zero-Trust API Networking

    While APIs will always be snowflakes, they no longer need to be our only business app that is exposed to the networks. By minimizing the attack surface, without compromising business velocity and API client experience, we can finally mitigate against our greatest API security risk. This also simplifies our day two API security, the solutions that inspect API queries after they enter our DMZ.

    Rather than trying to filter the internet, day two solutions can focus on detecting authorized API clients that may have been compromised. Similarly, our operations teams are freed to focus on sessions from authorized API clients rather than wading through oceans of data from port scanning, DDoS attacks, unauthorized API queries and attempts to interrogate public APIs to find new snowflake vulnerabilities.

  • We Will Control the World!

    We Will Control the World!

    One thing about centralizing control is that there always has to be someone in charge. That can be a very good thing or a very bad thing. In IT, it doesn’t even get far enough to be a terrible thing.

    Generative AI and increasing integrations have renewed the insistence from the crowd that it is certain they should control every little thing in your (virtual) data center. The idea that a given vendor should be the configuration and management point for a wide variety of tools – their own, competitors and nearby market tools – has been around forever. Vendors love the idea that they have that level of input into your infrastructure and, let’s be honest, many of us love the idea of having it taken care of for us.

    But there are several questions to ask here. First, what is the likelihood that vendor X will have a great level of support for the tools from other vendors in your infrastructure? I mean, before we even get into the idea that this desire for control is not at all altruistic (we’re not talking Stalin here, but we certainly aren’t talking Washington, either), there is the question of how AI will be able to learn about and configure/manage a competitor’s products. Watching wire traffic? Well, if the competitor’s systems are super-well-known and the format of configuration APIs/files are well known—maybe? But this leads us to the real problem.

    My advice to vendors would be to get their own products under control first. We work in a high-complexity field where the tools require a lot of specific domain knowledge to manage, and even more to manage well. Vendors have generally taken for granted that this level of knowledge would be available. Before making that level of knowledge unnecessary for competitors or spiderwebbing into competitor installs, make your tool(s) damn simple to use. That doesn’t mean you have to simplify it—you have the power of AI and are going to focus on configuration and management. So nearly eliminate configuration issues with your own products.

    Example: “Great Security Tool, I just added an API at https://devopsy.kinsta.cloud/rocks. Please lock it down so it is accessible only to users in the WeRule group, then generate a WAF rule to keep it from being used to access more than three records a day, and only allow GET and POST calls.” Most vendors across the industry aren’t even close to this level of simplicity, so why in the world would you trust them to configure/control other vendor’s products?

    There is a trend to make products cover more of the space they are in. This is very big in security right now, but you also see it in DevOps. CI/CD/CDD/VC/kitchen sink (KS)—I’m okay with that, as it does offer us simplified configuration and better cross-app communication. I think this is a better route than trying to serve as command-and-control for an entire tech stack that the vendor does not own. I still want point products and best-of-breed available, but a lot of us just don’t have the time and resources to implement five different products to get the job done—even if they’re “free,” as we all know, they are not free—so one vendor doing an entire chain is great for that use case, and often the shared information about the app, security, the network, etc. can work together better than the sum of its parts. So let’s stay in the “umbrella app” lane and not get carried away with trying to drive in the “manager of the entire world” lane.

    We’re busy, we’re tired and many of us are jaded. So don’t tell us how you can manage other vendors’ tools, tell us how you’re making it easier to manage yours. Thanks, from all of us in IT.

  • Salesforce Adds Ability to Apply Custom Security Policies to APIs

    Salesforce Adds Ability to Apply Custom Security Policies to APIs

    Salesforce today added the ability to customize security policies for application programming interfaces (APIs) to the Mulesoft Anypoint API management platform.

    Gerry Egan, vice president of product management for Mulesoft at Salesforce, said this capability makes it possible for organizations to use an Anypoint Flex Gateway Policy Development Kit to streamline development of custom API security policies that can be implemented as code.

    In addition to providing integrations with integrated development environments (IDEs), the Anypoint Flex Gateway Policy Development Kit also provides test policies that can be employed to ensure API security.

    That approach makes it simpler to ensure APIs are secure as they are being built and deployed within the context of an automated DevOps workflow versus having to rely entirely on a separate API security platform, he noted.

    The Mulesoft Anypoint Gateway already comes with security policies that DevSecOps teams can readily implement. Salesforce is now adding the ability to apply custom policies that DevOps teams can apply as it becomes apparent cyberattacks against APIs are becoming more sophisticated, noted Egan. Those policies can, for example, be used to remove sensitive data or replace it with a token, he added.

    Responsibility for API security is often not well established in many organizations, but as more accountability for API security shifts left toward DevSecOps teams, there is growing interest in addressing the issue as code when APIs are developed and deployed. While cybersecurity teams are responsible for maintaining cybersecurity after APIs are deployed, few of them have any insight into how an API is crafted. That’s a significant issue because cybercriminals have become more adept at both exfiltrating data via external facing APIs in addition to manipulating the underlying business logic to compromise a digital process, noted Egan.

    The Mulesoft Anypoint Gateway automates the process of building and applying security policies to APIs as part of the software development life cycle (SDLC) in a way that minimizes any impact on developer productivity, he added. That approach provides the added benefit of reducing the level of cognitive load developers would otherwise experience if they had to manually create software policies, said Egan.

    In general, most APIs are internally facing, but it’s not uncommon for them to suddenly be exposed to the internet as application use cases evolve. The best way to ensure API security is at the point of creation. Providing developers with API security tools gives them an opportunity to address security issues at the beginning of the application development process.

    There have already been several cybersecurity breaches involving APIs in the past year. It may only be a matter of time before there are additional breaches that could prove catastrophic. Regardless of the size of the API breach, the application development team that created and deployed the API in the first place is ultimately going to be held accountable for fixing it. The challenge is to find the simplest way to eliminate the issue in the first place at a time when the number of APIs being built and deployed only continues to explode.

  • Graylog Makes Free API Security Tool Available to Developers

    Graylog Makes Free API Security Tool Available to Developers

    Graylog this week made available a free version of its platform for securing application programming interfaces (APIs), intended to encourage developers to adopt best practices to secure them.

    The free version of Graylog API Security has all the capabilities of the paid version but is limited to 16GB of local rolling storage on a single node with a one-year renewable license.

    Graylog CEO Andy Grolnick said the goal is to make it simple for developers to use this edition of the platform as they are building APIs. As applications are deployed in a production environment, the expectation is organizations will then need to access the version of the platform that is designed to scale, he added.

    Based on technology Graylog gained with the acquisition of an API security platform originally developed by Resurface.io, this free version of Graylog’s API security platform is trying to help bridge the divide between the application development teams that create APIs and the cybersecurity teams tasked with protecting them, said Grolnick.

    Capabilities of the platform include API classification, discovery, risk scoring, an ability to capture API request and response payloads and continuous monitoring of API and threat signatures in real-time. Collectively, these capabilities enable Graylog API Security to provide high-fidelity alerts using a data lake that is embedded in the platform, noted Grolnick.

    That approach enables Graylog API Security to capture all API requests and response details to distinguish valid API traffic from malicious activity involving, for example, data exfiltration hiding under valid response codes.

    In addition, Graylog API Security automatically surfaces remediation guidance for developers who typically lack cybersecurity expertise.

    API security has become a major DevSecOps issue because cybercriminals are increasingly targeting APIs to steal data. As more externally facing APIs are deployed, the overall size of the attack surface that needs to be defended increases exponentially.

    A recent survey conducted by Enterprise Strategy Group (ESG) on behalf of Graylog found more than three-quarters of respondents (76%) have an average of 26 APIs per application. Most respondents are using open APIs for public consumption (67%) or are connecting applications with partners (64%).

    In general, most of those APIs are internally facing, but it’s not uncommon for those APIs to suddenly be exposed to the internet as use cases for applications evolve. The challenge is that few cybersecurity teams have visibility into APIs that, when manipulated by malicious actors, can expose code or provide a mechanism through which business processes can be compromised. This issue often exists because, in many organizations, it’s still not clear who is actually responsible for API security.

    Arguably, the best way to ensure API security is at the point of creation. Providing developers with API security tools gives them a chance to do the right thing from the beginning of the application development process, said Grolnick.

    There have already been several cybersecurity breaches involving APIs in the past year. It may only be a matter of time before there are additional breaches that could prove catastrophic. Regardless of the size of the API breach, the application development team that created and deployed the API in the first place is ultimately going to be held accountable for fixing it.