Tag: Uber

  • The Uber API Authorization Vulnerability

    The Uber API Authorization Vulnerability

    What Happened

    In September 2019, a critical bug was discovered on Uber API, which allows merchants, service providers and others to offer ride-sharing services to customers. Uber had exposed a vulnerable application programming interface (API) endpoint that allowed attackers to steal valuable data, including personally identifiable information (PII) records and authentication tokens of riders and drivers. The leaked authentication token could be used to perform a full account takeover.

    Luckily for the company, the vulnerability was discovered before harm could be done. But the case is an example of where traditional security systems can fail to find potential threats because they lack the business context for the application’s logic. Let’s take a closer look at what happened and the implications.

    Technical Details

    When a new Uber driver joins the platform through a referral link, their browser communicates with the API host  “bonjour.uber.com”. The registration process triggers an API call to the API endpoint of:

    POST /marketplace/\_rpc?rpc=getConsentScreenDetails

    The API receives the “userUuid” parameter from the client and returns details about the user. This information is used to populate the consent screen on the client-side:

    As part of a legit flow, the user should send only his own user ID.

    The API endpoint is susceptible to two types of API vulnerabilities:

    • BOLA (Broken Object Level Authorization). Because the program didn’t validate that the client who sent the request had access to the user represented by the user ID parameter, the client could access data of other users by changing the user ID.
    • Excessive Data Exposure. The API response contains a large JSON object with all the user’s details. The API returned this information, even though the client didn’t use it.

    Why Current Security Solutions Can’t Detect This

    When we look carefully at the exploitation flow, we find the attack involved a very subtle change in the API call, replacing one user ID with another. In order to detect such a small change in the traffic, a deep understanding of the business logic of the app is required.

    Current security solutions in the market lack that understanding, usually acting on the context of a single HTTP request. They don’t deeply understand important components of the application that are required to detect BOLA and other authorization vulnerabilities.

    How Traceable AI Solves the Problem

    The Traceable Defense AI approach to detect API attacks is very different than other security solutions. In a nutshell, it observes the data as it is passes through the API and the microservices of the app. It then uses machine learning algorithms to discover the application’s business logic, providing full visibility into the users and their roles, the API endpoints they communicate with, and the resources that the endpoints interact with behind the scenes.

    This methodology allows Traceable to create a baseline of what a legitimate user’s flow looks like, and hence to detect the most sophisticated and subtle API attacks, including BOLA.

    Interested to learn more?

    Watch our recorded demo and see Traceable Defense AI in action!

  • CNCF Advances Jaeger Distributed Tracing Project

    CNCF Advances Jaeger Distributed Tracing Project

    The open source Jaeger distributed tracing platform has officially graduated into the top tier of projects being stewarded by the Cloud Native Computing Foundation (CNCF).

    Designed to be employed within any DevOps environment, Jaeger is a tool for monitoring transactions across complex distributed computing environments. It consists of a backend for trace collection and analysis and multiple client libraries implementing the OpenTracing application programming interface (API) in various languages.

    Developed by Uber, Jaeger now integrates with tools such as OpenTracing, OpenTelemetry, Envoy and Prometheus to make it easier to discover the root cause of issues impacting transactions in complex IT environments. For example, DevOps teams can apply data mining techniques and machine learning algorithms to analyze data from a specific cluster.

    JaegerYuri Shkuro, a software engineer for Uber who works on the Jaeger project, said organizations may rely on tools such as Prometheus to monitor environments, but once an issue is identified, it typically takes a tool such as Jaeger to identify the root causes of an issue. The primary reason for that approach is the amount of data that distributed tracing tools would store if they were to run continuously, he said.

    Organizations that are already using Jaeger in production environments include GrafanaLabs, Northwestern Mutual, SeatGeek, Symantec, Ticketmaster, Uber, Weaveworks and Zenly.

    Jaeger is also starting to be bundled with other platforms, such as the Red Hat OpenShift Service Mesh. As that trend continues, more organizations are likely to become exposed to Jaeger as a tool for debugging a wide variety of issues.

    Although originally developed to address the inherent complexity associated with microservices, Jaeger also can be applied to monolithic applications, Shkuro said. While many of those monolithic applications eventually are likely to be deconstructed into a set of microservices, many of those applications are not going away anytime soon, he noted.

    Jaeger now has 15 active maintainers from four companies, with more than 1,200 contributors and 375 authors of commits and pull requests. It also has more than 9,000 GitHub stars and has been pulled from Docker Hub more than 10 million times.

    While distributed tracing as a concept has been around for a while, the rise of microservices is forcing IT teams to look for tools that make it possible to engage in deeper forensics. The dependencies between microservices coupled with the ephemeral nature of containers often make it challenging for IT organizations to discover the root cause of intermittent issues. Within IT environments that have adopted Kubernetes to run those containers, it’s also become easier to deploy Jaeger using tools as Kubernetes templates, Operator software and a Helm Chart.

    As developers are held more accountable for applications, interest in programmable forensics tools is rising. Performance issues that developers once viewed as a problem to be solved by faceless IT operations staffs are now being solved much faster by developers. As a result, application experiences have improved now that developers are seeking out tools such as Jaeger to identify issues that are preventing them from focusing their efforts on writing additional code. Given the long, difficult history of IT, there are more than a few IT leaders wondering what took them so long to figure that out.

    — Mike Vizard

  • 3 Examples of AI at Work in DataOps

    3 Examples of AI at Work in DataOps

    Artificial intelligence (AI) is making all the difference between innovators and laggards in the global marketplace. Yet, implementing a state-of-the-art DataOps operation involves a long-term commitment to putting in place the right people, processes and tools that will deliver results.

    In this post, we look at three organizations that are doing cutting-edge work in the field of DataOps. We look at the specific strategies they use and the results they’ve seen as they navigate the uncharted waters of DataOps.

    Uber: End-to-End DataOps

    Uber is very vocal about the way it leverages AI across its apps and services globally. The company runs all its machine learning (ML) models on a platform called Michelangelo and use many other proprietary tools such as Manifold, Horovod and Piper to optimize and scale these operations.

    Michelangelo helps manage DataOps in a way similar to DevOps by encouraging iterative development of models and democratizing the access to data and tools across teams. On top of this platform, Uber uses other tools to help data teams.

    Uber ran its models in Tensorflow and realized model training was taking too long because of the large data sets it was dealing with. Taking a page from Facebook’s data team, the company realized the power of distributed training. This would enable Uber to run data operations in parallel on hundreds of distributed GPUs, rather than on a few GPUs. Yet, managing all this would be a challenge, which is why Uber built a tool called Horovod. Now under the auspices of The Linux Foundation’s Deep Learning community, Horovod supports additional frameworks such as PySpark, Petastorm and Apache MXNet. Additionally, using mixed precision training, Horovod is able to optimize the use of memory when running models, and in doing so, it enables running larger data models or running numerous mini-batches of data in parallel.

    Manifold is a visualization tool that helps to gauge the performance of various models against each other. Rather than use a simplistic, two-dimensional axis to plot performance data, Manifold groups large quantities of model performance data into subsets that avoid duplicating the important indicators of performance. This reduces noise and helps speed analysis of the data. The lesson: Look beyond the trees to view the entire forest when it comes to metrics.

    By adopting an end-to-end approach to DataOps and leveraging specialized tools at every step of the pipeline, Uber is pioneering the use of AI in its large-scale consumer applications.

    Netflix: Offline First, Then Online

    Content recommendation is key to the user experience in the Netflix app. Highly relevant content makes the experience personal and drives higher engagement. However, making this happen behind the scenes involves working with large quantities of data and overcoming obstacles every step of the way.

    Netflix uses Spark to power content personalization. While the company has existing models in production, it always is testing new models and hypotheses to improve on existing ones. However, every new model needs to be tested against an older model and should be proven to be better before being implemented. Netflix does this model testing offline rather than online in the live app. The company makes a copy of historical data and performs offline feature generation on this data. The features are stored in a Hive store in AWS S3, and subsequent feature generation and model training is done in Spark. Once a new model is ready, it is run in parallel with a live model and, if it performs better, it is deployed.

    Key to the success of this operation is the logging of historical facts from online to offline. This includes Netflix-specific facts such as video, user and computation facts. Once this historical data is stored, a data worker is able to run simulation tests against it using the newly developed models. These simulation tests are a great way to test models without breaking things and save time by failing fast. Once tested and compared with live models, Netflix goes on to establish different SLAs for different microservice apps that these models serve. All this effort results in a highly relevant recommendation engine and a very personalized experience for Netflix users.

    Airbnb: Tapping Into Related Data Stores

    Airbnb uses AI to improve the search rankings and relevance of its “AirBnB experiences” offering. The company started simply, only recording users’ daily searches, clicks and bookings. Soon, it created features on top of this raw data, including features such as experience duration, price of experience and user reviews, which would help build models. Airbnb used a ranking model to give weight to experiences that had a better conversion rate, better user reviews and a lower price than those with just a higher booking volume. This improved experience bookings by 13%.

    Going further, Airbnb tapped into the other user data it had—homes booked by Airbnb users—to further personalize the ranking of experiences. This is where things get interesting, as AirBnB is able to show experiences that are more likely to match a user’s interests. This is only possible by looking beyond a single data store and looking for relevant data that may exist elsewhere in the organization. This personalization further improved experience booking by 7.9%.

    Similar to Uber, Airbnb uses a combination of offline and online scoring for models. While the historical data is great to get started with and run basic simulation tests, online scoring unlocks additional features such as user location, search filters and login status that wouldn’t otherwise be available offline.

    Conclusion

    AI is changing everything, and DataOps is no exception. By helping companies to make sense of their data more quickly and with less reliance on human effort, AI drives faster, more consistent results.

    The good news for most of us is you don’t need to be a major tech company such as Uber, Netflix or Airbnb to take advantage of AI-driven DataOps. Tools such as Unravel are making AI-powered data insights practical for organizations everywhere, without requiring them to build bespoke DataOps solutions.

    This sponsored article was written on behalf of Unravel.

    — Twain Taylor

  • DevOps Ethics: The Danger of Unethical Code

    DevOps Ethics: The Danger of Unethical Code

    The accident could not have come at a worse time for Uber. Already under scrutiny following several high-profile lawsuits for sexual harassment, intellectual property theft and other accusations that played at least as significant role in founder and former CEO Travis Kalanick’s departure, the ride-hailing firm’s driverless test vehicle was involved in a fatal pedestrian accident earlier this year. It was later revealed the test driver, who was paid to take over driving in case of an emergency, was watching an episode of “The Voice” and thus too distracted to notice a pedestrian who suddenly appeared out of the darkness directly into the car’s path.

    Pending the final verdict of the National Transportation Safety Board (NTSB) investigation, it remains to be seen whether the Uber test driver could or should have been able to avoid the pedestrian or had time to apply the brakes to lessen the impact. However, the Uber SUV’s self-drive sensor system reportedly failed to detect and react to the pedestrian due to software failings—in other words, the system failure likely can be attributed to just plain bad code.

    Prior to the accident, Uber software developers and engineers were already struggling with ethical concerns regarding algorithms they created that, among other things, allegedly were designed to exploit underpaid drivers, deny services to passengers and skirt governmental regulations. A former software engineer, Susan Fowler, also described a toxic culture when filing a sexual harassment legal complaint against Uber.

    However, the allegations levied against the company, even if true, can be seen as an anomaly in software development and DevOps culture in general. One can suppose, for example, that basic human decency dictates most software developers refrain from engaging in unethical behavior and designing code for nefarious reasons. Data from recent studies supports this assumption.

    According to a recently published meta study by Stack Overflow, only a fraction of more than 100,000 developers surveyed said they would write unethical code or “that they have no obligation to consider the ethical implications of code.” When asked what they would they do if asked to write code for an unethical purpose, 58.5 percent reported they would not, while 36.6 percent said it depended on what it was. Only 4.8 percent said they would.

    However, the report describes a lot of “ethical gray.” “Developers are not sure how they would report ethical problems and have differing ideas about who ultimately is responsible for unethical code,” Stack Overflow said in a statement.

    Meanwhile, the fact that many of Uber’s arguably poor ethical choices originated from upper management and its DevOps team points up how, conversely, it is up to DevOps to instill ethical guidelines for organizations to follow. To that end, DevOps must create, among other things, “full audit trails showing who developed what code and when and a culture that enables developers to voice ethics concerns without fear,” said Torsten Volk, an analyst for Enterprise Management Associates (EMA).

    “Knowing that a full audit trail is in place will make DevOps more likely to raise a red flag with their team leads,” Volk said. “Then, ideally, the outcome of this discussion should be documented in a source code management system.”

    In addition to sending the message to their developers that following ethical principles during code creation is the right thing to do, organizations should support an ethical culture for software development—as well as for the entire enterprise.

    “It is critical for the entire business to understand the net positive impact of ethical conduct in development on the businesses risk profile and overall bottom line,” Volk said. “Similar to GDPR compliance, there should be awards for outstanding ethical conduct in coding that can be published to the outside, showing customers the value of doing business with this company.”

    — B. Cameron Gain