Tag: Machine Learning in CI

At Uber, hundreds of engineers working from tens of development centres across the globe push changes to a single shared repository, commonly referred to as a monorepo. A monorepo is a single repository where all — or most — of a company’s code is consolidated rather than distributed across multiple repositories. According to Google’s case study, monorepos offer several benefits, including simplified dependency management, enforcement of consistent code standards and improved code shareability and reuse. However, as commit velocity increases and the number of active contributors grows, monorepos also introduce significant CI challenges — such as an ever-growing queue of pending changes, frequent conflicts, long land times and the critical need to keep the mainline green. A green mainline means that all build steps –  compilation, unit tests, and UI tests – consistently pass at every commit point in the repository’s history. 

Problem & Bottlenecks 

When the mainline goes red, it disrupts the release of new features, as no new code can be safely released. In high-stakes environments, even small delays can translate into millions of dollars in business impact. Beyond the financial implications, it also affects developer productivity and confidence. Engineers are often left uncertain whether test failures are due to issues in their own changes or because the mainline is already broken — leading to wasted time, frustration and reduced trust in the monorepos. Additionally, rollbacks become more complex, as teams must identify the last known stable version of the repository and coordinate reverts accordingly — slowing down recovery and increasing operational overhead. 

SubmitQueue’s Approach to Streamline CI 

SubmitQueue is Uber’s mission-critical merge queue system that gates all landings to always keep the mainline green at scale. It operates by providing an illusion of a single queue where every change is enqueued. It speculatively executes builds of queued changes and only pushes changes where the CI step completes successfully. It also considers build targets when evaluating changes – that is, if changes A and B do not change any shared build targets, they are considered independent and may be evaluated in parallel. 

Before SubmitQueue at Uber, the mainlines of the monorepos would often break (go ‘red’) due to developers racing to commit their changes, especially during the crunch periods to meet ship deadlines. Each time the mainline went red, commit activity would stop while someone identified the offending commit and rolled back the changes, which disrupted development and slowed everything down. Introducing this queueing system was seen as a significant boost in developer productivity for Uber developers. 

Key Innovations 

1. Speculative Validation: SubmitQueue doesn’t just validate one change at a time — it validates multiple in parallel using speculative builds of pending changes. This significantly improves queue throughput while maintaining correctness. 
2. Predictive Modeling in CI: SubmitQueue employs machine learning to predict the change success, and the build time and schedules the builds of changes to land faster while utilizing resources very efficiently. 
3. Conflict Analysis: It leverages a build system to determine conflicts between the changes, to evaluate changes independently if they are not conflicting 
4. Prioritization & Fairness: SubmitQueue incorporates policies to balance throughput with fairness, ensuring no one team monopolizes the queue. 

The Impact 

1. 100% green mainline uptime across thousands of commits per week across all the monorepos 
2. Significant reduction in CI resource consumption due to smarter speculative scheduling 
3. Improved developer productivity and velocity, enabling fast iteration without risking mainline stability 

Lessons Learned 

1. Continuous integration is more than just running scripts, They should be treated with the same engineering rigor as any other core platform service. 
2. Use of machine learning in CI can drastically improve throughput and reduce resource usage. 
3. Stability of the mainlines of the monorepos is directly proportional to the developer productivity and experience 

Conclusion 

Efficient continuous integration at scale requires smart scheduling algorithms, predictive modeling and platform thinking. At Uber, SubmitQueue has become foundational to ship code with confidence. As more organizations operate at a similar scale, systems like this will no longer be optional — they’ll be essential. 

To learn more about our approach, explore our publications: Keeping Master Green at Scale and CI at Scale: Lean, Green and Fast.