Once a product reaches a certain scale, A/B testing and cohort targeting stop being a nice-to-have. There are plenty of good commercial platforms for this, but at scale they push you into a trade-off: Pay a big vendor bill, or send your customers’ data to a third-party API just to figure out which button someone sees. When we built our own experimentation engine, the thing I worried about most was slowing down the request path — extra latency, plus a new single point of failure right when traffic peaks.
Therefore, I set one rule before writing any code: Deciding which experiment a user falls into can never mean an extra network call on the request path — not even a cached one. The variant has to resolve in memory, from data the service already has. Almost everything else in this post follows that one rule.
The reasoning is pretty practical. Experimentation platforms tend to add cost to the hot path in ways you don’t notice at low traffic — and that cost becomes very real exactly when traffic spikes. On our busiest days (holidays, big promos), we run at roughly 3x normal evening load, and that’s also when experiments matter most. I wanted that cost eliminated by design, not something we discover on a dashboard after the fact.
Build vs. Buy
We looked at Split, LaunchDarkly and Kameleoon. On paper, buying is the ‘safe’ choice, and I had to make the case against it to the management directly.
Cost mattered, but the bigger issues were fit and data exposure. None of the vendor cohort models matched how we actually wanted to segment users, and several of them evaluate cohorts by shipping user data to their own back end. For a product handling real customer data, I didn’t want to hand that data to a third party just to pick a variant. Building it ourselves also kept the capability in-house, so we could shape it around the product instead of shaping the product around someone else’s tool.
So, we built it — starting from an open-source feature-flag engine, and over time replacing enough of it that what we run now is basically our own system.
Two Services at Runtime
At runtime, the platform has two microservices.
One is the A/B service, which owns experiment and cohort logic: What experiments exist, how users map to variants and what each variant configures. The other is bff-customer, which handles the initial mobile interaction and publishes structured events into Kafka and the analytics pipelines.
That’s it for the runtime path. There’s more infrastructure around it — config versioning and rollback, an admin panel for defining experiments and segment tables, access control, warehouse jobs for analysis — but none of that sits between a user and their response. The kind of sprawl worth actually worrying about is runtime sprawl: Extra coordinators and decision services sitting between the user and the response. Experimentation platforms have a way of accumulating those, and it’s mostly avoidable.
Deciding the Variant Locally
The A/B service does not decide variants on the request path.
Every back-end service that takes part in an experiment runs a small client library, an SDK, which pulls experiment configuration from the A/B service, caches it in memory and refreshes it on a timer. After that first fetch, working out which cohort a user falls into is a local computation inside the service already handling the request.
In practice, the service builds a context from what it already has, then asks the SDK for a variant:
ABTestContext context = ABTestContext.builder() .installationId(request.header(“installation-id”)) // present for everyone, guests included .userId(currentUser.id()) // null for guests .operationArea(currentUser.city()) // maps to operationArea, e.g. “Berlin” .build();Variant variant = abTestClient.getVariant(“home_screen_layout”, context);if (variant.isEnabled() && “layout_b”.equals(variant.getName())) { return renderNewHomeScreen();}return renderDefaultHomeScreen();
No call to the A/B service, no database lookup, nothing on the network. The config is already in memory, so the request pays only for local work: Build the context, check targeting, hash the identifier, compare the bucket to the rollout range.
This isn’t a new pattern. Unleash, GrowthBook, LaunchDarkly and Split all evaluate locally against cached config, and that was the point: The mature shape of the problem is known. The real risk in building it yourself is subtler. You don’t fail to invent something new; you gradually erode the pattern you started from.
Two traps were worth avoiding, and both were tempting early on.
The first is turning the A/B system into a central runtime decision engine, where every service makes a call to ask which variant a user gets. It looks clean on a diagram. In production, it puts a product branch behind an extra internal hop; when traffic spikes, internal load multiplies and every service depends on that engine staying fast. If it slows, everything that calls it slows with it.
The second is resolving the variant at the gateway and passing the assignment downstream in HTTP headers, so downstream services compute nothing. This works on day one and degrades as experiments multiply: The header accumulates experiment names, variant IDs, config versions and targeting metadata, until someone encodes it into a binary blob to stay under header limits. Now you maintain a cross-service serialization format nobody set out to build.
Local evaluation itself is prior art; the work was in holding the boundary. Every service resolves the variant independently from cached config, request context and a deterministic hash. No extra hops, no header contract, no central bottleneck that scales with user traffic.
Deterministic Assignment and Per-Experiment Salting
Independent services reach the same answer without coordinating because assignment is deterministic. The SDK hashes a stable field (installationId for experiments that include guests, userId for authenticated-only ones) and buckets on it:
bucket = floorMod(hash(installationId + “:” + experimentSalt), 100) // 0..99, stable
Same input, same bucket, every service, every session.
The experimentSalt matters more than it looks, and I know because I shipped the version without it early on. If you hash the bare identifier and reuse that number across experiments, every experiment buckets the same user identically. A user in the top half of the rollout for one experiment lands in the top half of the next, and the next. Run two of those in parallel and their populations are correlated; you can no longer treat them as independent, and reasoning about them separately is quietly wrong. It’s an easy bug to miss, because the numbers just look a little too clean. Salting per experiment decorrelates them: A user’s bucket in experiment X tells you nothing about their bucket in experiment Y. The fix is one string concatenation.
Mutually Exclusive Experiments
Some experiments must not overlap. Per-experiment salts are the wrong tool here, since independent salts would let one user land in both. Instead, we put the conflicting experiments in a shared exclusion group with a single group salt and give them disjoint bucket ranges:
groupBucket = floorMod(hash(installationId + “:” + groupSalt), 100)experiment_A eligible if groupBucket in [0, 49]experiment_B eligible if groupBucket in [50, 99]
As they share the group salt, the partition holds: A user in A’s range is provably not in B’s. Once a user is eligible for an experiment, the per-experiment salt takes back over for the rollout gate and the variant split.
Therefore, there are two hashes doing two jobs. The group hash decides which mutually exclusive experiment a user can see; the experiment hash decides their variant. Collapsing them into one, which is what a bare hash(id) % 100 reused everywhere does, is the subtle bug: Exclusion buckets and rollout buckets become the same number, and two decisions that should be independent stop being independent. No shared state, no coordination, just services applying the same rule to the same salted hash.
One Experiment, Start to Finish
To make the parts concrete, here’s a single experiment from config to analytics.
Suppose we want to test a redesigned home screen, rolled out to half of all users, guests included, so stickiness is the installation ID:
experiment: home_screen_layoutstickiness: installationId rollout: 50%targeting: operationArea: [“Berlin”] # only this delivery area, for the first run variants: A: { name: layout_default } B: { name: layout_b }
The SDK in each service caches that config. When a request comes in, the service builds the context, the SDK checks targeting (Is the user in Berlin?), checks the rollout bucket (floorMod(hash(installationId + “:” + salt), 100) < 50?) and returns layout_default or layout_b. The service renders accordingly. The getVariant call above is exactly this. The user gets a response already aligned to their cohort, with no extra round-trip behind it.
The same resolution also emits an event asynchronously, for analytics. That event involves a distinction that caused us trouble later.
Assignment vs. Exposure
When the SDK resolves a variant, it emits an event to Kafka in the background, on the ABTestUserSegmentation topic, and the request never waits on it:
{ “experimentName”: “home_screen_layout”, “variant”: “layout_b”, “eventType”: “assignment”, “configVersion”: 47, “distinctId”: “device-or-mixpanel-id”, “userId”: “12345”, “serviceName”: “bff-customer”, “timestamp”: “2026-06-16T10:15:30Z”}
Two fields carry more weight than they look. configVersion records which revision of the experiment config produced this decision, so the warehouse can tell whether an assignment and a later exposure were made against the same config. Without it, a join can silently mix a user assigned under one revision with an exposure logged under another, and you’d never see the seam. The exposure event carries the same field for that reason.
eventType is the other. The event records that a service assigned a user to a variant; it does not record that the user saw it. That distinction sounds pedantic until it costs you a result.
For a back-end-only experiment where the branch always executes, assignment and exposure are effectively the same event. For UI experiments, they diverge. A service can assign layout_b to a user who closes the app before the screen renders; analyze on assignment and that user still counts, and their non-exposure pulls the measured effect toward zero. It’s a validity problem, not a logging detail: A variant that worked can read as ‘no significant change’ because part of the assigned population never saw it.
We handle it by splitting the signal. The back end keeps emitting the cheap assignment event. The client emits a separate exposure event, into Mixpanel, when the surface actually renders. The warehouse analysis joins on exposure for any experiment where the two can diverge. Assignment stays free on the request path; exposure gets measured where the render happens. This surfaced when we moved from back-end branching to layout experiments, which is when we stopped treating the SDK’s event as the final word on what a user experienced.
How We Trust a Result: SRM and A/A
Deterministic hashing produces clean-looking splits, which is a reason to check them rather than trust them. A hash bug, a salt collision or a targeting filter that quietly drops one arm all produce splits that look fine but aren’t. So, before a result is read, the analysis layer (not the SDK) runs a sample-ratio-mismatch check: Compare observed arm sizes against the intended ratio with a chi-square, and if the split is off beyond chance, discard the experiment instead of interpreting it.
A real example from one of ours: Arms of 259,962 and 259,039 on an intended 50/50 give χ²(1) ≈ 1.6, p ≈ 0.20. Within chance, no mismatch, safe to proceed. Failing this check has caught far more allocation bugs for us than hours of staring at dashboards ever could.
We also run A/A tests: Split one arm into two non-overlapping halves and confirm no difference. A clean A/A rules out gross allocation and randomization bugs but doesn’t validate the pipeline, since an instrumentation bias hitting both halves equally passes it unnoticed. I treat it as a sanity gate. Assignment being cheap and results being trustworthy are separate problems; this layer handles the second.
The Mobile Client
On startup, the app makes one call:
GET /bff-customer/v1/settings
It comes back with settings and the set of experiments the user is in. From then on the app attaches the right experiment context to its Mixpanel events, and back-end responses are already aligned to the same cohort logic. Mobile, back end and analytics agree without anyone passing assignment state around.
This is a fair thing to be skeptical about, so here is why it holds. The mobile app is another SDK consumer: It fetched config at startup and resolves the same salted hash every back-end service does, with the same id, salt, modulo and bucket. At a given config version, mobile and back end reach the same variant without anyone shipping the assignment over the wire. The remaining gap is the propagation window: If config changes after the app’s startup fetch, the app can lag a back-end service by a few seconds. That is what configVersion is for, so the warehouse can spot the mismatch instead of averaging over it, and an experiment that cannot tolerate the skew has the app re-read `/settings` on foreground. For most experiments, the skew is a non-event.
That alignment is what made debugging sane. When an event shows up tagged home_screen_layout = layout_b, everyone means the same thing by it. Two analytics sinks, fed independently: Mixpanel for product analytics on the client, the warehouse for the heavier back-end work. Neither sits on the request path.
From Random Splits to Segment Targeting
The first version bucketed by hash covered most experiments. Later, product wanted specific groups rather than random slices: Users who order groceries weekly, say, or who engaged with a past promo campaign.
You can’t express those with a modulo. So, we added segment tables: Curated user sets, each defined by analytics, CRM or marketing. Membership is a lookup rather than a computed bucket. An experiment points at a segment by name instead of a percentage:
experiment: weekly_grocery_promostickiness: userIdsegment: users_ordering_groceries_weeklyrollout: 50%variants: A: { name: standard_home_banner } B: { name: personalized_grocery_promo_banner }
At evaluation time, the SDK checks rollout and context as before, and also whether the user is in users_ordering_groceries_weekly. If a population can be expressed as a table, it becomes targetable, without hardcoding user lists in services or writing bespoke logic per experiment. That’s what pushed the platform from ‘A/B testing’ toward something closer to targeting and personalization. For the people running experiments, it’s the difference between a random 10% and exactly the users a hypothesis is about.
This is the main fork in the design, and worth planning for. Hash-based experiments are self- contained: The decision needs nothing beyond what the request already carries. Segment-based ones are not self-contained. A table of a few million IDs won’t cache in every service the way a few kilobytes of flag config does, so membership becomes a lookup against a fast store or a precomputed attribute kept near the service. Once that happens, the ‘all local’ property covers only the hash-based half; the segment half carries a real dependency. We added the segment path alongside the hash-based core rather than rewriting the core around it, so the simple experiments stayed simple. It wasn’t free.
The locality picture fits in one table:
Cost and Timeline
Adoption was cheap by design: A small change on the mobile side, a modest integration on the service side. From studying the commercial options to the first live experiment was about three weeks. The first production version was roughly 42 person-days: Two back-end engineers for three weeks, plus a couple of days each from mobile, an analyst and QA. In a typical month, the platform carried around 20 active experiments.
The property I care about most is the load shape. The A/B service’s traffic has nothing to do with user traffic. A service handling a thousand requests a second still polls config every few seconds and no more. The expensive thing, cohort math on the hot path, and the central thing, the config service, never touch. If the A/B service goes down, nothing user-facing breaks: Every SDK keeps serving the last config it pulled, and the worst case of an outage is that config changes stop propagating until it’s back.
One thing I won’t do is quote a p99 for the assignment itself, because there’s no honest network measurement to quote. The work is a string concatenation, a hash and a range check, in process, on data the service already holds. Microseconds, and nothing crosses the wire. The numbers worth trusting in this stack live elsewhere, in the Redis reads and the delivery layer of the sibling systems. For the branch itself, ‘in-process, negligible’ is the honest claim, and I’ll state that rather than inventing a percentile.
Where the ‘Free’ Claim Has Limits
The claim holds, with three caveats of uneven weight.
Local evaluation removes the hop that decides the variant, but not the hop that supplies the inputs. As long as targeting needs only installationId or `userId`, both already on the request, the decision is free. Once an experiment segments on something the service doesn’t hold, a profile attribute or a live behavioral signal, that data has to be fetched. Since we deliberately don’t carry context in headers, that’s a call to a profile or context service. The coupling we avoided for the decision comes back for the data.
Segment tables are the second. Since I covered them above, I’ll just name the cost: The first version of the platform didn’t anticipate them, and retrofitting a lookup path into a design that assumed everything was local took more care than it should have. We fixed it iteratively and kept the simple experiments simple, but the original design didn’t see it coming.
Determinism holds only while every service is on the same config version. SDKs refresh on a timer, so for the few seconds a change is propagating, one service can be on the new config while another is still on the old. Harmless for almost everything. For the rare experiment that can’t tolerate it, you pin a version per request and accept the cost.
None of these sank the platform. They’re where the design gets loose, and most cost us a confused afternoon in the warehouse before they ended up documented.
The Core Decision
Everything here rests on one choice: Where the variant gets computed — inside each service, behind a cached client, never in a central runtime dependency. Flat load on the config service, negligible request cost, resilience when it goes down, cheap adoption — all of that falls out of that one decision. The real costs showed up later, when targeting got richer and that trade-off was worth making.