Skip to main content
A/B testing lets you split live traffic between different ranking configurations and measure which performs better. You can test rule combinations, pipeline settings, or entirely different strategies against a control group.

​
Creating an experiment

1

Open A/B Testing

Navigate to Console > A/B Testing.
2

Start a new experiment

Click New Experiment.
3

Name the hypothesis

Enter a name and description. The description should capture your hypothesis, for example “Pinning new releases to top 3 will increase click-through rate by 15%.”
4

Configure variants

Switch to the Variants tab and configure at least two variants:
  • Control - the baseline experience (typically your current configuration).
  • Treatment - the change you want to test.
5

Set traffic fractions

Set traffic fractions using the sliders. They should sum to 100%.
6

Create the experiment

Click Create Experiment.
The experiment starts in draft status and does not affect live traffic until you move it to running.

​
Variant configuration

Each variant has:

​
Config overrides

Use config_overrides to control which rules apply per variant:
  • include_rule_ids — only these rules apply for users in this variant. All other rules are skipped.
  • exclude_rule_ids — these specific rules are skipped. All other rules apply normally.
This is how you test the impact of a specific rule or set of rules against a control group.

​
Traffic assignment

Assignment is deterministic and consistent:
1

Hash the request identity

For each request, the engine computes hash(user_id + experiment_id) mod 1000.
2

Match the traffic bucket

The result is matched against cumulative traffic fraction buckets.
3

Reuse the same assignment

The same user always gets the same variant for a given experiment.
This means:
  • No cookies or session storage required.
  • Assignment is consistent across requests and devices (as long as the user ID is the same).
  • You can run multiple experiments simultaneously — each experiment assigns independently.

​
Experiment lifecycle

To change status, open the experiment and click the desired status button on the Setup tab.

​
Measuring results

The Results tab shows computed metrics for each variant.

​
Available metrics

​
Lift calculation

For each treatment variant, the Results tab shows lift vs. control:
  • Positive lift (green) means the treatment outperformed the control.
  • Negative lift (red) means the treatment underperformed.
  • Lift is calculated as (treatment_metric - control_metric) / control_metric.

​
Refreshing metrics

Click Refresh metrics to recompute from the latest data. Metrics are computed by:
1

Query served users

Query all users who were served recommendations during the experiment window.
2

Reassign variants

Deterministically re-assign each user to a variant using the same hash as the live engine.
3

Aggregate metrics

Aggregate served impressions and user events per variant.
Refresh as often as you need — each click pulls the latest data.

​
Best practices

  • Run experiments for at least 1-2 weeks to account for day-of-week effects.
  • Don’t change rules mid-experiment unless you intentionally want to measure the impact of the change.
  • Use meaningful sample sizes. If one variant has very few users, the metrics will be noisy. Ensure traffic fractions give each variant enough volume.
  • Document your hypothesis in the experiment description so you can review what you were testing months later.
  • Complete experiments when done. This sets the end date and freezes the measurement window.