A/B Testing Personalization

It’s best to A/B test your Personalization implementation before going live with it to all your users. This guide assumes knowledge of Algolia’s A/B testing feature and walks you through how to set up A/B tests for Personalization specifically.

Since you can save one Personalization strategy per application at a time, you can’t test different Personalization strategies against each other. You can A/B test non-personalized search versus personalized search or different levels of Personalization impact.

If you’re already sending events and including the userToken on your searches, you can A/B test Personalization entirely through the dashboard. For an overview of the A/B testing feature, please refer to the guide on creating and running A/B tests.

Testing non-personalized versus personalized results

Once you’ve completed gathering user data and configured your Personalization strategy, it’s time to test personalized against non-personalized results. Since you can enable Personalization at query time, you don’t need to create a replica index for this A/B test.

A/B testing Personalization isn’t a replacement for simulating Personalization while configuring your strategy. You should always simulate various users as part of your configuration process.

Using the dashboard

1. Before beginning, ensure that Enable Personalization is off for the test index in the Indices section of the dashboard. You can find this and other settings under Configuration. Disabling Personalization here establishes that the default variant (A) won’t use Personalization.
2. In the Personalization section, ensure that the Personalization impact is a non-zero value. This ensures that when you enable Personalization for variant B, it has an effect.
3. Navigate to A/B testing section of the dashboard. Click New test in the top left.
4. Name your A/B test—something like “Non-personalized vs. personalized results” is appropriate.
5. Name and select your variant indices. Name Variant A “Non-personalized” and Variant B “Personalized.” Select the same index for both.
6. Click + Add query parameter under Variant B. In the modal that appears, select the Personalization tab. Toggle Enable Personalization and Tested value to on.
7. Select the percentage of traffic to send to variant B (the personalized version). The more confident you are in your Personalization implementation, the higher value you can use. If you want to expose fewer users to Personalization, select a lower value. The more uneven the traffic split, the longer the test could take to reach statistical significance.
8. Select the test duration. You should use at least two full business cycles. For example, if you see a predictable conversion trend over a week-long period, you should set the test to last at least two weeks. A shorter period might not include seasonality effects and end with biased results. Though not recommended, you can always stop a test early.
9. Click Create on the bottom right.
10. Once back to the A/B testing home screen, verify that you’ve set up your test as planned. If the setup isn’t correct, select Delete A/B Test and begin again.
    • Both variants use the same index.
    • Variant B displays enablePersonalization:true underneath the index’s name.
    • The test is Running for the desired time.
    • The Traffic split between the variants is as planned.

Once the test is complete (and throughout the test if you wish), interpret the results according to the guide.

Testing different Personalization impacts

Suppose you’ve already established that Personalization is creating a better experience for your users and helping you meet your business goals. In that case, you may want to fine-tune the Personalization impact—that is, the strength Personalization plays in your ranking formula. Since you can set personalizationImpact at query time, you don’t need to create a replica index for this A/B test.

Using the dashboard

1. Before beginning, ensure that Enable Personalization is on for the test index in the Indices section of the dashboard. You can find this and other settings under Configuration. Enabling Personalization here establishes that both variants will use Personalization.
2. In the Personalization section, set the Personalization impact to the non-zero value you want to use for variant A.
3. Navigate to A/B testing section of the dashboard. Click New test in the top left.
4. Name your A/B test—something like “Personalization impact on Products index” is appropriate.
5. Name and select your variant indices. Include the desired Personalization impact score in the names. For example, you could name Variant A “Impact = 50” and Variant B “Impact = 70”. Select the same index for both.
6. Click + Add query parameter under Variant B. In the modal that appears, select the Personalization tab. Toggle Enable Personalization and Tested value to on.
7. Toggle Personalization Impact on, and enter the test value for Variant B.
8. Select the percentage of traffic to send to variant B.
9. Select the test duration. You should use at least two full business cycles. For example, if you generally see a predictable conversion trend over a week-long period, you should set the test to last at least two weeks. A shorter period might not include seasonality effects and end with biased results. Though not recommended, you can always stop a test early.
10. Click Create on the bottom right.
11. Once back to the A/B testing home screen, verify that you’ve set up your test as planned. If the setup isn’t correct, select Delete A/B Test and begin again.
    • Both variants use the same index.
    • Variant B displays enablePersonalization:true and personalizationImpact:70, or whatever you entered for the test value underneath the index’s name.
    • The test is Running for the selected time.
    • The Traffic split between the variants is as planned.

Once the test is complete, and throughout the test if you wish, interpret the results according to the guide.

You shouldn’t run multiple A/B tests on the same index concurrently. If you want to test both Personalization in general and different impact scores using the same index, you need to complete one test after the other.