How to best A/B Test your Single Page App (client side)

Published in

R&D and Stuff

5 min readMay 9, 2015

The problem with A/B testing Single Page Applications (SPA) is that the client loads only once. No more page refreshing after that, which means that open source libraries like Split (rails) or Feature API (php) can’t help us.

Before I describe the approach I took, I’d like to list my requirements from an A/B testing framework for a SPA:

Javascript API to instantiate a new experiment.
Let’s say I want to test several versions of my on-screen tour. The input for the API should be the experiment name, output will be one of the the experiments variations (tour1/tour2/tour3, ..)
Javascript API to report that a goal was reached.
In order to know which tour was most effective, we need to measure and report our target goal. Let’s say we want to see how the tour helps with purchasing products on our site, we’ll report a “Purchase” event whenever the user completed the checkout process.
Server side to save the statistics and show a report which variation converted the most to my desired goal.

To accomplish the above, I’ve tried two solutions:

Sixpack — Open source framework. Python backend (which I hosted on Heroku), various API flavours (Javascript among them).
Optimizely — Popular SaaS solution. They recently changed their free pricing plan to running unlimited experiments and users.

Although Sixpack worked nicely, after Optimizely changed their plan I decided not to host my own A/B testing solution and switched to them. This post contains step by step instructions and code examples how to implement client side A/B tests (javascript based) using Optimizely.

Optimizely — Setup

I wont go into detail here, you basically create an account and add the javascript snippet to your <head> tag.

Optimizely — Create a new experiment

When you create an experiment, enter your main site’s URL address:

2. After you create the experiment you will be redirected to the experiment editor. Click the “Options” button to your right:

Select “Activation Mode” and change it to “manual”. This allows us to trigger the experiment manually with the Javascript API.

3. It’s time to decide how many variations our test should have. By default the experiment is created with 2 variations (Original and Variation 1). We want to have another variation, so click on “Add Variation” in order for the panel to look like this:

4. Save your experiment (top right button).

Application code — activate experiment and get variation

In our code we now have to display one of the 3 tour variations. This means calling the Optimizely API to get a variation number (0, 1, or 2) and then setting the tour according to that.

I’m using Angular, so I wrote a service to handle this. Here’s the function which activates the experiment and gives you the variation number. We then use the return value of this function to control which tour we show the user:

this.abtestActivateExperiment = function(experimentId) {
  window['optimizely'] = window['optimizely'] || [];
  window.optimizely.push(["activate", experimentId]);
  var variation = window.optimizely.variationMap[experimentId];// If variation exists return it, otherwise return default of 0
  return (typeof variation === 'undefined' ? 0 : variation);
};

I have added some protective code to check that we indeed get a variation, and if not return a default variation (0). Optimizely support told me that calling activate returns immediately and does not need to invoke a http call to the server in order to get the variation, so this should be fine, but just in case..

You can discover the experiment id from the url address when editing the experiment (for example: https://www.optimizely.com/edit?experiment_id=2867090169).

Optimizely — setup goals

What we’ve accomplished up to this point is that each user will get one of the variations of our experiment. Now it’s time to measure which variation is most effective, and we do that by setting goals for the experiment.

In the Optimizely experiment editor, click the “Set Up Goals” button (top right corner):

By default, Optimizely measures the users engagement on your pages, but we’d like to setup a custom goal which is “User purchased an item”. Create a new goal and make sure you select to track “Custom events”:

We will later report the “purchase” even from our code, and Optimizely will automatically calculate which tour variation converts best to this goal. Notice that you can create several goals per experiment. This is very helpful if you want to measure several parameters.

Optimizely — start experiment

This is a good time to start our experiment. Finish creating your goals and click “Start experiment” in the main editor screen.

Application code — report that a goal was reached

In order to report a goal was reached, we send a custom event to Optimizely. In our case, we need to send an event named “purchase”.

The code for this is fairly simple, here is the function I use in my angular service:

this.abtestEvent = function(event_name, event_properties) {
  event_properties = typeof event_properties !== 'undefined' ? event_properties : {};window['optimizely'] = window['optimizely'] || [];
  window.optimizely.push(["trackEvent", event_name, event_properties]);
};

Notice that you can send additional properties to the Optimizely event. It’s not relevant for our case, but the function supports this as well.

Troubleshooting

There’s a useful Optimizely function you can use in order to get their debug logs. In your browsers console use

window.optimizely.push("log");

Summary

That’s it for now. I would love to hear how other people are A/B testing their SPA’s, so feel free to share your thoughts!

Oh, and please help me test my experiments by checking out my product!