Multivariate Testing with WebPagetest

Posted in Front End Engineering, Open Source, Web Performance, WebPageTest on by Rick Viscomi.

For your advanced web performance testing needs, WebPagetest’s out-of-the-box functionality can do most of the heavy lifting for you. The scripting interface alone is powerful enough to handle most advanced use cases. For everything else, we have some other strong tools at our disposal, including multi-location and bulk testing. These tools, however, are mutually exclusive. In other words, you can test an array of pages on a single location/browser setup or you could test a single page on an array of location/browser setups. An experimental new feature called multivariate testing (MVT) hopes to change the way you think about bulk testing.

wpt-mvt6

The prototype interface is very simple; you add or remove variables and each combination results in its own test. In the above example, a home page is tested on both its desktop and mobile subdomain pages. Each page is tested on both Android and iOS devices and each of those tests is run over simulated desktop and mobile connection speeds. So for the low cost of selecting a few items from some pulldown menus, eight tests are generated on the fly.

How it Works

The combinations of tests are generated from a simple representation of the test variables. A serialized representation of the example above would look something like this:

{
  "label": "{url} ({browser} - {connection})",
  "url": ["example.com", "m.example.com"],
  "where": ["Public_Dulles"],
  "browser": ["iOS", "Android"],
  "connection": ["Cable", "3G"]
}

The central test controller for WPT, runtest.php, was modified to accept these new test instructions in a fork of the main WPT repository. For now, there are four variables: URL, location, browser, and connection type. The values of each variable are iterated in a series of nested loops, which is how the number of tests is able to grow exponentially with each new value.

Test Label Templates

You may have also noticed a different kind of syntax being used to label each test. With so many variables, MVT needed a new way to uniquely identify each test. The solution was to use a templating syntax already familiar to front end developers; curly braces around the variable name. As each test is generated, the variable placeholders are replaced with their current values, so “{browser}” is replaced by “Chrome” in one test and “Firefox” in the next.

The example above uses a simple template, “{url} ({browser} - {connection})“, to show the changing values of the three variables.

The Results

wpt-mvt7

The MVT results use the same tabular interface as the multi-location and bulk testing with the added benefit of having unique test labels displayed under the “Test” column to easily distinguish between tests. When you have many tests pending, this page is a great progress indicator. Even while other tests are running, you can still drill down into or compare completed tests. In fact, the new label templates are most useful on the comparison page, which is where labels are most heavily used.

Try it Now

You can try out MVT right now by going to rviscomi/webpagetest on GitHub and patching in the “mvt” branch to your own private instance. Or, if you’d rather see it in action first, check out the demo video on YouTube. If you have any issues or feedback, you can comment on the feature request in the main repository.

What’s Next

After thorough testing I hope to submit a pull request to the main repository to have MVT included in all future private instances. In the best case, a limited version of MVT may even be available on the public instance at webpagetest.org. Limited MVT on public instances is extremely important due to the exponential nature of the feature. It’s very easy to spin up many tests at once and consume a lot of shared resources, resulting in a negative experience for the rest of the users, so a cap on the number of variable combinations may be needed.

Some other TODO items for MVT include:

  • exposing configuration of basic test parameters like number of runs, first/repeat views, private/public, etc
  • scripting and custom connection speeds
  • a better UI to indicate that browsers are location-dependent
  • integrating entire MVT UI into the home page (possibly replacing multi-location and bulk testing)
  • ability to limit or disable MVT entirely (settings.ini)
  • expansion of templating syntax, with user-defined template variables
    Example script: setDns example.com {dns} with the dns variable defined as a querystring parameter, like dns[]=1.0.0.1&dns[]=1.0.0.2. This would also make {dns} available as a variable in the label template.

I’m excited to share MVT with the WebPagetest community and see what you all are able to do with it.

One thought on “Multivariate Testing with WebPagetest

  1. Pingback: Velocity Londres 2013 - Jour 3 - Oxalide

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>