Category Archives: Open Source

WebPageTest Private Instance in 5 Minutes or Less

Posted in Front End Engineering, Open Source, Web Performance, WebPageTest on by Rick Viscomi.

If you’re like me and you have ideas to help improve the WebPageTest public instance UI or you want to crush some bugs, you have a couple of development options. If it’s just a tweak to the styles or similar, you could verify the change live in browser developer tools and submit a small pull request to the GitHub repository. New features that are more complex can’t be done as easily on the fly. A development server should be used to ensure the changes work as intended and don’t break anything. So what are the options for testing WebPageTest locally?

Running WebPageTest on your laptop


My first choice is to run the development server on my laptop running OSX. For how easy it is, there are surprisingly few guides written on getting WebPageTest set up for ad hoc development. Here’s an overview of the process:

  1. Get a clean copy of the code.
  2. Configure the private instance.
  3. Configure Apache to point to the server.

The goal is to accomplish these three steps in under five minutes, so I may take some shortcuts or make assumptions about your development environment. I’ll be sure to note these along the way.

1. Clone the latest version from GitHub

If you want simple, this is it. Just download the master at https://github.com/WPO-Foundation/webpagetest/archive/master.zip. Save it anywhere.

Note: If you already have the GitHub Desktop application installed, here’s a shortcut that will clone the repository for you: github-mac://openRepo/https://github.com/WPO-Foundation/webpagetest

It would also be proper version control etiquette to create a new branch for your changes rather than work in the master branch.

2. Point the private instance to the public test agents

This is key. If we’re only interested in making changes to the UI, it’s totally unnecessary to spin up your own test agents. If your change doesn’t depend on live data at all, you could skip this step. For example, the home page is static and as long as you don’t have to submit a test, you can work entirely local. However, most of the site involves configuring and analyzing tests, which definitely requires live data. The trick is that you can let the public instance do the actual testing workload while your private instance simply pulls in the data and displays it.

The WebPageTest API

Now is a good time to bring up the API. Of course, if you already have an API key you can skip this part.

If you want to communicate with the public instance, you’ll need to be authorized. All API users have a unique key, which can be immediately obtained by filling out a short form at http://www.webpagetest.org/getkey.php. After submitting, you’ll be emailed a key. Save this for the next part.

Using the public test agents

If the goal is to simply get test results from an arbitrary test agent, any of the Dulles browsers will do as they’re provisioned for a healthy amount traffic — not that we’ll be producing much. To use one of these test agents, we’ll need to create a file at www/settings/locations.ini. Note that the source code is bundled with a locations.ini.sample file for reference.

The contents of this file should be:

[locations]
1=Public_Dulles
default=Public_Dulles

[Public_Dulles]
1=WPT_Dulles_Chrome
label=”Dulles, VA”
group=Public

[WPT_Dulles_Chrome]
browser=Chrome
label=”Dulles, VA – Chrome”
relayServer=”http://www.webpagetest.org/”
relayKey=[YOUR API KEY HERE]
relayLocation=Dulles:Chrome

The relay server and location ensure that we use the public instance and the relay key authorizes the requests. You can skip the overhead of creating a blank file and copy/pasting the configuration above by downloading it as a gist from GitHub and saving it to the www/settings/ directory. Remember to use your own API key.

3. Configure Apache

OSX already includes Apache, which is needed to run the server, and PHP, which is needed to render the pages. This last step is only a matter of flipping switches and linking the two with WebPageTest.

Open the Apache configuration file at /etc/apache2/httpd.conf and update the following settings.

Enable PHP

#LoadModule php5_module libexec/apache2/libphp5.so
By default, PHP is disabled. Enable it by changing this to: LoadModule php5_module libexec/apache2/libphp5.so.

Give Apache read/write permissions

User _www
Group _www

Apache needs file permissions to save the test data. Instead of granting the Apache user (_www) these permissions, just run Apache as yourself. Change this to:
User [YOUR USERNAME HERE]
Group staff

In case you’re not sure, your username is the output of the whoami command.

Point to the server code

DocumentRoot "/foo"
<Directory "/foo">

This sets up the base file path from which requests will be served. By default it is pointing to some local file path (like /foo). So to point localhost to the WebPageTest web server, change this to:
DocumentRoot "[PATH TO WEBPAGETEST'S WWW DIRECTORY]"
<Directory "[PATH TO WEBPAGETEST'S WWW DIRECTORY]">

Note: If you’ve already got Apache configured for something else, instead you may want to point to WebPageTest’s www directory for only a unique port number.

And that should be it! If you start Apache and navigate to localhost in your browser, you should see the WebPageTest home page. Now you’re ready to start the real work of making the UI changes.

You can find out more about how to use WebPageTest to analyze the performance of websites in my new book, Using WebPageTest.

Using WebPageTest is complete!

Posted in Front End Engineering, Open Source, Web Performance, WebPageTest on by Rick Viscomi.

After two years from inception to production, I’m proud to say that the book I’ve coauthored with Andy Davies and Marcel Duran is finally complete.

We owe huge thanks to the production team at O’Reilly for their hard work whipping the book into shape and making this a reality. Most importantly, we honor Pat Meenan for tirelessly maintaining the tool for nearly 10 years. We’d also like to thank Ilya Grigorik, Lara Hogan, and Tim Kadlec for generously allowing us to quote them on the back cover of the book. We are also in debt to Steve Souders, who has done so much for the web performance community and is even responsible for many key WebPageTest features including the filmstrip. As Steve has kindly written in the foreword of the book:

WebPageTest is the leading web performance tool in the world today. It is easy to use, provides the performance metrics that matter, and is pioneering new ways to measure the actual user experience that websites deliver. In 2009’s Even Faster Web Sites, I wrote that WebPageTest “hasn’t gotten the wide adoption it deserves.” Fortunately, that’s no longer true. In fact, now there’s even a book about it! Read on and find out how to get the most out of WebPageTest to help you deliver a web experience that is fast and enjoyable.

It’s our hope that this book will invigorate readers into leveraging the power of WebPageTest to more effectively improve the performance of their sites. To get an ebook or soft cover copy, check out usingwpt.com.

If you’ll be attending the Velocity Conference in Amsterdam this week, Andy Davies and I will be signing free copies of the book. Come say hello!

Multivariate Testing with WebPagetest

Posted in Front End Engineering, Open Source, Web Performance, WebPageTest on by Rick Viscomi.

For your advanced web performance testing needs, WebPagetest’s out-of-the-box functionality can do most of the heavy lifting for you. The scripting interface alone is powerful enough to handle most advanced use cases. For everything else, we have some other strong tools at our disposal, including multi-location and bulk testing. These tools, however, are mutually exclusive. In other words, you can test an array of pages on a single location/browser setup or you could test a single page on an array of location/browser setups. An experimental new feature called multivariate testing (MVT) hopes to change the way you think about bulk testing. Continue reading

Higher Quality Screenshots with WebPagetest

Posted in Front End Engineering, Open Source, Web Performance, WebPageTest on by Rick Viscomi.

WebPagetest default image quality vs pngss=1

The advanced settings section of WebPagetest (WPT) includes a “Capture Video” option, which will record the page load of the tested web page as the user would see it. This makes it possible to provide visual performance metrics, such as visually complete time, visual progress percentage, and the filmstrip/video features on the Visual Comparison page. This is disabled by default, but once set it will be remembered the next time you visit the page. Even without this option, WPT still captures screenshots of the page at the start render, document complete, and fully loaded events.

These screenshots are saved as low-resolution JPEG images, which is usually good enough for most uses. However, sometimes you need to get a better look at what’s going on. Fortunately, WPT has you covered. Continue reading

How I got a job at Google

Posted in Front End Engineering, Google, Open Source, Quora, YouTube on by Rick Viscomi.

This was originally posted to Quora as an answer to the question What are the ways to utilize 6 months to build a skill-set to get into Facebook or Google?.

I graduated about three years ago with a BS in Computer Science and the only thing I wanted to do was work for Google. Before I graduated, I did well on a phone interview and was invited to interview on-site at YouTube for a Software Engineer position. I did the interview, walked out feeling great about my performance, and not too long after I got the dreaded rejection message. It took a long time and a lot of reflection to realize what went wrong. Continue reading

Querying HTTP Archive to Measure jQuery Plugin Reach

Posted in Front End Engineering, Google BigQuery, Open Source, trunk8 on by Rick Viscomi.

trunk8 on Big Query

The Problem

When I wrote my first jQuery plugin trunk8 last year, I was really excited to see its popularity grow, as people starred and forked the GitHub repository, wrote about it in blogs, and played with the demo. But I wanted a better way to visualize the reach of the plugin, so I did what any other data nerd would do; I created a new GitHub project called red dwarf to query the GitHub API for users who starred trunk8 repo and I mapped their positions using the Google Maps API.

I could see how many people simply “like” the GitHub repository, I could read blog posts about how useful people find it, I could monitor my site analytics for traffic to the demo page, and with red dwarf I could even see the geographic concentration of the people who like it. But this still wasn’t enough data! Continue reading

A Beginner’s Guide to trunk8

Posted in Front End Engineering, trunk8 and tagged , , on by Rick Viscomi.

trunk8trunk8 is an intelligent text truncation plugin to jQuery. When applied to a large block of text, trunk8 will cut off just enough text to prevent it from spilling over.

Unlike conventional truncation that just limits the character length of text, trunk8 measures the content area for spill-over and intelligently chooses the text that best fits in the given space.

Here’s how to use it. Continue reading