How to run a WebPageTest test

Table of contents

In this post I’m going to examine all the individual settings from the homepage of WebPageTest (WPT). You could consider this blog post a prequel to my two other blog posts related to WebPageTest: ‘How to read a WebPageTest Waterfall View chart’ and ‘How to read a WebPageTest Connection View chart’. Both blog posts give you information on how to interpret the results of the tests you run off the back of what I’m about to show you in this post.

Simple testing tab

The first section I’ll cover goes against the WPT ordering of the option tabs. The first tab is ‘Advanced Testing’, but I’m going to introduce the ‘Simple Testing’ tab first just so we cover some basic concepts before diving into the finer details.

The simple testing tab allows you to quickly start testing without the need to delve into the more advanced configuration options.

Most of this is probably pretty self-explanatory, but some of the details will help you understand how WPT works.

URL input field

Short and sweet, this is the URL you want WPT to examine. For the public instance of WPT this URL will need to be publicly accessible on the internet. If you need to test private sites that are on an internal network, or behind a basic HTTP Basic authentication there are options that I will cover later in the post.

Test Configuration

When you select the ‘Simple Testing’ tab almost all the custom configuration options are preselected for you. The dropdown gives you a set of options for you to choose from:

  • Mobile - Slow 3G (400 Kbps download, 400 Kbps upload, 400ms Latency)
  • Mobile - Regular 3G (1.6 Mbps download, 768 Kbps upload, 300ms Latency)
  • Mobile - Fast 3G (1.6 Mbps download, 768 Kbps upload, 150ms Latency)
  • Mobile - 4G (9 Mbps download, 9 Mbps upload, 170ms Latency)
  • Desktop (5 Mbps download, 5 Mbps upload, 28ms Latency)

The mobile devices available are a Motorola G4, and all test using the default Chrome browser. The desktop device is a virtual machine (VM) running Chrome. The Moto G4’s are real devices all currently located in Pat Meenan’s basement in Dulles, Virginia (not a joke, 100% true).

As for the different connection speeds available and how that works, a whole separate blog post could be written about this topic. Pat describes the setup in the forum here, and in the WPT documentation here. These resources are a few years old now so a more up to date TL;DR; is that most phones reverse-tether their networking through a Raspberry Pi where netem is used for traffic shaping. Quoting from Pat in the forum post below:

FWIW, we did a LOT of testing at Google with actual networks and dummynet. With dummynet we could simulate a given set of conditions just about exactly (with more consistency) but it’s worth noting that it’s exactly that - one set of conditions. In the case of the default 3G profile I have set up right now, it’s basically modeled around the 60-70th percentile of measured connections globally so it doesn’t match an exact market - some will be faster and some will be slower.

The main things that you can’t duplicate with traffic shaping are:

Radio start-up delay - given a few seconds of idle, most phones will shift the radio into low power modes and it can take anywhere from a few dozen ms to several seconds before the radio comes back and is able to transmit data. It’s more important for multi-page navigations where the radio can turn off between each step but it could also add completely dead time to the beginning of a page load (but that case is completely out of the control of the site owner). In really extreme cases it’s possible for the radio to turn off while waiting for a server response - but if your TTFB is that long you have serious issues to look into anyway.

Network architecture - on traffic-shaped WiFi you will still be going out through your normal ISP’s network (Verizon Fios in the case of Dulles). You won’t go through any carrier proxies, deal with them routing the traffic back to a central egress point or coming from a Carrier’s mobile IP block in case there is any server logic tied to it.

The agents themselves work perfectly fine on actual networks and the test results come back over the wired connection from the tethered host that is driving the agents so the data use is just for the actual browsing and isn’t really why I chose to use WiFi. The main issue is the insane variability using an actual carrier network. The performance varies quite significantly from day to day and even within a day which makes it REALLY hard to compare results to each other.

Old WPT documentation from Pat can be found in this link here.

The traffic shaping models come from real world connection data. For example the 3G profile is based around the 60-70th percentile of measured connections globally in 2014. Latencies haven’t changed much since then so they are still relevant today. It’s worth noting that these are approximations. Some connections real users experience will be quicker, others will be slower.

The whole purpose of this dropdown is to give you an idea of what a user with the selected device and connection speed will experience when loading the URL you want to test.

Include Repeat View

When WPT first loads your URL, the browser has what is known as a ‘cold cache’. All assets that are needed must be downloaded. By enabling this checkbox the test will run twice on the defined page (browser closing between test runs). This is a great way to test the caching strategy for the page being tested, as the second run will be using a ‘warm cache’. So in theory if the page has an efficient caching strategy configured for the page assets, very little will need to be downloaded on the second run.

Run Lighthouse Audit

Again, quite self-explanatory. As well as running a WPT audit across the URL, a separate audit will be completed using Lighthouse. This separate audit will use ‘3G Fast’ as the connection speed (as this is the default in Lighthouse tests), and any connection settings selected via the dropdown will be ignored. This Lighthouse report will be run on an emulated Moto G4 with 4x simulated CPU slowdown.

Running a test

Now we’ve gone over what everything in the simple testing tab is, it’s time to run a test. Simply enter the URL to be tested, select your options and press the big yellow ‘Start Test’ button to the right.

Enter the test details into the tab and click the 'start test' button to the right to kick off the testing.

The next screen you will see will look like below. Giving you details about the URL being tested, the test instance being used, number of test runs (more on this later), and the amount of time the test has been running.

Test started screen with the distinctive arrows telling you which stage of the test is currently running.

Now the time it will take to complete the test depends on the test settings selected. So for example, a ‘Slow 3G’ setting is going to take much longer to load a page than ‘4G’. But don’t worry, you don’t have to keep the page open while the test runs. Tests run completely independently of your web browser. You can return to the test later via the same test URL (or the ‘Test History’ link in the main navigation).

Note: At this point I highly recommend signing up for a WPT account. If you plan on doing a lot of testing WPT will save all of your tests so you can refer to them again at a later date should you need to.

Once the testing is complete you will see a result that looks similar to below:

Test completed page with lots of stats and information to dig through.

The returned test results are immutable and can’t be modified once saved (or even deleted). If you scroll further down the page you’ll be presented with a huge amount of interesting web performance data for you to dig through. The information on this page is beyond the scope of this article, but if you are interested in knowing more about the resulting waterfall charts, I have a blog post all about them here. For the similar looking but different connection view chart, I have a blog post here you can read.

Advanced Testing tab

So now we’ve covered some of the core concepts of how to run a simple WPT test, let’s get stuck into the more advanced (and interesting) settings you can modify. The bulk of the rest of this article will be spent in this section.

The advanced tab gives us a lot more options play with, including granular control over the location and browser used in the test.

Again there are some self-explanatory fields, but I will cover them here for completeness.

URL input field

This is identical to the ‘Simple Testing’ explanation given above.

Test Location

This is one of the major differences from the ‘Simple Testing’ setup. Under ‘Advanced Testing’ you are given the options of the type of device you wish to use for testing, and geographically where in the world the test is run from. So for example you can choose from a set of real (not simulated) Android or iOS devices (all located in Dulles, Virginia). You are also given the option to choose from many test locations dotted across the globe. There are way too many to list here, but you can find detailed information about each location on the Location Status page. This page is also a great way to spot the busy locations and avoid them should you want your test be added to a shorter queue.

Now you may be asking why test location is important. Well depending on your users location in relation to your origin server (assuming a CDN isn’t used), this latency will have a big difference on the overall page performance. There’s actually a whole chapter dedicated to this topic in High Performance Browser Networking by Ilya Grigorik. So which location should you choose? Well if you have any analytics data available for the site you want to test, it would be worth checking where the majority of your users are browsing from. If 90% of your users are coming from a particular country, then it would make sense to choose a test location in (or as close to) the country as possible. This will give you a more accurate representation of how most of your users experience your website.

Browser

The next setting to choose is the browser you want to test your site with. Now these will change depending on the device / location you choose in the ‘Test Location’ dropdown above it. This is another setting that can be driven by your user data if you know the most popular browser visiting your site. Or if you are more interested in how different browsers handle specific server / page setups, this is a great way to directly compare them using the huge amount of data returned from each of the tests. Most of the time you may only be interested in how your site performs on a Chromium based browser (since they are pretty popular!). If so, go right ahead and choose Chrome.

Advanced Settings dropdown

Beyond this point is where the number of configuration options ramps up rapidly. Each of the tabs beneath this dropdown has its own set of options to configure independently.

Test Settings tab

This tab contains most of the ‘high-level’ settings for the test. Some of these will look familiar to areas covered above in the ‘Simple Testing’ tab.

Connection

Pretty obvious really: what connection speed do you wish to replicate when the test is being run. I covered a fair amount of how this works in the ‘Test Configuration’ section above. But this time there are many more options available for you to choose from. A full list of options as well as the bandwidth and latency values available can be found here.

It’s worth noting that you should always select some form of bandwidth shaping from the options. ‘Native Connection (No traffic shaping)’ is an option, but the results returned won’t really be comparable from test run to test run over time. It’s likely to be an incredible quick connection, so not representative of most of your users. Also, as the amount of traffic changes from hour to hour, it’s likely that the true download / upload will differ depending on how busy the network is at that point time. By selecting some form of traffic shaping you can be sure that this variable is consistent across test runs.

Number of Tests to Run

When you enter a URL for WPT to test it doesn’t only test the URL once, it will run the same test multiple times. This is because network latency and traffic can have a huge effect on how a website performs, and these variables are constantly changing. In order to have a more accurate (and representative) result, the test should be run multiple times. For example, should you run a single test run that has very little traffic across the network at that point in time, the resulting performance data will come back very positive. Whereas in reality the network isn’t always like this. So by repeating the test WPT is sampling these variables at different points in time and looking for a happy median.

Note that between each standard test run no cache is shared (unless ‘repeat view’ is selected), so each time WPT visits the page it is starting from a ‘cold cache’. On the public instance of WPT you can run a maximum of 9 test runs. If you set up your own private instance you can run as many runs as you choose. You should always select an odd number of test runs due to the fact that WPT will select the median result once the test has completed. An odd number of runs allows WPT to easily select the run that sits in the middle in terms of performance.

Repeat View

This was covered in the ‘Include Repeat View’ above. But it’s worth mentioning that if you select ‘repeat view’, this will be doubling the number of tests run against that particular page. For example, say you have 9 runs set: each run tests with a first view (cold cache), then again with a repeat view (warm cache). So the page will be visited 18 times. The median run for each first view and repeat view is then highlighted in the ‘Performance Results’.

What metric is used to select the median run? By default WPT will use the ‘loadTime’ metric, but you can also use the ‘SpeedIndex’ metric to perform this comparison too. Simply append ?medianMetric=SpeedIndex to your test result URL and you will see the ‘Performance Results’ summary change. In the examples below, the top image is using ‘SpeedIndex’, the bottom image is using LoadTime. Notice how the run selected for the ‘repeat view’ median changes from 3 to 2.

Performance results summary using the speed index to select the median run.
Performance results summary using the load time to select the median run.

Capture Video

Very simple: do you want to capture a video of the page loading under the conditions you have selected?

Why is this useful? Well, a really important aspect of web performance is the perceived performance a user experiences. If a site loads completely in 8 seconds, but it goes from a blank screen to fully rendered over the space of a single frame, a users perceived performance isn’t going to be as good as if the site loads more incrementally over the same time period. If you can get the first pixels rendered to the screen within the first couple of seconds or so, it gives a user something to look at. The user perceives that something is happening during this loading period, giving them the perception that the page is loading quicker.

The ‘Capture video’ setting allows you to examine the page load frame by frame and also exactly how the browser is rendering the page. This is incredibly useful when you come to comparing results. WPT has the ability to render multiple video’s side by side from different tests, thus allowing you to demonstrate exactly what difference your changes to the site made (both good and bad!).

Label

The last input field for this tab is a free text area allowing you to give a particular test a user friendly label. Should you wish to compare tests before and after a particular set of changes, this is a good way to remind yourself at a later date. Note that this is an optional field and it can be added (or modified) once the test has completed. This is very useful when it comes to comparing tests via the ‘Filmstrip view’, as can be seen in this section here

Advanced tab

This tab gives us a whole bunch of exciting advanced settings to play with.

Image of the advanced tab with all the settings displayed.

I’ll go over each of them individually.

Stop Test at Document Complete

When a WPT run completes it will usually wait for the ‘Document Complete’ event from the browser to fire and all network activity over the network to go quiet for 2 seconds. At this point WPT considers the page to have completely loaded. But under certain conditions this may not always happen. For example, old versions of Internet Explorer didn’t fire the onload event when it was still waiting for something to download. So the test would simply time out after 2 minutes waiting for this to happen. To avoid this you can check the ‘Stop Test at Document Complete’ checkbox. This will force the test to finish once the document is complete (onload event has fired). This point is represented by a blue vertical line on the WPT waterfall chart.

Disable JavaScript

Very easy option to explain. Simply disable JavaScript in the browser that is running the test. Not to get too preachy, but this is a great way to test what your users will see when the JavaScript for a site fails in some way (e.g. network error, blocked, malformed code etc). Remember the use of the Progressive Enhancement methodology is a great way to give users in this situation a simplified experience (not just a blank page).

Clear SSL Certificate Caches

This is a setting that is useful when the browser is being run on a Windows based instance. Windows by default doesn’t clear certificates. When intermediate certificates already exist in the cache this can prompt a browser to hide certificate checks (Chrome with EV certs, and Internet Explorer (IE)). Around the end of 2012 WPT changed the default behaviour for clearing the OS certificate caches which caused SSL negotiation times to increase and skewed the results (since it was clearing all the certificates including root providers). In this case IE was having to validate root certificates too, which takes time. Since there’s no way to clear only leaf certificates, the option to clear all SSL certificate caches was moved to the advanced options to be run on a per-test basis and disabled by default. This isn’t an issue for linux instances because there is no certificate caching. I have a whole post about the impact of SSL certificate revocation on web performance should you be interested. Quoting what Pat said in the forum post below:

With the 2.10 release I changed the default behavior for clearing the OS certificate caches.

Somewhere around 6 months ago I started clearing the OS certificate caches which cased SSL negotiation times to go up significantly for the IE agents that were running on Windows Vista or later (IE on XP does not do CRL/OCSP checks).

There was quite a bit of push-back because it was clearing all of the certificates, including the certs from the root providers so IE was validating things like verisign’s root cert which is something that will hardly ever happen to end users.

The real world lies somewhere in between but I don’t have a way to just clear the leaf certificates and it was skewing site’s performance enough that it was hiding other issues so I made it an option on a per-test basis and it is disabled by default (it’s available in the advanced settings if you want to re-enable it for a given test).

Ignore SSL Certificate Errors

Sometimes a browser will show a SSL certificate error if the certificate chain is broken, or a certificate simply doesn’t exist. Possibly caused by differences in the root certificate lists or other browser-specific policies. Another reason could be if you are running a private WPT instance to test a domain that doesn’t have correct SSL certificates configured. In these instances you can check the ‘Ignore SSL Certificate Errors’ checkbox to bypass this issue. Note: This doesn’t work in Firefox as Firefox has disabled any way to ignore these types of errors for security reasons. So it may only be a setting that works in IE and Chrome.

The quote from the forum post is below:

Firefox disabled any way to ignore cert errors as a security measure so it no longer works for Firefox (at this point it may be Chrome and IE-only). The legacy Windows agent used to force it by intercepting the API calls internal to Firefox but that is no longer possible with the cross-platform agent.

The only way to do it now for Firefox is to run a private instance of WPT and install the certs (or a custom root) on the test machines so they see the self-signed certs as valid.

Disable Compatibility View (IE Only)

There’s a lot of history behind this setting. For web developers who worked on sites through this time (myself included) it will probably bring back frustrating memories. A TL;DR; is basically in order for newer versions of IE to not ‘break the web’, the browser implemented a feature that allowed it to mimic older versions (e.g. make IE8 render like IE7 did). This was needed because of certain IE specific coding practices that existed and were very popular. This setting forces IE to load in ‘Strict mode’ no matter what the DOCTYPE or http-equiv="X-UA-Compatible" meta tag says. (Thankfully) it is very unlikely you will need this setting on the modern web.

Capture network packet trace (tcpdump)

Although WPT can give you a lot of information about the order in which a page loaded as well as all the intricacies involved in this process, sometimes you need to dig a little deeper into the low level details. This is where capturing a network packet trace comes in extremely useful. To do this WPT uses a command-line tool called tcpdump. What this tool does is allow us to display TCP/IP and other packets being transmitted or received over the network for the instance that is running the test. If checked WPT will save what is known as a Packet CAPture (PCAP) file that can be downloaded on the test result page:

Once selected you get the option to download the PCAP file in the results.

The great thing about packet level capture when done via a WPT instance is the isolation. The only network traffic that is being captured is solely related to the page being tested. For example, if you were to run a capture on your local machine you’d most likely have to filter so much data due to all the other processes running on the machine in the background.

So what can you do with this PCAP file once you have it? Well, you can use tcpdump -r [capture_filename] to read in the actual file to your local version of tcpdump and examine the internals using the CLI. Or if you are after a GUI interface then it’s hard to beat Wireshark. If you are decoding HTTPS traffic you will also need the TLS Key Log file. Follow the instructions under the ‘Load your data in Wireshark’ header from this page on how to decode secure traffic. If you are looking to capture traffic from iOS devices then you are going to need to go through quite a few other steps that Andy Davies covers in his ‘Capturing and Decrypting HTTPS Traffic From iOS Apps Using Frida’ blog post.

A really quick way to open Wireshark and not worry about all the settings, grab the PCAP and the TLS keylog file and run the following command:

wireshark -r your-pcap-file.pcap -o tls:keylog_file:your-keylog-file.keylog

Word of warning, Wireshark is an incredibly powerful (and complex) bit of software with a steep learning curve associated with it. But once mastered, it unlocks a whole new world of data for you to examine.

Save response bodies

By default when a WPT test runs the response bodies of the requests are discarded once downloaded. This is fine for most cases, but what about if you wanted to store these bodies for future reference. Say you were making a change to a pages <head> tag in order to improve performance. Wouldn’t it be great to be able to view that HTML in the future once you knew which version performed best. This is exactly what this option does. You could almost consider it ‘view source’ functionality for WPT tests.

It’s worth nothing that WPT only does this with text based responses (HTML, CSS, JS etc). Images and other binary formats aren’t stored in this process. There currently isn’t an option to enable this functionality, even when using your own private instance, but Pat has mentioned that it wouldn’t be too hard to add if there were more requests for it in the future.

Preserve original User Agent string

One issue running WPT against a live site is that it will make impressions on your analytics data. Since to an analytics tool a visit from a WPT browser looks like any other browser visiting the site. For that reason WPT appends a PTST string to the User-Agent (UA) string. A random example of this is given below:

user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 PTST/201209.031210

Using the ‘PTST’ string you can easily filter out any data with it in the user-agent header. Now you may be asking why PTST? What does it stand for? Well, it actually stand for PageTeST, which was the name of the IE plugin version of WPT before there was a web interface in the version we see today.

User Agent String

This setting allows you to completely override the UA string should you wish to do so. For example if I enter ‘my-custom-ua-string’ into the input box, the header for every request will be:

User-Agent: my-custom-ua-string PTST/201223.161206

Now if you wanted a completely clean UA string without the PTST information you’d simply check the ‘Preserve original User Agent string’ checkbox as well. That will give you the following output:

User-Agent: my-custom-ua-string

This functionality isn’t supported by all browsers. For example it doesn’t work in IE11 and Edge. It does work in Chrome (including mobile device versions), Firefox, and Safari on iOS.

Append to UA String

This input box allows you to append a custom string to the end of the UA string in the file requests from the chosen test browser. This advanced functionality was added off the back of a request by Rick Viscomi that can be seen here. Additional functionality was added in this request that allows the test ID (and other details) to be appended to the UA string when you run a test via the API. This allows you to extract the test URL from your logs in the logging software you use. See this thread for more details on how to do this.

Capture Full Size Video

By default WPT will capture a video of the page load (assuming the ‘Capture Video’ checkbox is checked in the ‘Test Settings’ tab). But it won’t capture the full size video. By default it captures a video which is scaled down by approximately 50%. In my simple test using Chrome:

  • Default: 408px x 360px
  • Full: 904px x 680px

The dimensions of the video captured depend on what device the test is running on. For example, running a test on a mobile device will produce a video with mobile phone dimensions.

This setting is extremely useful if you are looking to examine the finer visual details of how a page is loading, or if you are wanting to show the video in a presentation.

What this setting also unlocks is the full size images in the ‘filmstrip view’. Selecting ‘Huge’ will give you a filmstrip full of images where the largest side is 600px. You can further increase their size to the full native width if you modify the thumbSize query parameter in the URL for the test.

Minimum test duration

As mentioned earlier in the ‘Stop Test at Document Complete’ explanation, by default WPT will stop a test once the ‘Document Complete’ event has fired and the network has gone quiet for 2 seconds. But there may be some instances where you want to force a test to run for a minimum number of seconds. A use case where I’ve used this setting is in testing the web performance of a 404 page. Without this setting WPT would see the 404 status code and immediately stop the test. Another example where you may use this setting is for long running XHR requests. If they take longer than 2 seconds the response will never be captured.

Custom headers

This textarea allows you to add custom headers to all requests made by the browser to the server. This can be useful when it comes to various debugging scenarios. You add each custom header on a newline:

Add custom headers to every request that the browser makes to the server. Make sure to add each on a newline.

For example these request headers could be used to prevent the test agent from being blocked by a server, or another way to exclude tests from your analytics data.

Inject Script

The inject script textarea is an incredibly useful tool when it comes to debugging. When the test runs the script will be injected into the page once the document exists. This will be as near to the opening <head> tag as it can be. But the exact position in the source usually depends on what browser is being used.

Some example use cases for this setting include:

Quote from Andy Davies about cookie banner testing below:

I tend to work out what cookie or local storage entry the overlay is checking for and then add it via a WPT script, or inject JS

It’s alway good practice to wrap your custom script in an Immediately Invoked Function Expression (IIFE), that way you know your variable & function names won’t clash with what is already set on the page. Here’s a very simple example of a script you could add to the textarea:

(function(){
  // this will trigger a font load
  var customFont1 = new FontFace('custom font name', 'url([FONT_URL_HERE])', {
    // other font properties here
  });

  // add the font to the document
  document.fonts.add(customFont1);

  // monitor the font load (optional)
  customFont1.loaded.then((fontFace) => {
    // log some info in the console
    console.info('Font status:', customFont1.status); // optional
    // log some info on the WPT waterfall chart
    performance.mark('wpt.customFont1Loaded'); // optional
  }, (fontFace) => {
    // if there's an error, tell us about it
    console.error('Font status:', customFont1.status); // optional
  });
})();

Here we are loading a new font into the page programmatically using the CSS Font Loading API. We are adding a performance mark to the WPT waterfall chart so we know the exact point at which the font is loaded. There are so many uses for the ‘Inject Script’ functionality, I’m sure there could be a whole blog post dedicated to it.

Important note: If you are looking to inject a script while using WPT scripting (‘Script’ tab, covered later), you will need to use a different method. Either exec or execAndWait. Documentation for these can be found here.

That brings us to the end of the ‘Advanced’ tab. Next on our list is the ‘Chromium’ tab.

Chromium tab

The ‘Chromium’ tab has a whole bunch of settings specific to the Chromium based browsers (Chrome, Edge, Opera, Brave, Samsung, Vivaldi).

Tab dedicated to chromium settings.

Capture Lighthouse Report

This was covered above in the ‘Simple Testing’ section above under ‘Run Lighthouse Audit’.

Emulate Mobile Browser

This setting isn’t just emulating the viewport settings of the device selected in the dropdown. Behind the scenes there is a reference VM that tests the performance of all the real devices in the device lab. WPT uses this data to throttle the CPU to closely match the performance of the real devices. If you examine the device options in this dropdown you will see it closely matches the real devices available in the lab. The setup for this emulation can be seen in the code here.

It’s worth noting that the results from using this setting are still estimates of the performance compared to a real device. A real device will have different RAM available, and the CPU architecture will be different. Remember that you will still need to throttle the network connetion via the ‘Test Settings’ tab (covered earlier).

Capture Dev Tools Timeline

If you are running the Chrome browser this setting allows you to download the full DevTools timeline from the browser running the test. This file includes network, layout, and JavaScript parsing / executing data. The file can be opened via your local version of Chrome DevTools, or you can open it in the online viewer using the ‘(view)’ link:

The timeline link highlighted is available on the test summary page and leads to more detailed information as to what the browser main thread is doing.

This data gives you an incredible amount of detail about what is happening on the browsers main thread. If you are interested in finding out more about this functionality I’ve written about it in a bit more detail here.

Capture V8 Runtime Call Stats

I’m replicating the order listed in the ‘Chromium’ tab, but this setting makes a lot more sense if you read ‘Capture Chrome Trace’ and ‘Trace Categories’ first. This setting is effectively a shortcut to enable the ‘Capture Chrome Trace’ setting and only select the v8 and disabled-by-default-v8.runtime_stats categories. This setting is mainly used by the Chrome browser development team to easily collect data on where the v8 engine is spending its time. There are many advantages to collecting this data in a WPT instance: ‘clean’ data with only a single tab running and no extensions, easily repeatable results, bulk testing thousands of pages using the API, and collection of data from a set of real devices.

Capture Chrome Trace (about://tracing)

This setting is a reference to the chrome://tracing/ tab that you can access in Chrome. It gives you access to an incredible amount of underlying data not only from a page load perspective, but also the underlying performance of the Chrome browser itself at a system level. Once enabled you can access this data via the following links:

The trace link highlighted is available on the test summary page and leads to more detailed information as to what the browser main thread is doing.

Note: A word of warning about enabling this option. If you are only interested in the web performance of a web page, DON’T enable this option. Some categories can add a lot of overhead to the tests, which in turn can result in the skewing of overall timings. This setting is primarily targeted at browser engineers, not web developers. For more information about this setting and the data visualisations available using it I have written about them more here.

Trace Categories

This setting is used in conjunction with the ‘Capture Chrome Trace’ setting above, and is only relevant when the checkbox is checked. The amount of data returned when capturing a Chrome trace is huge. Now you may only be interested in one particular type of data. This input box allows you to select only the data you are interested in (e.g. for blink data, delete all other categories listed). Again as mentioned above, you only really need to be concerned with this setting if you are a browser developer.

Capture Network Log

This setting is similar to the ‘Capture V8 Runtime Call Stats’ mentioned earlier. It is effectively a shortcut to enable tracing and select only the netlog category in the ‘Trace Categories’ input field. Enabling this also passes the command-line flag to store the non-trace version of the netlog too: chrome://net-internals generates a JSON file that can be used for debugging, and the trace-version is layered on top of this. Enabling this option makes WPT capture both netlog versions.

Enable Data Reduction

This setting is a browser option that users can choose to enable on their devices. For example, if a user is on a limited data plan they may prefer to view what may be considered a ‘cutdown’ version of a website. A site with fewer images, different image compression, slightly different (and more data efficient) page design.

In practical terms WPT will append the save-data: on header to all of the requests sent to the server. More info about this header can be found on the Save-Data MDN page, and I’ve written a blog post about its usage here. I also highly recommend this talk by Tim Vereecke from London Web Performance, November 2019 - Data-s(h)aver strategies. It has a whole heap of information about this header and how you could be using it.

Host Resolver Rules

This setting is basically just like a hosts file only with wildcard(*) support. It’s also what WPT uses for Chrome when the setDns WPT script command is used. So for example entering MAP * 127.0.0.1 into the input field forces all hostnames to be mapped to 127.0.0.1 (you’re not going to get much back if you use this exact command!).

An example use case for this setting is if you are testing your live website, but you wanted to map some requests back to your development server instead. Very useful for some areas of testing and debugging.

Command-line

Chrome comes with a whole bunch of options you can enable & disable via the command-line when you run it. Some change behaviour of features, others are for debugging or experimenting. You can enter these command-line options into this input box and they will be passed through to Chrome in the test.

A specific use case I’ve used this for is to disable HTTP/2 using the --disable-http2 flag. So if you want to compare the performance of HTTP/2 vs HTTP/1.1 for the same site using WPT, simply pass in this setting and it will force the use of the HTTP/1.1 protocol. You can then compare the results of HTTP/1.1 vs HTTP/2 using WPT.

Auth tab

This is a much simpler and self explanatory tab compared to the previous tabs. This tab allows WPT to access a website that is sitting behind HTTP Basic authentication (BA). Useful if you wish to test a site that isn’t publicly accessible.

Allow WPT to access a website behind HTTP authentication. Image shows input fields for a username and password.

As the warning says in the tab, make sure you use a test account when using this functionality! You don’t want to be sharing an account with higher privileges with anyone by them viewing the test results. Note that when in use the test won’t appear in the ‘Test History’ tab in the main navigation because it is automatically set to private.

Script tab

The script tab allows you to automate testing via the use of the WPT scripting language.

The script tab holds a textarea that allows you to enter scripts using the WPT scripting language.

The documentation for the script commands can be found here. Possible use cases for these scripts include:

  • Following a user journey across multiple pages
  • Acceptance of cookie banners and the setting off cookies
  • Login to a member area for performance testing
  • Test a sites performance with third-parties blocked
  • Performance testing Single Page Applications (SPAs)

An example of a very basic script looks like this:

setCookie https://www.gov.uk/ cookies_policy={"essential":true,"settings":true,"usage":true,"campaigns":true}
setCookie https://www.gov.uk/ cookies_preferences_set=true
setCookie https://www.gov.uk/ global_bar_seen={"count":999,"version":8}
navigate https://www.gov.uk/

This script sets three cookies that will hide the cookie banner on GOV.UK then navigate to the homepage. This example can be useful if you want a clearer picture of the Cumulative Layout Shift (CLS) seen on a page (since cookie banners can cause a high CLS value on page load).

Script includes sensitive data

There may be times where you need to include sensitive data in your test script (e.g. login cookie data, personal information etc). If you share these results with other people you may not want them to be able to see this data. In this case select this checkbox and the cookie headers and WPT script will be discarded in the results and no longer viewable (see images below):

Checkbox unchecked

With the checkbox unchecked the original script is visible in the test result.

Checkbox checked

With the checkbox checked the original script is removed from the test result.

Discard all HTTP headers

When enabled, all of the request headers will be removed from the reporting in the test results. So clicking on a request and navigating to the ‘Request’ tab, the corresponding panel will be empty.

Block tab

The block tab does “exactly what it says on the tin”, it blocks requests and domains listed in the two separate textareas.

The block tab allows you to block certain requests, allowing you to see the the difference these requests can have on performance.

These textareas take a space separated list of strings. These blocks cause what is known as a ‘fast fail’. The browser will try to grab the resource, fail, then move on.

Some use cases for blocking include:

  • Testing to see how fast a site would be without adverts
  • Testing to see how fast a site would be without an A/B testing library

For this you would run a standard test, then run a second test with said request(s) blocked, then compare the two versions.

Block Requests Containing

This textarea allows you to block individual requests by doing a substring (case-sensitive) match. An example of what you could paste in the textarea and what it blocks is seen below:

// block all requests containing 'spacer.gif', 'ckeditor', 'texedit', and 'www.example-domain.co.uk'
spacer.gif ckeditor texedit www.example-domain.co.uk

Note that the blocked requests don’t appear in the resulting waterfall at all, but you can see they were blocked at the top of the test page in the test details:

Blocked URL strings are displayed in the results at the top of the page.

This method works by using the Chrome extension API (when testing in Chrome). An extension is used that blocks requests much like a standard adblocker does. Because of this, there can be some time increases due to the overhead of the extension API. When it comes to blocking domains (like we have with www.example-domain.co.uk in the example above), you are better to use the ‘Block Domains’ setting detailed next.

Block Domains

Very similar to the ‘Block Requests’ functionality, only this time it accepts a space separated list of hostnames (full, not wildcards). An example of what you could paste in the textarea and what it blocks is seen below:

// block Google Fonts, Google Analytics and Google Tag Manager
fonts.googleapis.com www.google-analytics.com www.googletagmanager.com

The great thing about the block domains functionality is that it runs at the DNS level rather than via the extension API (in Chrome). So it has less overhead compared to matching URL substrings. But for that reason it only works with full URL strings.

Note: It is possible to use both blocking functions via WPT scripting using the block and the blockDomains commands. It’s also possible to do the inverse with the blockDomainsExcept command where you block all URL’s except the one listed. Andy Davies has a great article on ‘Measuring the Impact of 3rd-Party Tags With WebPageTest’ where he uses this technique to great effect.

SPOF tab

SPOF stands for Single Point Of Failure. The textarea accepts a list of hosts that you want to silently fail (one host per line).

The Single Point of Failure tab accepts a single host per line that should silently fail. Paste them into the textarea given.

Now you may be asking “What’s the difference between ‘Block’ and ‘SPOF’?”. As mentioned before when you block a URL substring or domain it will ‘fail fast’. The browser will know almost instantly and move on with the page load. SPOF on the other hand will redirect the request to blackhole.webpagetest.org and then silently fail. This is essentially leaving the browser hanging waiting for a response.

A great use case for this is to establish what happens when one of your third-party dependencies fails (e.g. an ad-network, a/b testing tool, JavaScript CDN). Do you really want your site to slow to a crawl because a third-parties infrastructure (outside of your control) fails? This can happen even to the largest tech companies in the world as Google recently showed. So it’s a good idea to test if you have a hard dependency with a third-party now, rather than wait for the failure.

Custom tab

The custom tab allows you to define custom JavaScript that will run at the end of a test to collect custom metrics (that will be returned in the results).

The custom tab consists of a single textarea for inputting JavaScript that will return custom metrics.

Custom Metrics

This is where you paste in the name of the metric being captured along with the JavaScript to be executed at the end of the test (remember to return a value from the JavaScript!). An extremely simple example of this in action can be seen below:

[num-total-scripts]
return document.querySelectorAll('script').length;

This example, taken from Tim Kadlec’s excellent ‘WebPageTest Custom Metrics with Request Data’ article simply counts the number of script tags on the page and returns them via a custom metric. The results look like this:

Custom metrics can be viewed inside an individual test run data.

Custom metrics data is also added to the JSON data that you can query via the API. More complex examples can be seen in the WPT documentation related to Custom Metrics.

Hidden settings

WPT comes with a number of hidden settings that aren’t displayed in the UI by default. To enable these settings you must pass in a query parameter.

Bulk Testing Tab

You can enable this setting by appending ?bulk=1 to the WPT URLNote: You will need an account to use this setting. This will give you a whole new tab under the ‘Advanced Settings’ called ‘Bulk Testing’:

The bulk testing tab is now visible in the UI when appending `?bulk=1`.

In terms of usage it’s pretty self explanatory. This setting adds the ability to test multiple URL’s all using the same test settings via the UI. You can either paste in a list of URL’s into the textarea, or upload a file with a list. Make sure each URL is on it’s own line. For example, if you wanted to test multiple pages on the BBC News website you would paste this:

https://www.bbc.co.uk/news
https://www.bbc.co.uk/news/coronavirus
https://www.bbc.co.uk/news/politics
https://www.bbc.co.uk/news/technology
https://www.bbc.co.uk/news/stories

You can also add labels to individual test pages like so:

Homepage=https://www.bbc.co.uk/news
COVID=https://www.bbc.co.uk/news/coronavirus
//...

Once you’ve submitted the test you can bookmark the page and come back to it later. No need to keep the window open, the tests run independent of the browser window:

Here we see the URL's we want to performance test running.

Once completed WPT gives you a handy table with specific metrics listed, as well as links to many sources of raw data you can examine further:

Here we see the completed results for the 5 URL's we entered into the bulk testing tab.

The page even gives you the ability to compare test URL’s via the checkbox and compare button to the left hand side of the table. Doing so will direct you towards the compare view page which I talk about more in the ‘Visual comparison tab’ section below.

It’s also worth noting that all of these tests are visible from the ‘Test History’ link in the main navigation. They will each be listed separately, and a link to the ‘Bulk Test’ summary table can also be seen on the list. Again, you can compare results from this view as well:

The bulk test results appear on the 'Test History' page too.

Hidden input fields

There are also quite a number of query parameters that won’t display anything in the UI, but they will add a hidden input field to the page source and pull the query value from the query parameter. These aren’t all limited for use on the homepage, many can be used on the results pages too (e.g. ‘result’, ‘details’, ‘compare’).

iq (?iq=)

With this you can specify the JPEG image compression level used for screenshots and video capture. Value range between 30-100.

pngss (?pngss=)

Set to ?pngss=1 to save a full resolution lossless PNG of the fully loaded page being tested.

discard (?discard=)

The discard parameter allows you to discard the first x number of tests, but it is highly likely that this isn’t implemented in the new agents. The value provided is the number of test runs to be discarded.

timeout (?timeout=)

This is the maximum time a test is allowed to run in seconds. The default is 120 seconds.

appendua (?appendua=)

This parameter accepts a string that will be appended to the user agent string. See the ‘Append to UA String’ section for more details about this setting.

keepvideo (?keepvideo=)

This parameter if set to ?keepvideo=1 when you run a test, will instruct WPT to keep the raw video without any additional compression so the frames can be analysed further.

medianMetric (?medianMetric=)

This parameter it used to instruct WPT which metric will be used to decide the median run. By default loadTime is used, but you can also pass in SpeedIndex if you wish to use that. I discussed this setting in the ‘Repeat View’ section earlier.

affinity (?affinity=)

I’ll refer to the API docs for this explanation:

Specify a string that will be used to hash the test to a specific test agent. The tester will be picked by index among the available testers. If the number of testers changes then the tests will be distributed to different machines but if the counts remain consistent then the same string will always run the tests on the same test machine. This can be useful for controlling variability when comparing a given URL over time or different parameters against each other (using the URL as the hash string).

tester (?tester=)

This allows you to specify a specific test machine to run your test on. Note that this value must match a ‘Computer Name’ from the list of test machines found here, else it will never run.

minimal (?minimal=)

In theory this should run a cut down version of the tests optimised for individual run speeds. But at the moment it doesn’t actually do anything. So consider it a work in progress.

noopt (?noopt=)

Enable / disable the optimisation checks as seen under the ‘Performance Review’ section in the test results (e.g. keep-alive, GZip, CDN Usage etc). Enable this by setting ?noopt=1, this will speed up your tests.

debug (?debug=)

This setting keeps the debug output of the agent along with the test results. Mainly used for debugging test instances.

throttle_cpu (?throttle_cpu=)

I touched on the methodology behind CPU throttling for mobile emulation in the ‘Emulate Mobile Browser’ section. The throttle_cpu parameter allows you to throttle the CPU for Chromium based browsers (Chrome, Edge, Opera, Brave, Samsung, Vivaldi). This excludes the physical Android devices. The amount goes from 0 (no throttling) upwards. The highest value in the settings ini is 5.5 for MotoE and AndroidOne emulated devices. This signifies that these devices will have 5.5x slowdown.

browser_width (?browser_width=)

Set the width of the browser window you wish to use in the test. Set using display pixels. Usage: https://www.webpagetest.org/?browser_width=800

browser_height (?browser_height=)

Set the height of the browser window you wish to use in the test. Set using display pixels. Usage: https://www.webpagetest.org/?browser_height=600. Set both height and width by listing query parameters: https://www.webpagetest.org/?browser_width=800&browser_height=600.

width (?width=)

Here we set the width of the actual viewport using CSS pixels. Usage: https://www.webpagetest.org/?width=320.

height (?height=)

Here we set the height of the actual viewport using CSS pixels. Usage: https://www.webpagetest.org/?height=640.

thumbsize (?thumbsize=)

This setting is used on the filmstrip page (compare.php) and it controls the size of the largest side in the in the resulting thumbnails in the filmstrip. For example, if the thumbnails are landscape, this value will refer to the width of the image. For portrait it will refer to the height of the thumbnail. You can enter a custom value, or you can select the preset thumbnail sizes from under the filmstrip. I’ve written a little more about the filmstrip view here.

fps (?fps=)

How many frames per second are captured by the video for desktop devices. This is set to 10fps. Increasing this value adds overhead to the tests.

discard_timeline (?discard_timeline=)

Don’t keep the DevTools traces and timeline data after the test has finished and they have been processed.

htmlbody (?htmlbody=)

Set this value to ?htmlbody=1 if you only want to save the response of the base HTML page. This is a slightly different setting than discussed in the ‘Save response bodies’ section above which saves all the test responses.

disable_video (?disable_video=)

This setting stops ffmpeg from running on the instance (?disable_video=1). Usually, even if video=0 and a video isn’t being captured, ffmpeg still runs.

lighthouseThrottle (?lighthouseThrottle=)

Use Lighthouse’s version of CPU throttling with the Lighthouse audit rather that WPT’s own version.

warmup (?warmup=)

This used to allow for a throw-away test run before collecting data to warm up server caches. This functionality is no longer hooked up.

Visual comparison tab

Now that we’ve gone over a lot of the basic and advanced settings above, explaining this tab is much easier. This tab is essentially a set of shortcuts allowing you to quickly compare two sites with each other using a real device on a network connection speed of your choosing.

Here we are comparing two sites against each other on a mobile using a 4G connection.

Once the test has been started simply sit back and wait while WPT completes 3 test runs across each URL.

Here we see both tests listed, and the stages they are at in terms of the testing process.

Once the tests have completed on all URL’s, you will be shown the standard compare view for each of the sites under the corresponding test conditions:

Visual comparison of the results on the standard compare view page.

This view allows you to visually compare the loading speed and render process for each of the sites. If you scroll down the page you can also compare waterfall charts. You’ll also be presented with 6 graphs with information ranging from ‘Visual Progress’ all the way through to ‘Total Bytes’ loaded by each page (and their corresponding filetype). I’ve written about the compare view in the links below if you wish to know more:

Traceroute tab

This may be one of the least well used of the main tabs available in WPT. The tab allows you to complete an Internet Control Message Protocol (ICMP) trace route to a specific host / IP address from a particular test location. The test can be run multiple times (default is 3), and then you are presented the results via the UI:

Traceroute showing 3 runs from Dulles, VA to CNN.com.

In the example above I’ve chosen a test instance in Dulles, VA and the host edition.cnn.com. The test has been run 3 times, and each test lists 4 hops along with the time to get to each hop, and their corresponding IP address. If there’s no response back you will simply receive a blank page once the tests have completed.

Quote from Pat’s forum post below:

You can now run traceroutes from the test agents through the web UI. It’s still pretty experimental but you can get to it here: http://www.webpagetest.org/traceroute

Here is a sample traceroute to o.aolcdn.com from the Dulles test agent.

When there’s no response you’ll get a blank entry (may update this to display a * so it looks more like the command-line). The traceroutes are done with ICMP for now (eventually I’d like to implement it as a TCP port-80 traceroute but that’s a lot harder to do on Windows than it should be).

This is essentially the same as the traceroute command that you can run in a terminal window. Only instead or running it from your local machine you are running it from the location of the test instance.

A use case for this functionality would be if you were interested in the traffic across the network between the instance and site server at any point in time. If you happen to be seeing unusually poor performance from an instance to the site server, running a traceroute between the two will allow you to evaluate and understand why this may be occurring.

Common Scenarios

In this section I’m going to list a few scenarios you may want to test and the settings you will use. I’ll add more of these over time as I encounter them.

Testing HTTP/2 against HTTP/1.1

Testing a site using both protocols is fairly straightforward using WPT. That is assuming your server already has HTTP/2 enabled. The method for testing relies on the fact that browsers that don’t support a specific protocol will simply ‘fallback’ to a version that they do. This is in order that browsers ‘Don’t break the web’. If a browser were to simply fail on encountering an unsupported protocol, progress on the web would be very difficult!

Step one is to test the site using HTTP/2. Since it is already enabled, this is really just a standard test run. So select your device and connection speeds then run the test. Remember to add a label to the test so you know which version you are looking at later:

Run the first test, label with HTTP/2 and select your settings.

Next on the list is to run the HTTP/1.1 version of the test. Keep all the same settings, you just need one minor adjustment in the ‘Chromium’ tab. In the ‘Command-line’ input enter --disable-http2. Remember to add a label marking it as HTTP/1.1. We then run the 2nd test:

Set up the Chromium tab to disable HTTP/2 in Chrome.

This command-line flag has disabled HTTP/2 in the browser so in order to communicate with the server it is having to fallback to HTTP/1.1. Now that we have both tests we are able to compare the results in WPT and examine which performs better.

Compare the results using WebPageTest by selecting the checkbox and clicking the 'compare' button.

Using the command-line to disable features is a great way to test the difference between new and old features. The challenge often lies in finding the right command-line flag. This page should help you find these.

You could use the same method to test:

  • the difference between TLS versions using --ssl-version-max
  • how a website loads without webfonts using --disable-remote-fonts
  • the difference preconnect makes using --disable-preconnect to disable it (assuming the flag is still available).

Testing a site without third-party scripts blocked

In this scenario we use the WPT scripting language to block third-party scripts. I’ll give a brief overview but you can read a much more detailed version of this technique by reading the blog post ‘Measuring the Impact of 3rd-Party Tags With WebPageTest’ by Andy Davies.

First we will need to run a standard test run. This can be done via the UI or with a script. Here I’m using the UI:

Run the times website through WPT without any modifications.

Next we need to run the same test but with a small tweak. We will use the ‘Script’ tab to write a very simple script that will navigate to ‘https://www.thetimes.co.uk/’ and block all other domains:

In this script we are blocking all domains except for the thetimes.co.uk, then navigating to it.

Once both tests are complete it is time to compare the results:

Compare the two sets of results to see what the impact of those third-parties are.

Now the method for blocking other domains is quite a drastic one, as it may be blocking domains that the site requires (e.g. a CDN or an assets domain). But as a quick test to see the impact these domains have of performance it works really well. The stats from the 2 tests say it all really:

  • Visually complete: 11.2s vs 3.7s (block vs no-block)
  • Fully Loaded: 9.6s vs 40.8s (block vs no-block)
  • Requests: 39 vs 177 (block vs no-block)
  • Bytes: 1.1 MB vs 2.4 MB

The test above gives some quick insight into the impact third-party domains can have on performance.

Useful tips & tricks

In this section I’ll document any useful tips and tricks I run into related to running a WPT test.

How do I see all the WPT advanced settings?

Here’s a little tip from Andy Davies and Simon Hearne. What about if you don’t want to click through all the advanced tabs to find the setting you are looking for? Wouldn’t it be great if you could see them all on the same tab. Well, now you can with this handy little snippet:

document.querySelectorAll('.ui-tabs-hide').forEach(el => el.style.display='block')

Simply paste this into your browser DevTools console and execute it. The resulting page will look like this.

Conclusion

We come to the end of another long blog post related to WebPageTest. In the post I’ve tried to document every setting I could find related to running a WebPageTest test, and also give you an explanation as to what they do and how they can be used. Please consider this a living document, so if you have spotted any errors or I’ve missed anything, please do let me know. Credit will be given in the post changelog. If you would like to learn a lot more about WebPageTest, I highly recommend the book: “Using WebPageTest” by Rick Viscomi, Andy Davies, Marcel Duran.

Finally, another huge thanks go to Pat Meenan who I’ve inundated with random questions while writing this post!


Post changelog:

  • 31/12/20: Initial post published.
  • 31/12/20: Added info about needing an account for bulk testing, and the handy Wireshark snippit for faster examination. Many thanks to Andy Davies for this info!
Loading

Webmentions