Synthetic Testing and Real User Monitoring

Synthetic Testing and Real User Monitoring are the most important tools in your performance toolbox. But they do different things and are useful at different times and many developers only spend time mastering one of these tools and only see a part of their performance problems, like trying to hammer in a screw.

Let’s look at these tools, what they measure, and when to use them.

Synthetic Testing

Synthetic Testing measures the performance of a website under a controlled environment. Examples of this are Lighthouse audits from Chrome Devtools or Pagespeed Insights. The test simulates the location, latency, bandwidth, browser, and device in order to approximate the experience of a visitor to your website.

For a Synthetic Test to be accurate and valuable, you need to know things about your likely visitors: where they are, what kind of network they are on, and what device they are using. Then the test needs to accurately simulate these characteristics. Both of these things are difficult.

The internet is a big and diverse place, and developers don’t always know enough about our users. We can make guesses, but because we often run on fast networks with new laptops, we often overestimate the capability of our users because. It’s fast on my machine.

Plus, you likely have more than one kind of user that should be tested. Some of your users will visit from laptops at work. Others will try and login on a phone from the train while commuting, or on their tablet with flaky wi-fi from the coffee shop. Each user will have a different perspective on performance, and would need to be simulated with a different test.

The biggest benefit of a this kind of tool is that you can run a Synthetic Test on your website right now, regardless of whether you have any users. And the results will probably tell you about your biggest performance problems.

The test will be flawed, and that’s okay because it will give you an idea of performance. Synthetic Testing will never tell you how fast your website really is—only how fast it might be right now.

Real User Monitoring

Real User Monitoring is just that: real. Real User Monitoring (or RUM) records the actual performance from users that visited your website. RUM doesn’t guess or simulate a user, it just records the actual performance they experienced.

Real User Monitoring is more accurate than Synthetic Testing, but there is also more noise and more delay.

RUM data will inherently include data from all users, even that guy using a GameBoy to browse your website from Mongolia. You’ll have to apply statistics to the data to understand what it really means—things like medians, percentiles, and distributions. Used correctly, RUM data tells you how your fastest users, typical users, and worst users experience your website.

The biggest limitation of RUM is the delay. RUM can’t tell you how fast your site will be until users start visiting it. You’ll have to release that change and measure the impact to see if your site sped up–or not. Synthetic Testing can make some guesses at the performance early, which help find obvious problems, but to really prove your site is fast, you have to use RUM.

Some folks at Google even looked at this, comparing websites Synthetic performance with RUM. Almost half of sites with perfect Synthetic Tests failed the minimum requirements for Core Web Vital scores! Half! Half of people bragging about their perfect Lighthouse score are being penalized by Google for poor performance.

Signal vs Noise

Synthetic Testing and Real User Monitoring is about Signal vs Noise. Synthetic Tests don’t have much noise—each Lighthouse test you run is a valid measurement of performance for those conditions. Run the test again with the same conditions and there will be very similar results.

But as the Google research showed, there is not a lot of signal in those synthetic results either. That Lighthouse report isn’t how any user will experience your page (unless they are browsing your website from your laptop on your network).

Real User Monitoring is the opposite. Each bit of data you get from RUM is how your website really performed for a visitor. But those visitors can be wildly different. Some will have an awesome experience on your website. Others will think they are still on AOL (read: old really slow internet).

The trick is, which users do you care about? If you are building a site for corporate users in the United States, then it doesn’t matter what the performance is for mobile users in Ukraine. RUM tools like Request Metrics help you filter out noise and aggregate the data to give you a clearer picture of your target user.

Both Synthetic Testing and Real User Monitoring are valuable tools for any developer that wants to build fast websites. Use Synthetic Testing, like Lighthouse, to test your changes before release. It will help you catch obvious mistakes.

And use Real User Monitoring tools like Request Metrics to see if that change really sped things up. You don’t know how fast your website is until your visitors tell you.