The Limitations of Lighthouse
Google Lighthouse helps you identify page performance issues. It generates an overall performance “score” to make you feel good (or bad) about your site’s speed. This score can be useful, but has some limitations.
Lighthouse is an automated tool for assessing web page quality. It generates metrics for performance, SEO, accessibility and more. Google has been promoting it as THE way to measure website quality. We’ll focus on the performance part of Lighthouse and the ways Lighthouse scores fail to tell the whole story. Try it on your site with Lighthouse in Chrome Dev Tools.
Lighthouse Is Synthetic
Lighthouse attempts to simulate the 85th percentile user experience. To do this it slows the page load by throttling network and CPU speeds. By default, this throttling is done even when using “Desktop” mode.
On my computer, the Request Metrics login takes .4 seconds to load in Chrome. Lighthouse on the same computer takes more than double that time. Lighthouse does not represent MY page experience. Does it represent the 85th percentile user’s experience? No one can tell, we must trust that Lighthouse has simulated the “common user” appropriately.
Lighthouse can only guess what your common user looks like. Different applications have different user bases. Users of a developer focused business tool have much different behaviors than a consumer focused social application.
Lighthouse Does Not Understand Your Web Page
Lighthouse does not know the purpose of your page. It does not know how important the page is or how it will be used. Because it does not know these things, Lighthouse must make assumptions about what constitutes “good” or “bad” performance. This can lead to scores that are not representative of a real user’s experience.
An example of this is Google’s own product, Gmail. Gmail is a remarkably snappy single page application (SPA). Google has spent a huge amount of time and effort to make it seamless, quick and responsive. However, Gmail takes a fair amount of time to load initially.
Lighthouse does not know any of this. Testing the Gmail inbox results in a terrible Lighthouse score even though the actual user experience is very good:
Lighthouse Is Not Performance Monitoring
A lighthouse performance audit can’t tell you everything. Lighthouse is a synthetic test and must guess what your website’s average user looks like. It can’t tell you which pages are slow for your real production users or what kind of experience they are having. It can’t tell you which pages are the most important or the most trafficked by your users.
Desktop to mobile ratio varies wildly by use case. Lighthouse can’t tell you whether to focus on desktop or mobile performance. There’s not much use optimizing for mobile on a desktop heavy app and vis-a-versa for mobile focused sites.
Ultimately, we don’t care what score Lighthouse gives us, we care how things are going for our production users.
Conclusion
The best way to know which pages are slow is to monitor your actual users’ performance. It’s easy to do this with a website performance monitoring service like Request Metrics. Request Metrics monitors the page performance of your live production users in real time. It tells you which pages are slow for actual users.
Once you know a page is slow, Lighthouse is a great tool for investigating pages with known performance issues.
