Sensai Release Notes: Updated Site Speed Health scores

Summary

• The Site Speed Health Score algorithm was too generous in certain situations.
• We have adjusted the algorithm to better reflect the overall state of the internet today.
• You may see a reduction in your score.
• All of your historical data is updated to use this new algorithm.

Why we made this change

Our mission is to build a faster web.

We can’t do that without a clear definition of "fast" that's attuned to the web today.

We believe that to truly drive forward site speed improvement teams must integrate RUM and Synthetic performance data in one tool. However, these two data sources occasionally deviate: what looks fast on a developer's machine may not result in a fast UX for real users.  

By working with our clients' digital teams through the Sensai Early Access program, we continue to learn which areas of our algorithms and platform need tuning.

This change represents a minor version bump to how we define "fast".  

How we determined new scores

We have tweaked the parameters of our Site Speed Health Score algorithm to better represent a performant experience both in the wild, and in the lab. Some of these changes include:

• While passing the Core Web Vitals assessment is critical to your digital success, there is still room to do better. Therefore we have adjusted the scoring so that passing Core Web Vitals represents 80% Site Speed Health. Previously this was closer to 95%.
• Weights have been adjusted between metrics, with an additional focus on the upcoming Interaction to Next Paint (INP) metric. The relevance of First Input Delay has been reduced in step.
• Some pages get a lot of real user traffic, but no CrUX data is logged (e.g. checkout pages). For this reason we have more closely aligned the Synthetic and RUM curves with real data from the web.

How we correlate RUM and Synthetic performance data

You've surely seen a few too many 100% Lighthouse scores resulting in failing Core Web Vitals. Or sites where some pages just don't get enough traffic to expose real-world CrUX data. For these reasons we've aligned the data from Sensai's synthetic audits with real traffic from a selection of over 3000 websites.

Here's how we calibrate these disparate data sets:

• Start by fetching Google's CrUX data for 3000+ domains under surveillance in Sensai.
• We then run 15,000 synthetic audits against these websites, generating a representative "synthetic state of the web" dataset for a given point in time.
• Next we perform linear regression analysis to calibrate the synthetic dataset to the RUM data from CrUX.
• Finally, we tune a few additional factors educated by millions of rows of Sensai client data, to account for factors inherent in synthetic vs RUM testing (e.g. synthetic tests with more than one page view will be hitting a pre-warmed cache).

What is the effect of the new tuning?

The following shows the same website data from before and after this change came into effect. This site in particular has exceptionally fast Rendering, independent of the other metrics. Furthermore, the increased weight of INP and reduction of FID can be seen in the reduced Interactivity score.

BEFORE‍

AFTER

Additional bugfixes in this release:

• Synthetic FCP lag was inflating Rendering scores through miscalibration. This has been resolved.
• The Dashboard Page Type table is now using the same time range as Gauges. Looking at a year of data will still show trends along with the current state of performance, not a blended average of the year.

That's all for this update. If you have any questions please get in touch. We're happy to help.

‍