Testing Methodology

This page describes, once, how every Image Load Lab experiment is run. Individual guides link here instead of repeating the rules, and they only note where a test departs from them.

Equipment and software

All runs use one Windows 11 Pro desktop (build 26200, later 26300, Intel Core i5-13400F, 32 GB RAM) with Google Chrome driven by Playwright 1.63.0. Chrome updated itself twice during the test series of September and October 2026: the in-browser encoder test and the first screenshots used Chrome 153.0.8010.53, most page-loading experiments ran on Chrome 154.0.8037.58, and the later priority, tool and request-audit runs used Chrome 155.0.8059.82. Each guide names the version its data came from, and every data set records it. Images are encoded with sharp 0.35.5 (libvips 8.18.7, MozJPEG 0826579, libwebp 1.6.0, libaom 3.15.1) unless a guide names a command-line encoder (cwebp 1.6.0, avifenc 1.4.2, oxipng 10.2.1). The full version list is saved with every data set as env.json.

The image corpus

• 20 photographs released under CC0 or in the public domain, downloaded from Wikimedia Commons with their author, licence and source URL recorded (see image credits). They cover landscapes, night scenes, food, macro, fur and feathers, text in a photo, architecture, interiors and two scans of artworks.
• 4 synthetic images generated by code and labelled as such: a smooth gradient, a flat vector illustration, dense line art and a bar chart.
• 8 real screenshots of this website captured by Chrome at 1280 × 800 CSS pixels, at 1× and 2× device pixel ratio.
• 3 synthetic RGBA assets (soft shadow, translucent overlay, antialiased logo) for transparency tests.

Before any codec test, each photograph is decoded once, rotated according to its EXIF orientation, converted to sRGB and resized with Lanczos3 so its long edge is 1600 pixels. That lossless PNG is the reference every encoded file is compared against. Downscaling a large camera original also removes most of the compression noise the original JPEG carried.

Quality metrics

File size alone says nothing about whether an image still looks right, so every lossy file is decoded and scored against its reference with three metrics:

• SSIMULACRA2 (libjxl 0.12.0) is the main metric. It runs from about −100 to 100, where 100 means identical. The libjxl documentation gives these anchors: 90 is “very high quality” (likely indistinguishable from the original at 1:1 from a normal viewing distance), 70 is “high quality” (artefacts are hard to notice without comparing to the original), 50 is “medium quality” and 30 is “low quality”. We use 70 as our default “high quality” target and 80 as a stricter target between those anchors.
• DSSIM 3.4.0: 0 means identical; larger values mean more visible difference.
• SSIM (Wang et al., 2004) on the luma channel with an 11-pixel Gaussian window, σ = 1.5, computed with scikit-image 0.26.0. It is included because it is widely quoted, not because it is the best predictor of what people see.

When a guide compares formats “at equal quality”, it means this: each image is encoded at a ladder of quality settings, and the file size needed to reach a threshold (for example SSIMULACRA2 = 70) is interpolated between the two nearest ladder points on a logarithmic byte scale. We never extrapolate beyond the tested ladder; an image whose lowest setting already exceeds the threshold, or whose highest setting never reaches it, is left out of that comparison and the guide says so.

Metrics disagree with each other and sometimes with human eyes. Every guide that relies on them also shows 100 % crops of the files so readers can judge the artefacts directly.

Loading tests

• Fixture pages are plain HTML served from a local Node.js server with Cache-Control: no-store, so no theme, plugin, font or advertising script can change the result.
• Each load uses a fresh browser context with the HTTP cache disabled.
• The slow profile matches Lighthouse’s mobile preset: 150 ms round-trip latency, 1.6 Mbit/s download, 750 kbit/s upload and 4× CPU slowdown, applied through the Chrome DevTools Protocol. A “fast” profile (20 ms, 20 Mbit/s, no CPU slowdown) is used where network timing is not the question, such as which srcset candidate is chosen.
• Viewports: mobile 390 × 844 at DPR 3, tablet 768 × 1024 at DPR 2, desktop 1440 × 900 at DPR 1.
• Largest Contentful Paint and Cumulative Layout Shift are read from the browser’s own PerformanceObserver entries. Bytes and request times come from DevTools Protocol network events, which are the same numbers the DevTools Network panel shows.
• Timing experiments are repeated 5 or 7 times and the median is published together with the minimum and maximum. Selection experiments (which file did the browser download?) are repeated 3 times.

Units

In our tables and charts, 1 KB means 1,024 bytes. Where precision matters, the exact byte count is given. Percentages of savings are always relative to the first file named in the comparison.

Encoding time

When encode time matters, each encode is timed with a high-resolution clock three times in a row and the median is kept. Encoders use their default threading. Times are wall-clock times on the bench above and are useful for comparing settings with each other, not as absolute speeds for your server.

Reproducing a result

Each guide links to a CSV file with every measured row. The complete set of scripts (Node.js and Python) is published as a single archive on the image credits page, together with the list of source photographs, so the corpus can be downloaded again from Wikimedia Commons and every experiment re-run. Results will differ slightly with other encoder versions, and that is itself worth reporting.

What these tests do not show

• One machine and one browser engine. Safari and Firefox decode and schedule images differently.
• Simulated networks are repeatable but smoother than real mobile networks.
• Twenty photographs are a reasonable sample, not every kind of image. We report how many images each conclusion holds for, and the exceptions.
• No test here measures ranking, revenue or conversion effects.