Should You Base Your Air Quality Monitor Purchasing Decision on AQ-SPEC Results?

One of the biggest challenges with consumer-grade air quality monitors is knowing whether or not they are accurate. The best way to find out about the accuracy of a sensor is to compare it to a standard reference instrument. Unfortunately, in the case of particulate matter (PM₁, PM_2.5 and PM₁₀), a reference-grade air quality monitor costs tens of thousands of dollars, putting it out of the range of most organisations, individuals, and businesses. This means there are very few organisations or people out there with the equipment needed to assess the performance and accuracy of air quality monitors. On top of this, there is also a substantial knowledge barrier, which means even fewer are capable of assessing air quality monitor performance in an unbiased way.

One of the most trusted organisations and one of the few capable of carrying out this testing is South Coast AQMD (Air Quality Management District). This organisation has an extensive database assessing the accuracy of various consumer and professional-grade air quality monitors. While many people are already aware of the AQMD AQ-SPEC (Air Quality Sensor Performance and Evaluation Center) testing program, relatively few are aware that other groups are also carrying out similar testing in other parts of the world. Some of them have even more extensive testing!

Today, we want to discuss two of these other databases. I imagine that for most people, this will be your first time hearing about them as they’re both significantly less well-known than AQMD’s AQ-SPEC. However, if you want to assess the performance of an air quality monitor you’re considering purchasing, they’re worth looking into.

The first of these databases is run by Afri-SET as part of its Evaluations Program, and the second is AIRLAB, managed by AirParif. While not all platforms cover the same monitors, there is a good chance that at least one has tested any given monitor. With that said, you might notice that sometimes the results from each organisation differ - and sometimes, quite considerably! Why is this? Today, we will discuss each of these databases and explain why they sometimes differ and how they should (or if they should!) sway your purchasing decision.

Why Is AirGradient not on AQ-SPEC?

Since we expect a lot of people to ask this question, we want to briefly explain why AirGradient does not appear in the AQ-SPEC database before diving into the comparisons. Basically, each company only gets one chance to submit devices to AQ-SPEC, and we want to ensure we are fully ready before submitting our samples.

Specifically, we wanted to wait until we had our own reference instrument set up in our testing chamber before submitting devices. We now have the reference instrument (Palas Fidas) and are working on improving our monitors’ accuracy, and we hope to submit it to AQ-SPEC soon.

That said, the results are likely a long way away as AQ-SPEC currently has a relatively long delay before any results are made public - even after the monitors are submitted. With that mentioned, let’s dive into the databases themselves.

Test Results

Naturally, we first wanted to look at the results from each database to see if the results were similar or different. Since each database has tested a different range of monitors, finding a device tested by all three programs is quite a challenge. Therefore, we collected the results from various monitors - some of which have not been tested by all three organisations.

Furthermore, while all three databases test different parameters and present the findings differently, they all provide R-squared values for both PM_2.5 and PM₁₀ results from the monitors they’ve tested. These values show the correlation between the readings from the device in question and that of a reference-grade air quality monitor. A value closer to 1 represents a better correlation (1 is a perfect correlation), and numbers closer to 0 represent a lack of correlation (0 shows no correlation).

Since each organisation gives R-squared values, regardless of the rest of their test methodology and presented findings, we chose to compare these values today. If you want further data, some of the organisations (particularly AIRLAB - under the “Reports” tab, make sure you activate the toggle “Detailed report”) also provide far deeper analysis on their website.

It’s important to note that all organisations test three samples of each monitor, but their R-squared values are presented differently. For example, AQ-SPEC and AIRLAB present a range, whereas Afri-SET gives an average R-squared value based on the results of the three individual monitors. I’ve decided to present the findings as the organisations presented them for the sake of clarity.

About the data: The data for each of the three sources was collected from the following pages. AQ-SPEC Summary Tables & Reports, Afri-SET Evaulations, and AIRLAB Microsensors Challenge 2023 Reports.

Data from Afri-SET and AIRLAB is shown in one-hour averages, and AQ-SPEC data includes both 5-minute and one-hour averages as per the data on the AQ-SPEC Summary Tables & Reports page.

PM_2.5

*2022 AirQuality Egg results from AQSpec used. The Afri-SET version was not mentioned.
**Kunak Air Pro results were used for Afri-SET results as they did not test the Lite version. However, as per AQMD and AIRLAB, the Lite monitor is more accurate.
***R-squared values are normally significantly higher for longer time intervals. AQ-SPEC uses 5-minute and 1-hour means, and both Afri-SET and AIRLAB also use hourly averages for the R-squared values. However, Afri-SET also shows 5-minute means in its detailed monitor reports.

PM₁₀

*2022 AirQuality Egg results from AQSpec used. The Afri-SET version was not mentioned.
**Kunak Air Pro results were used for Afri-SET results as they did not test the Lite version. However, as per AQMD and AIRLAB, the Lite monitor is more accurate.

All three organisations provide two sets of data, so let’s first discuss why there are six columns in the table above.

AQ-SPEC: AQMD’s testing from the AQ-SPEC program compares monitors both in the field (in this case, in California) and, for some monitors, in a lab. It’s important to note that the lab evaluation can be considered under ideal conditions, and it represents the performance of a sensor when in best-case conditions. However, as you can see from the difference in results between the lab and field evaluations, these results aren’t applicable to most uses.

As mentioned above, the AQ-SPEC field evaluation results are shown as a range, as while all organisations test at least three of each low-cost monitor, AQ-SPEC and AIRLAB present the findings as a range where Afri-SET presents an average between the three monitors. The narrower the R-squared range is given by AQ-SPEC, the closer the readings of the three monitors are to each other, i.e. the higher their precision.

Read more: AQMD AQ-SPEC Field Evaluation Documentation

Afri-SET: Afri-SET also provides values for two tests - one during the Ghanaian dry season and one during the wet season. We will discuss this in more detail soon, but as you can see in the tables, this significantly impacts accuracy, and these results are shown to indicate the real-world performance of these monitors in a West African setting.

Read more: Afri-SET Evaluation Documentation

AIRLAB: AIRLAB runs two two-month tests as part of its process. These tests are carried out in very different locations, with one taking place in France and the other in Thailand. Since the conditions in these two locations can lead to significantly different results, both are presented as individual results.

Read more: AIRLAB Microsensors Challenge 2023 Evaluation Documentation

Findings and Discussion

If you look at the table above, you’ve probably already noticed how significantly the results can vary. Not only between organisations but even between testing by the same organisation in different locations. This is because the conditions for every test are different. Different organisations use different reference stations, and even within the same program, different references can be used. Despite being in a category that we generally refer to as ‘references’, there are actually often variations in readings between these devices. We will be releasing a full article on this topic soon!

Perhaps more interestingly, weather conditions have a significant impact on sensor performance. This is most evident in the Afri-SET results, where the location and reference monitors used stayed consistent, with only the seasons changing between the dry period and wet period testing.

This is because an important factor that can influence the accuracy of low-cost particle sensors is the humidity, which differs greatly between California, Ghana, Paris and Bangkok. While we might expect consistent results here (for example, monitors consistently perform worse in higher humidity environments), we can see that the results vary, indicating that there are differences in how the monitors are calibrated and what conditions they are optimised for. Regardless of what situations the monitors are intended for, it’s important to note that humidity has a big impact on accuracy.

Particle type and composition are topics within themselves but are often overlooked. While we often use terms like PM_2.5 as broad terms and assume all PM_2.5 is the same, many different forms of particles are included under this umbrella term. For example, during the dry season in Ghana, a significant amount of PM_2.5 is dust from the Sahara. On the other hand, PM_2.5 in Bangkok or Paris is more likely to be composed of particles emitted by vehicles. Sensors can be calibrated in different ways, making them more sensitive to certain particles than others.

To make matters even more complicated, some organisations like AQ-SPEC don’t perform their testing of all monitors simultaneously, meaning that even AQ-SPEC results aren’t directly comparable. While Afri-SET and AIRLAB carry out their testing of all monitors at the same time, exposing them to the same conditions, monitors tested under the AQ-SPEC program can differ vastly depending on when the year the monitor was tested. This isn’t to say that the AQ-SPEC results aren’t helpful, but rather that we shouldn’t base purchasing decisions entirely on the results a monitor achieves in its testing.

Of course, there are also many other factors that can influence the accuracy of monitors, and they quickly become very complex. For example, monitors often display different accuracies at different particle concentrations, and while this is somewhat location-dependent, it’s a whole different factor that is often overlooked in these results. However, it’s out of the scope of this article to discuss every single variable that can influence a monitor’s performance in these reference comparisons.

Perhaps the only consistent finding across all datasets is that monitors perform worse when measuring PM₁₀ than PM_2.5.

What Do These Findings Mean?

At this point, we’ve seemingly thrown even more uncertainty into the mix. If reference stations cannot even agree, how can we determine whether our consumer-grade air quality monitors are accurate? Furthermore, if even testing within an organisation isn’t consistent, can we compare the results? Well, we first need to realise these difficulties before we can make an informed purchasing decision.

First, let’s clarify that these programs - regardless of the exact organisation in question - are far from useless. At AirGradient, we believe that it’s better for you to consult any of these programs (even individually) than not to consult them at all. They all offer helpful information.

However, where possible, we highly recommend looking across multiple sources. Instead of relying entirely on a single database’s results, looking at the monitor’s performance in different situations is important. While many monitors only appear in one of the databases we’ve mentioned, many studies also compare individual monitors to reference equipment. Therefore, it’s important to consider multiple sources when considering which monitor to purchase.

Finally, while monitor performance can vary - even in the same location - you might want to give more weight to the database that best fits your situation. For example, if you live in Thailand, it makes sense to give AIRLAB’s Bangkok testing results more weight in your decision, as these best reflect the performance you can expect from your device.

Updated 19 Feb 2025 to correct an error in QuantAQ data and added sources.