It’s Time to Open the Air Quality Monitor Black Boxes.

Can you really trust the air quality data you rely on? Prompted by a lack of transparency within the field of air quality monitoring, that’s a question we set out to learn more about a couple of months ago. In a field in which public health and environmental research and policies are directly based on data from air quality monitors, how much of an impact do ‘black box’ monitors and sensors have on muddying our results?

In the last quarter of 2024, the AirGradient team began to receive reports of an unusual issue that we had not encountered before. Customers - and some members of our team - began to see their air quality monitors regularly measuring 0 μg/m³ of PM_2.5. While, at face value, this is fantastic news because it indicates there are very few airborne particles present, we quickly noticed that something was off. Customers who used to see low, but still present, PM_2.5 concentrations were now seeing zeroes, whereas customers with older monitors continued to see the same readings as before.

Our company hadn’t made any changes that would have this kind of impact on readings, and, as we began to look into the issue, we discovered that it wasn’t caused by us, but rather, the PM sensors we rely on had been changed without any warning. As with most air quality monitor companies, we source our components directly from the sensor manufacturer, and unless the manufacturer is transparent, we don’t know exactly how these sensors and their algorithms operate.

In this way, the sensor itself is a black box, and even we, as an air quality monitoring company, don’t know exactly how they function. On top of this, most air quality monitor manufacturers are also far from transparent about how the monitors themselves work, leading to different levels of black boxes, which leads to confusion, but much worse, can lead to inaccurate or unexplainable results.

Prompted by this first-hand experience of black boxes within air quality monitoring, we wanted to know more and ran a survey on LinkedIn.

So, just how much of an issue are these levels of black boxes?

Who Took the Survey?

Throughout the time that the survey was open, we had 45 total responses. To better understand the impact of transparency issues in air quality monitoring, we also asked each respondent to classify themselves and their position within the field. As it turns out, the responses came from individuals across multiple sectors, highlighting a broad spectrum of experiences and concerns.

The majority of survey participants were researchers (21 respondents), with many coming from leading institutions worldwide. In addition to academic and independent researchers, we heard from:

• 10 professionals from the private sector working on air quality solutions, such as sensor manufacturers, environmental consultants, and data analysts.
• 5 participants from non-governmental organizations (NGOs) focused on air quality and environmental advocacy.
• 4 government officials and regulators, including state air monitoring scientists and local air quality agencies, also participated.

Why Transparency Matters

The vast majority of respondents use air quality monitor data very regularly, and this makes them heavily impacted by any lack of transparency. Out of the 45 people who replied, more than half use air quality monitors and their data very frequently, 11 use them frequently, while eight use them occasionally. Only one respondent rarely or never uses them. Importantly, since the majority of the respondents are in research, private sector, NGOs, and governmental roles, non-transparent data that leads to incorrect results could have a significant impact.

Air quality monitoring plays a crucial role in public health, environmental research, and policy-making. From tracking pollution trends to informing regulatory decisions, reliable air quality data helps researchers, governments, and private companies make data-driven choices that affect millions of people.

However, the reliability of this data depends not just on the sensors themselves, but on how the data is processed. If the algorithms used to calibrate, filter, or average sensor readings are not transparent, users cannot fully understand how the data was produced - or whether it might be inaccurate or biased.

When asked, ‘In your opinion, how important is transparency in the data processing methods of air quality monitors for scientific research and other applications?’ the overwhelming consensus from the survey was that transparency in air quality monitoring is critical:

• 88% of respondents stated that transparency is “very important” for scientific research and other applications.
• Only one respondent considered it “not very important.”

These findings confirm what we have long suspected: without transparency, the credibility of air quality data is at risk. Researchers and practitioners need to know how measurements are processed in order to trust the data, validate findings, and ensure consistency across different monitoring systems.

In particular, a few respondents mentioned that it should be clearly stated whether or not data is altered in any way. For example, is the data from a sensor raw (unadjusted) data, or does it have some form of correction - such as a relative humidity correction - applied?

Here are just a couple of the issues we encounter when air quality data is not transparent:

• Lack of reproducibility: Without knowing how data is processed, scientists cannot compare or replicate results. This isn’t just an issue between different monitors but even within data from the same monitor at different times, as algorithms can be adjusted through firmware updates. If users aren’t notified, there is no way to know about these changes, even when using the same device.
• Hidden biases: If manufacturers apply undocumented corrections or averaging techniques, data could be skewed without the user’s knowledge. This can also make comparisons between measurements from different monitors extremely difficult. For example, some sensors have calibrations applied that lead to better results under certain conditions ( such as in different climates and when measuring different particle types) but worsen results under other conditions.

The issues with transparency begin at the research level, but they can cause far greater issues down the line. If researchers can’t fully trust their data or trust it without knowing about these hidden factors, it can lead to obstructed or incorrect studies. This might lead to serious longer-term consequences, if this data is used for larger studies on which government policies might be based.

The Black Box Problem

One of the most striking takeaways from the survey is how frequently professionals encounter challenges due to the lack of transparency in air quality monitoring. A staggering 84% of respondents reported that they had faced issues in their work because of undisclosed or proprietary data processing methods, showing that when manufacturers do not disclose how they calibrate, filter, or process raw sensor data, users are left in the dark.

The biggest concern surrounding these “black box” air quality monitors is data integrity. Without knowing how readings are adjusted, averaged, or corrected, researchers, policymakers, and industry professionals struggle to validate the accuracy of the data they rely on. This concern was strongly reflected in the survey, with 61% of respondents stating they were “very concerned” about undocumented compensations, calibrations, or averaging methods, while another 32% were “concerned”. Only three respondents were neutral on the matter.

A lack of transparency also creates barriers for those trying to interpret air quality measurements. 82% of respondents reported difficulties in understanding unexpected or anomalous readings due to unknown processing methods. This not only affects scientific research but also impacts decision-making in regulatory and environmental management contexts. When air quality monitors are used to assess compliance with pollution standards or guide public health responses, unverified and unexplained data can lead to flawed conclusions and ineffective policies.

Ultimately, the black box problem erodes confidence in commercial air quality monitors. Professionals who depend on this data want to ensure that what they see on their screens reflects reality - not an opaque algorithm’s interpretation of it.

How “Black Boxes” Affect Perception

Now that we know just how much of a problem these black boxes are, we wanted to know how proprietary algorithms impact trust in air quality monitors by asking the question, ‘To what extent do you agree with the following statements about the consequences of the lack of transparency in air quality monitor algorithms? [The lack of transparency reduces my confidence in interpreting the data.]’.

Survey respondents overwhelmingly distrust proprietary “black box” monitors, with 86% expressing some level of distrust. Half of the respondents strongly agree with the statement that they distrust proprietary monitors, while 36% agree. Only 11% stated that there is no difference between black-box and open-source monitors, while some users did not respond to the question.

If even the researchers collecting data from the monitors lack trust in their monitors and the data they gather, how can organisations and governments trust it enough to create new policies and laws?

In short, trust in air quality data is directly tied to transparency, and no respondent said that black boxes increased their trust in a device. Instead, all either remained neutral or mentioned that black boxes decrease their trust, indicating that manufacturers embracing transparency will be what researchers, regulators, and industries turn to.

Transparency Creates Trust

The survey results make one key message clear: air quality professionals want greater transparency in how data is processed. From researchers to regulators, users need to understand the algorithms and adjustments applied to raw sensor data to ensure accuracy, reproducibility, and trustworthiness.

When asked about the most important aspects of transparency, respondents emphasized four key priorities:

• Detailed documentation of data processing methods – 63% of respondents rated this as “very important”, with another 32% considering it “important”.
• Access to raw sensor data – 70% of respondents called this “very important”, as unprocessed data allows for independent verification.
• Open-source firmware and algorithms – While not as universally prioritized, 48% of respondents considered this “very important”, and 34% found it “important”.
• Clear communication of algorithm changes by manufacturers – 73% of respondents said this was “very important”, reinforcing the need for consistent, transparent updates.

Opaque Companies Lose Trust

The survey results reveal a growing demand for transparency in air quality monitoring, presenting both a challenge and an opportunity for manufacturers of both sensors and monitors. With 88% of respondents stating they would be more likely to use a monitor with transparent data processing, it’s clear that openness is a key factor in selecting monitors.

Furthermore, researchers, regulators, and industry professionals overwhelmingly agree that undocumented algorithms and hidden data processing methods undermine trust, hinder research, and create barriers to accurate environmental assessments.

The survey results send a strong message: transparency in air quality data processing should be a focus for all manufacturers, who can use this as a chance to differentiate themselves by embracing transparency. Companies that provide detailed documentation, access to raw data, and open-source algorithms will gain trust and credibility among researchers, regulators, and industry professionals. Transparent practices also encourage collaboration and innovation, allowing scientists and developers to improve algorithms and enhance air quality monitoring accuracy.

The Low Cost Sensor Revolution Is Incomplete

Just a few decades ago, widespread air quality monitoring was simply not feasible. The emergence of low-cost sensors has changed everything, sparking a sensor revolution that has made air quality data accessible to millions. This has led to a dramatic increase in awareness about the health impacts of pollution and has played a key role in influencing air quality policies globally. But while we’ve come a long way, the revolution is not yet complete.

With these black boxes that continue to exist, we’re losing opportunities to enhance accuracy and foster innovation. Proprietary approaches stifle innovation and can hinder decision-making, ultimately leading to reduced public trust.

To continue the revolution, we want to call for more openness and transparency in air quality monitoring. Imagine a world with truly open air quality data - a future where air quality data is freely accessible and trustworthy. This is what we should aim for.

Better public health reduces environmental injustice, and better-informed decision-making in air quality starts with the companies producing the sensors and monitors that we rely on. Therefore, we want to call on anyone reading this to hold those companies - including us at AirGradient - accountable.

The air we breathe is too important to be left in a black box.