Demographers must work together so that officials can produce numbers all can trust
9 August 2022 – There’s something strange about Minnesota’s COVID-19 vaccination data. Since summer 2021, official vaccination rates for elderly Black and Asian American residents have been 100%. That, of course, is wrong.
Every demographer recognizes the problem: a mismatch between who’s counted in the numerator (people vaccinated) and the denominator (population tallies). In a changing population, denominators can quickly become out of date. Newly available data suggest that numbers of elderly Black people in Minnesota swelled by more than 20% between 2018 and 2021, but back when the state began tracking vaccinations, the most recent available denominators were years out of date. Pairing 2021 vaccination figures with population estimates from even a few years ago badly overestimates vaccination rates.
A faulty denominator also generated alarm in England when it misleadingly suggested that COVID-19 infections in 40–79-year-old residents were higher among vaccinated than unvaccinated people. The problem? An overestimated denominator led infections to be linked with an unrealistically large pool of unvaccinated people, depressing the apparent infection rate.
Especially sensitive to bad denominators is excess mortality, a workhorse tool that captures how many more people died than would ordinarily be expected to; a too-small denominator inflates estimates of excess death rates in two ways. If estimates assume a population has fewer people than it really does, we expect too few deaths — and then the resulting overestimate of how many deaths are ‘excess’ is attributed to too small a population.
Denominators go awry by being out of date (as in Minnesota) or by drawing numerators and denominators from sources capturing different sets of people (as in England). Even slightly outdated denominators quickly become inaccurate when a radical event disrupts births, deaths or migration. Major social crises often disrupt all three.
To solve the denominator problem, demographers, who specialize in measuring populations, must come together. It’s time for them to provide detailed guidance on constructing denominators when official data are out of date. We need a collaboration that can unify efforts to deal with outdated denominators and, crucially, make them accessible to epidemiologists, health officials and others.
Minnesota’s vaccination data illustrate the bind that researchers and decision makers are in. It’s because the state has done a laudatory job of offering useful numerators, reporting vaccinations by race and age together, that its denominator problem is so acute. As subpopulations become more defined — as in those elderly Black populations — the data become more useful for understanding inequalities and planning health campaigns, but the relevant denominators are the least up to date.
As an individual researcher, I can adjust denominators however best suits my research question, making reasonable guesses about how a population has been changing. But public-health officials working in highly polarized environments are loath to do this, lest they be accused of cooking the numbers and playing politics. They understandably default to unadjusted — but outdated — data.
And individual researchers’ freedom to adjust the denominator comes at a cost: when each article constructs denominators differently, results can’t be compared easily across studies. Public-health officials tell me they are frustrated that every study makes different adjustments. And even skilled researchers might not realize how heavily their estimates depend on how they update population (or don’t) — some excess-mortality analyses don’t even say what denominator data are used. I myself have been stunned to see how much this can matter in my own work.
Instead of a situation of every analyst for themselves, we need a collective solution — one that would be sufficiently authoritative, clear and applicable for demographers to build on and for public officials to embrace.
What would that involve? We’d need specific, open-sourced software routines for updating older data using recent statistics (births, deaths, migrations) and for combining newer data with projected trends for specific populations. A collective solution would also require guidance, grounded in empirical and simulation analyses that answer questions such as, how great a population shock (such as a spike in deaths from an infectious disease) must there be before it’s a bad idea to project older trends (that might no longer apply) into the present? Such questions rarely matter much in normal times. In a crisis, they can be crucial.
Demographers are well positioned for this effort. We work across a variety of academic disciplines, so can draw on broad expertise in, for example, measuring mortality and migration, estimating populations in small areas and developing forecasting models.
Consensus is crucial, because public officials can only adopt techniques that are accepted as legitimate and unbiased. A major international initiative could produce something genuinely useful within a year, and a fuller set of answers within a few years. Reliable estimates are needed for this pandemic and the next ones. They will be needed as climate and other disasters strike, allowing us to detect deaths even as reporting practices lag.
The best approach to future crises is to begin now.