The promise and problems of including ‘big data’ in official government statistics
The Australian Bureau of Statistics (ABS) will soon announce the kinds of information it will collect in the next national census in 2021. If international trends are a guide, “big data” will comprise a growing part of ABS data collection and analysis.
This may promise greater timeliness and efficiency compared to the traditional paper-based census, but using big data to measure populations and economies is not without challenges.
Debates about how democratic governments should count the people they serve are ongoing in Australia, the U.S. and in India. The use of digital technologies for state measurement seems likely to intensify these debates as significant questions emerge around the practice.
Public data gathering has high stakes
For centuries, states have counted and categorized people. Census data and other official statistics are used for government planning and budgeting, to determine political districts for elections, and for many other purposes. Official statistics also help to shape a population’s sense of itself. For these reasons, state counting practices have often been controversial.
In Australia, changing census practice has been a part of ongoing debate about ensuring First Nations people are properly represented. Historic undercounting of Aboriginal and Torres Strait Islander people was redressed by the abandonment of language in the census that referred to blood quantums — which are now widely accepted as racist — alongside other factors.
In the U.S., state counting is likewise a matter of intense dispute. California is among those states currently suing the U.S. federal government because of a question about citizenship status the Trump administration has proposed adding to the 2020 Census. California argues fewer non-citizens will complete the census if the question is included. This would lead to a lower population count and reduced federal funding for states with high numbers of non-citizens.
Big data use in official statistics is growing
New issues of this kind are likely to emerge as government statistics offices around the world introduce digital data into their work.
The UN is currently spearheading efforts by member states to explore the use of new, digital data sources and technologies for official statistics. The ABS is involved in this endeavor. Since late 2017, for example, the ABS has been analyzing supermarket scanner data to try to improve CPI (inflation) measurement.
Other possibilities being explored for the use of digital data to improve state measurement include: