This article is part of weekly updates on new developments in the use of AI methods and tools of surveys (households, individuals, farms…) and administrative data for official statistics

Coverage Period: 17–23 November 2025

Key words: AI, survey research, official statistics, machine learning, data quality, household surveys

Introduction

This weekly update provides a summary of new developments in the application of Artificial Intelligence (AI) in survey research and household surveys. The report covers recent trends in data editing, cleaning, processing, analysis, reporting, and dissemination, offering valuable insights for researchers and statistical offices. The findings are based on a review of news articles, academic papers, and industry publications from the past week.

Key Developments This Week

This week saw significant discussions around the transformative potential of AI in market research, with a strong focus on synthetic personas and digital twins for simulating human responses [1]. Data quality remains a critical theme, with new AI-powered tools emerging to automate data validation and cleaning [2]. The role of AI in official statistics was highlighted in the IMS Presidential Address, which projected that AI-generated data could exceed 80% of total data by 2030 [3]. Furthermore, a new study demonstrated the successful application of machine learning algorithms for analyzing population-based household survey data [4], while another introduced a framework for using generative AI in data storytelling [5].

AI in Market Research: Synthetic Personas and Digital Twins

A recent article in the Harvard Business Review detailed how generative AI is set to revolutionize the $140 billion market research industry through the use of synthetic personas and digital twins [1]. These AI-generated proxies for human respondents promise to reduce the time and cost of traditional survey methods.

By using publicly available or proprietary data to simulate human responses to questions and surveys, these new tools promise to allow marketers to conduct research and experiments without the time, cost, and participant burden of traditional interviews or surveys.

— Harvard Business Review [1]

Synthetic personas represent a composite individual or group, and can be used in two ways: a top-down approach for a single best answer, or a bottom-up approach creating a “silicon sample” with response variability. Digital twins, on the other hand, are individual-level AI replicas built from detailed customer data. Research from Columbia Business School’s Digital Twins Initiative shows promising results, with 88% relative accuracy in test-retest benchmarks, although they are not yet “ready for prime time” [1].

Data Quality and Automated Editing

Data quality remains a paramount concern as AI adoption grows. AYTM, a market research platform, introduced its Data Centrifuge system, an AI-powered quality guardian that uses NLP and machine learning to identify and remove low-quality survey responses from bots, speedsters, and those with gibberish answers [2]. The system operates on the philosophy of revealing authenticity rather than fabricating data, and includes proactive defenses like “honeypot traps” to detect LLM-generated survey answers.

Recent academic research also highlights the increasing shift towards automated data editing in statistical agencies. A 2025 paper by K. Švambarytė emphasizes the need for automated techniques to improve data processing efficiency in turnover statistics for service enterprises [6]. Similarly, a paper in the Statistical Journal of the IAOS discusses streamlining data workflows through automated decision-making to address the time-consuming nature of manual editing [7].

Machine Learning in Household Surveys

A study published in BMC Infectious Diseases demonstrated the application of five supervised machine learning algorithms to population-based survey data from sub-Saharan Africa [4]. The research, which analyzed data from 123,132 women, used models such as CatBoost, XGBoost, and LightGBM to predict awareness and perception of HIV pre-exposure prophylaxis.

The CatBoost model achieved the highest accuracy at 91%. The study also utilized SHAP (Shapley Additive Explanations) to identify the most influential predictors, which included education, media exposure, and healthcare utilization. This research showcases the potential of ML to extract actionable insights from large-scale household survey data.

Table 1: Performance of Machine Learning Models on Household Survey Data [4]

AI for Data Dissemination and Storytelling

Generative AI is also being explored as a tool for data dissemination and reporting. A paper in the Journal of the Association for Information Science and Technology introduced the AI-DIKW (Data-Information-Knowledge-Wisdom) framework for co-designing data-driven stories [5]. This framework uses generative AI as a co-designer to help data storytellers frame and edit narratives at four stages: extracting insights from data, enriching them with context, adding meaningful next steps, and tailoring the story to specific audiences. This approach has significant implications for statistical offices looking to make their findings more accessible and engaging for a wider audience.

The Future of AI in Official Statistics

The increasing prevalence of AI-generated data presents both challenges and opportunities for statistical agencies. In his presidential address to the Institute of Mathematical Statistics, Tony Cai projected that AI-generated data could surpass human-generated data as early as 2026 and exceed 80% of total data by 2030 [3]. This shift necessitates the development of principled frameworks to validate, trust, and interpret AI-generated data.

The address emphasized that statistics is central to AI, with core principles like inference, interpretability, and uncertainty quantification being more critical than ever. Statistical agencies have a crucial role to play in ensuring that AI-driven systems are scientifically valid, ethically designed, and rigorously evaluated.

References

[1] Korst, J., Puntoni, S., & Toubia, O. (2025, November 15). The AI Tools That Are Transforming Market Research. Harvard Business Review. Retrieved from https://hbr.org/2025/11/the-ai-tools-that-are-transforming-market-research

[2] AYTM. (2025, November 12). Meet your quality guardians: How we use AI to protect your research. Retrieved from https://aytm.com/post/meet-your-quality-guardians-how-we-use-ai-to-protect-your-research

[3] Cai, T. (2025, November 15). IMS Presidential Address: Statistics at the Crossroads – Challenges and Opportunities in the Age of AI. Institute of Mathematical Statistics. Retrieved from https://imstat.org/2025/11/15/ims-presidential-address-statistics-at-the-crossroads-challenges-and-opportunities-in-the-age-of-ai/

[4] Terefe, B., et al. (2025, November 14). Machine learning to examine adequate awareness and positive perception of HIV pre-exposure prophylaxis among women in sub-Saharan Africa: evidence from 2021-2024 surveys. BMC Infectious Diseases. Retrieved from https://pmc.ncbi.nlm.nih.gov/articles/PMC12619335/

[5] Lo Duca, A. (2025, November 13). Using generative AI to co‐design data‐driven stories. Journal of the Association for Information Science and Technology. Retrieved from https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.70036

[6] Švambarytė, K. (2025). Machine learning methods for automated data editing of the turnover of service enterprises. Vilnius University. Retrieved from https://epublications.vu.lt/object/elaba:229583427/

[7] Sirello, O., Bogdanova, B., et al. (2025). Metadata in the age of AI: The role of official statistics in securing a virtuous cycle. Statistical Journal of the IAOS. Retrieved from https://journals.sagepub.com/doi/abs/10.1177/18747655251342649

Contact: bakodramane@gmail.com