Measuring Disinformation’s Impact Remains a Challenge for Researchers

Researchers combating disinformation have made significant strides in recent years, developing frameworks and methodologies to understand how false information spreads and affects societies. Despite this progress, measuring the actual impact of disinformation campaigns remains one of the field’s most persistent challenges, according to a comprehensive analysis by EU DisinfoLab.

The question at the heart of this challenge is deceptively simple: what constitutes “impact” when it comes to disinformation? Should researchers focus on online reach metrics, real-world outcomes, or shifts in public attitudes and behavior?

EU DisinfoLab recently updated its Impact-Risk Index, which offers specific indicators to measure the impact of individual false claims. This framework complements other methodologies like the Breakout Scale, designed to evaluate broader influence operations.

The analysis examined six major frameworks currently used in the field, divided into comprehensive disinformation-lifecycle frameworks (ABCDE Framework, Disarm Framework, and CIB Detection Tree) and impact-focused frameworks (Breakout Scale, Impact-Risk Index, and Response-Impact Framework).

Different Approaches to Understanding Impact

Terminology varies significantly across frameworks, with terms like “effect,” “degree,” “effectiveness,” “changes,” and “impact” highlighting different aspects of disinformation’s reach and harm. This terminology shapes how impact is measured.

The ABCDE and Disarm frameworks balance considerations of reach and harm, while the CIB Detection Tree and Breakout Scale emphasize reach metrics, though the latter approaches such metrics with skepticism. The Impact-Risk Index heavily focuses on engagement indicators, while the Response-Impact Framework uniquely examines the effectiveness of countermeasures.

“Prioritizing reach implies focusing on audience metrics – online and offline – while addressing harm requires emphasizing online actions and shifts in public opinion, attitudes, and behaviors,” the report notes.

Quantitative vs. Qualitative Measurement

Most frameworks prioritize quantitative methods for practical reasons, though researchers acknowledge qualitative techniques are essential for comprehensive understanding. Platform metrics like cross-platform spread, engagement rates, and reach form the backbone of quantitative analysis.

Several qualitative factors receive varying attention across frameworks:

• Content format and language receive limited analysis in most frameworks, though the Impact-Risk Index identifies format variety as a potential amplification factor
• Media amplification is addressed by almost all frameworks, focusing on whether disinformation spreads from fringe outlets to mainstream media
• Human amplifiers (including public figures, recurrent disinformers, and more recently AI personas) are recognized in several frameworks as critical vectors
• The Breakout Scale uniquely considers the number of distinct communities reached by disinformation
• Psychological impacts like shifts in beliefs, institutional distrust, and polarization are acknowledged by most frameworks
• Offline consequences, particularly calls to action resulting in real-world events, are highlighted as significant indicators

Technological Considerations

While most frameworks remain technology-neutral, the updated Impact-Risk Index specifically addresses artificial intelligence’s role in disinformation. The framework examines how AI capabilities like deepfake creation, large-scale translation, AI-operated accounts, and real-time generated content can amplify disinformation’s impact.

Gaps in Current Methodologies

The analysis identified several significant gaps in existing frameworks:

1. Only the Breakout Scale and Impact-Risk Index provide detailed, actionable guidance on conducting impact measurements, while others remain conceptual
2. Social and psychological dimensions of impact are frequently mentioned but rarely explored in depth
3. All frameworks involve some level of speculation when drawing conclusions about impact
4. Most frameworks lack built-in mechanisms to flag high-risk cases that would help prioritize resources
5. The distinction between effective impact (demonstrated offline consequences) and potential impact (inferred risk) remains blurred in most frameworks
6. Limited attention is given to measuring the effectiveness of countermeasures

“Understanding how frameworks measure impact is fundamental to filling gaps and envisioning opportunities,” the report concludes. “Current methodologies show progress but remain fragmented. Greater alignment on terminology, stronger integration of qualitative and psychological factors, improved alert mechanisms, and a clearer emphasis on countermeasures are all necessary to advance impact assessment in disinformation and influence operations.”

As disinformation tactics continue to evolve, particularly with the integration of artificial intelligence tools, the need for standardized impact assessment methodologies becomes increasingly urgent for researchers, platforms, and policymakers seeking to counter these threats effectively.