EDMO Policy Research and Analysis, coordinated by the Centre for Media Pluralism and Media Freedom, EUI

A bit more than a year after an updated version of the Code of Practice on Disinformation was adopted, the first pilot measurement of Structural Indicators was published by TrustLab on September 26, 2023. This pilot, commissioned within the Code of Practice, is an important contribution to the process of designing, testing, and implementing sound and comprehensive Structural Indicators that unveil relevant dimensions of the disinformation phenomenon and how they evolve over time. Primary goal of such Structural Indicators is to understand the effectiveness of measures adopted by various platforms signatories of the 2022 Code of Practice on Disinformation, individually and collectively, and to further inform policy developments in this area.

The European Digital Media Observatory (EDMO), more specifically its Policy Research and Analysis team (CMPF-EUI), has been actively involved in the process of developing Structural Indicators by participating in all the relevant working groups of the Code of Practice Task-force. The Task-force is designed as a permanent feature of the new Code where signatories, together with EDMO, ERGA and relevant third-party experts regularly review these indicators. In general, structural indicators are used to monitor the implementation of certain policies (e.g. employment, health, media etc.), and against the objectives to be achieved. Most commonly, such indicators use socio-demographic variables and key structural components for longitudinal measurements, with the objective of observing structural transformations. As such, Structural Indicators of the Code are key to understanding dimensions of disinformation online and how they evolve over time. Moreover, in this context Structural Indicators should be an element of transparency and public scrutiny.

Structural Indicators are a specific commitment (41) under the 2022 Strengthened Code of Practice on Disinformation. To achieve this Commitment, signatories established a Working Group in June 2022 and in early September EDMO tabled a first proposal for Structural Indicators to initiate discussions within the Group. EDMO’s initial proposal included six areas of measurement: prevalence, sources, audiences, and demonetization of disinformation, as well as the impact of and investment in fact-checking, and investment in the implementation of the Code. All these indicators were to be measured for each member state and for the EU as a whole. The proposal explored the potential of engaging the existing networks and resources within the Code (i.e. various signatories) and EDMO (i.e. fact-checking community and local EDMO Hubs). The EDMO proposal strived to be comprehensive, in order to reflect the fact that disinformation is a multifaceted and complex phenomenon. It also envisioned access to platform data as a key precondition for effective monitoring of disinformation in that environment.

While in the course of autumn of 2022, several platform signatories had worked towards significantly increasing their data availability, the datasets and data points offered by them did not allow for satisfactory cross-platform Structural Indicators. To overcome this stalemate, in January 2023, platforms and other signatories of the Code committed to evaluating whether one or more third parties should be selected to deliver the first set of Structural Indicators by the first reporting period scheduled for July 2023. TrustLab was selected and contracted for a pilot analysis. Due to a very short timeframe and limited resources negotiated with platform-signatories, who were funding this exercise, the TrustLab pilot was limited to 2 indicators (prevalence, sources) and 3 countries (Poland, Slovakia, Spain). The analysis encompassed 6 platforms (Facebook, Instagram, LinkedIn, TikTok, Twitter (X), YouTube). During the process, Twitter, later renamed to X, left the Code of Practice, but remained included in the pilot analysis.

TrustLab’s pilot was framed and inspired by the EDMO proposal for Structural Indicators, but is a completely independent implementation that uses its own methodology and metrics. While measurements on absolute and relative mis/disinformation post engagements, as well as the ratio of disinformation actors and their characteristics, perform along the lines of the EDMO proposal, the discoverability metric is a novelty put forward by TrustLab. Discoverability reflects a ratio of mis/disinformation posts among pieces of ‘sensitive content’. More specifically, it is a share of mis/disinformation content in selected results of an active search for specific keywords (related to disinformation claims), using each platform’s native search function. While there seems to be a general agreement on the key structural areas (indicators) to examine, this departure between the EDMO proposal and the TrustLab pilot confirms our earlier remark that there is no one approach to sampling and metrics. At this stage, various approaches are valid to explore. Each will have its limitations but may also offer insightful perspectives on a dimension of mis/disinformation.

Structural Indicators are scheduled to be measured and reported bi-annually. EDMO continues to support the process towards comprehensive and complete Structural Indicators whose measurement involves all EU member states. To this end, we have established an EDMO Expert Group on Structural Indicators for the Code of Practice on Disinformation, where more than 20 experts, on a voluntary basis, contribute to advancing the methodology and designing a desirable framework for a regular, independent and sustainable implementation of the Structural Indicators.