by Giovanni Zagni and Tommaso Canetta

Over the 24-25 March 2023 weekend, an image of Pope Francis dressed in a puffy, bright white coat went viral across the globe. A few days before, a set of images of Donald Trump being arrested and thrown in jail was viewed by millions of people. Those images were not real: Trump was not in police custody and the Pope never wore that chunky jacket. They were produced using generative artificial intelligence (AI), a technology capable of producing content in a variety of formats – textual, audio, or visual – from instructions provided by the users, the so-called “prompts”.

It is possible that, in both the aforementioned cases, many users spotted the false content (which was quickly covered by fact-checkers around the world). In the second case, for example, it was easy to check that no news portal had covered such a newsworthy arrest. But many other users surely mistook the images for real. In the case of Pope Francis, the incident was called potentially “the first real mass-level AI misinformation case”.

The authors of those false images are known. The American outlet Buzzfeed News interviewed the 31-year-old construction worker from the Chicago area who originally posted the AI-generated image of the Pope on the social news and discussion forum website Reddit, as well as in a Facebook group devoted to AI-generated art, with no intention for them to become a global sensation. Trump’s arrest series was originally tweeted by Eliot Higgins, the founder of the Osint project Bellingcat – oddly enough, with no watermark or other clear sign of its inauthenticity, a basic precaution used by many fact-checkers to avoid the inadvertent spread of false content.

A striking coincidence

Even if generative AI tools such as Midjourney or Dall-E have been around for some months now (both were released in an open beta version in July 2022), it is probably no coincidence that their crucial role in disinformation incidents with a very broad relevance happened in the same weeks of another technology entering the public discourse: generative AI with textual outputs, most famously OpenAI’s GPT and similar large language models (LLMs).

The wide discussion around the ChatGPT chatbot – and the following iteration, GPT-4, which among other things provides as a result and accept as prompts both text and image – and its consequences for the media and information system runs in parallel with the new challenges originated by the older, but not less impactful, generative AI tools with visual outputs.

The end of March 2023 marks the beginning of a new era for mis- and disinformation, one in which a technological advance ushers in new dynamics in the production and spread of false and misleading content.

These dynamics are only partially similar and, in our opinion, need to be separately evaluated and addressed.

1. Textual outputs

Generative AI tools with primarily textual outputs, such as OpenAI’s ChatGPT or similar LLMs, have many potential risks for the production and spread of mis- and disinformation – setting aside other problems that do not strictly have  to do with information quality, such as privacy concerns (which, on 31 March 2023, led the Italian data protection authority to issue a ban against OpenAI’s ChatGPT, subsequently suspended for Italian users) or aggressive/unsettling results.

First of all, they are prone to what are known as “hallucinations”, i.e. factual mistakes in the information provided to the user (we advise against using a term such as “hallucinations”, which implies an humanization of the technology). Public presentations of LLMs by tech giants such as Microsoft and Google were marred by mistakes, including in promotional videos. Experts warn that such mistakes are not simple glitches of the system, but inevitable outcomes of the technology at its current state. As an article in the MIT Technology Review vividly sums up, «AI language models are notorious bullshitters, often presenting falsehoods as facts. They are excellent at predicting the next word in a sentence, but they have no knowledge of what the sentence actually means». In sum, the way in which they are designed and their current state of development make LLMs good at some things (such as writing code) and bad at (many) others, among them playing word puzzles or… providing accurate information.

Secondly, there is the issue of scale. LLMs are made to produce texts at will, including about conspiracy theories or urban myths, and many commentators have pointed out that this risks producing “a fake news frenzy”. It is important to point out that OpenAI recognized the possible misuse of its technology for disinformation campaigns and tried to forecast future developments in a paper developed with independent researchers that addresses “potential threats and potential mitigations”.

We believe that the most concerning aspect connected with chatbots using LLMs stems from the direct access it gives to the users to a personalized, human-like, conversational way of presenting content. Compared to traditional search engines, tools like ChatGPT provide a unique answer that could be easily interpreted as the best or most accurate result to their online search for information. Users – especially when not appropriately informed about the technology, its strengths and weaknesses, and what it is built and currently able to do – could be tricked into giving more trust, or to rely more heavily upon, the results provided by a conversational chatbot than other, more open-ended tools for looking up information, making it more difficult for the users to properly assess the veracity of the results, and ultimately to exert their critical thinking.

To make things worse, users access LLMs without mediation: while through traditional search engines they access a list of results that were produced and published elsewhere, in some sense scanning what is already out there, through chatbots they are able to directly access “new” information, produced by the LLM in response to their particular prompt.

This unmediated access makes that information practically impossible to fact-check, unless the user decides to take them outside the closed system of their private back-and-forth with the machine. LLMs will produce false or misleading results and will deliver them directly to the users, while until now disinformation had to be published somewhere – be it a website, a social media account, a podcast – for it to be accessible to other users.

In our opinion the main problem originated by LLMs is not one of quantity or even quality of the information, but one of distribution. Informative campaigns for the public are needed in order to make clear the limits of the tool, as well as its risks.

In conclusion, the jury is still out, and will be for some time, on the potential uses of generative AI for fact-checking. A preprint published by researchers at the University of Zurich found out that ChatGPT was able to accurately categorize fact-checked statements as true or false in 72% of cases. This is still a long way to go, as one out of four mistakes is too low an accuracy for the tool to be used effectively.

2. Visual outputs

The issue with visual disinformation generated by AI is largely different. In the current stage, it is the user that always has to give a prompt to the system in order to get the image. In this case, and contrary to the textual outputs, it is not possible that the user questions the machine without knowing that the result is different from reality. Disinformation, here, can only be produced intentionally.

This is in some sense a simple evolution from the current production of visual disinformation: until now, this was produced simply by presenting old images and videos, e.g. miscaptioned and re-contextualized to portray current events. Crude manipulations, but more often just out-of-context photos, were able to have an already significant impact. The amount of content that generative AI could provide to who produces this kind of disinformation is potentially endless.

Beside the problem of quantity, there is also an issue with quality. With the technology becoming more and more advanced, it is possible to foresee a future when the results will be indistinguishable from real pictures to the untrained (or even expert) user. It is therefore crucial that fact-checkers, as well as common users, are provided with technological tools able to detect AI-generated images and videos. Other solutions could also be explored (albeit with significant issues), such as a mandatory watermark on AI-generated visual content or other technical markers (e.g. as part of metadata).

In conclusion

The generative-AI technology has also shown other worrisome aspects and possible applications in fraudulent activities and disinformation campaigns, such as automated scam calls or fake paper submissions. The technology is constantly evolving and new challenges, as well as opportunities, will arise with it.

From the perspective of disinformation, chatbot-like tools based on generative AI give users unmediated access to results that are not necessarily trustworthy or accurate; visual outputs of generative AI tools give malicious actors an unprecedented ability to produce realistic images and videos.

More research is needed to better understand how to properly assess the risks posed by artificial intelligence and how to tackle them, and at the dawn of a new era for disinformation it is crucial to maintain a common effort by media professionals, fact-checkers, developers and common users on how to ensure that this technology, with its many exciting aspects and its innovative potential, is used in an enriching, ethical and responsible way.