Fact-Checking | Islamists' Misuse of Artificial Intelligence for 'Deepfake' Apocalyptic Narratives

The website 'Poligraph', a fact-checking project by Voice of America, has published a report on a new study. This study addresses how an Islamist network is misusing artificial intelligence to promote 'apocalyptic narratives' and increase concerns about the misuse of AI. The report states that a deepfake of Elon Musk, founder of 'Tesla' and 'SpaceX' and current owner of 'X' (formerly Twitter), confirms the emergence of the Twelfth Imam of Shia Islam in an interview with CNBC host David Feber. This deepfake combines parts of a real interview with a fake voice as part of a propaganda campaign aimed at strengthening the apocalyptic narrative of Islam. A report published earlier in September on the 'Global Network' website mentions that a network of fake accounts has disseminated hundreds of TikTok, YouTube, and X videos in which a Pakistani man claims to be in direct communication with Muhammad [the Prophet of Islam] and God through his dreams. He considers himself the last Imam of Shia Muslims and threatens that 'anyone who does not accept him will burn in the deepest part of hell.' Daniel Siegel, one of the authors of the report, told Voice of America that after the arrest of Imran Khan, the former Prime Minister of Pakistan, 'we quickly noticed the use of AI in spreading a narrative by a group that believes the apocalypse is near and the savior is in Pakistan.' The group's use of AI has two aspects: deepfake imagery and deepfake audio. The released videos show a rapid montage of AI-generated images in which the 'savior' confronts prophets and stands in heroic poses. This network also utilizes deepfake audio, including creating voices of prominent Islamic scholars and even Barack Obama, the former President of the United States, to lend legitimacy to its apocalyptic narrative. According to the report, this anonymous network is also attempting to elevate Pakistan's status in the Islamic world. Bilwa Chandra, another author of the report, states, 'These actions position Pakistan powerfully from a religious perspective among the Muslim community.' He added that this network is also trying to unite Muslims worldwide. 'Thus, it is possible that this network could garner support from Muslims for something far scarier and more dangerous than the current narrative.' The authors of the report say it is not exactly clear which AI tools were used to produce these materials, which are in English, Urdu, Arabic, Bengali, Indonesian, and Malay. Measuring the impact of this campaign, like most deepfake campaigns, is difficult, but data indicates a thriving online presence. Siegel and Chandra state that after their report was published, this network targeted them in several ways. Chandra emphasizes the importance of identifying better ways to regulate how AI is used. According to him, 'It is really important to examine all the different ways this technology is used by both large and small actors, governmental and non-governmental.' This article is based on the Poligraph report, a fact-checking project by Voice of America.