Emerging Shadows: Unmasking Cyber Threats of Generative AI

Knowledge Graph of Debate

Session report

Richard Watson

AI development has rapidly advanced, leading to a faster and more accessible IT landscape. This development has made IT more accessible to individuals and organizations alike. However, this rapid progress has also raised concerns regarding the associated threats that come with AI technology.

One of the primary concerns is the potential for AI to enhance the authenticity of malware and enable the creation of deepfakes. Malicious actors can leverage AI-powered techniques to create sophisticated and realistic cyber threats, which can pose significant risks to individuals and businesses. Deepfakes, in particular, have the potential to undermine trust and integrity by manipulating and fabricating audio and video content.

Businesses are increasingly incorporating AI into their operations, but many struggle to effectively govern and monitor its use. This poses a challenge, as the gap between the utilization of AI and the capabilities of IT and cybersecurity to manage it can result in vulnerabilities and risks. Data poisoning is a specific concern, as it can have adverse effects on critical business processes by deliberately targeting and manipulating datasets used in AI models.

The governance and risk management frameworks need to be updated to effectively handle the complexities of AI in business settings. Organizations must address the unique challenges posed by AI in terms of privacy, accountability, and ethics. Furthermore, the integrity of the data used to train AI models is crucial. AI models are only as good as the data they are trained on, and any biases or errors in the data can produce flawed and unreliable results.

Establishing trust in AI models is also vital. Many individuals have concerns about the use of AI and are hesitant to trust companies that heavily rely on this technology. The ability to explain AI decisions, protect data privacy, and mitigate bias are essential to building this trust.

Furthermore, there are concerns about surrendering control to AI technology due to its immense knowledge and fast assimilation of new information. People worry about the potential misuse of AI in areas such as warfare and crime. Policy measures, such as President Biden’s executive order, have been introduced to address these risks and manage the responsible use of AI.

The field of AI and cybersecurity faces a significant talent gap. The demand for skilled professionals in these areas far exceeds the available supply. This talent gap presents a challenge in effectively addressing the complex cybersecurity threats posed by AI.

To tackle these challenges, organizations should create clear strategies and collaborate globally. Learning from global forums and collaborations can help shape effective strategies to address the risks and enhance cybersecurity practices. Organizations must take proactive steps and not wait for perfect conditions or complete knowledge to act. Waiting can result in missed opportunities to protect against the risks associated with AI.

Integration of AI is necessary to combat the increasing volume of phishing attacks. Phishing attacks have seen a substantial increase, and AI can play a crucial role in detecting and preventing these attacks. However, operating models must be transformed to ensure effective integration of AI, ending with human involvement for a thorough and closed-loop activity.

AI and generative AI have the potential to frustrate criminals and increase the cost of their activities. By utilizing AI technology, criminal activities can become more challenging and costly to execute. For example, applying AI and generative AI can disrupt the metrics and cost-effectiveness of certain criminal operations, such as call centre scams.

In conclusion, while AI development has brought significant advancements and accessibility to IT, there are numerous challenges and risks associated with its use. These challenges include the authenticity of cyber threats, governance and monitoring issues, data integrity, trust-building, talent gaps, control concerns, and the potential misuse of AI. Organizations must address these challenges, develop effective strategies, collaborate globally, and integrate AI into their operations to ensure cybersecurity and responsible use of AI technology.

Dr. Yazeed Alabdulkarim

The analysis highlights the escalating threat of cyber attacks and the challenges faced by cybersecurity defenses. This is supported by the fact that 94% of companies have experienced a cyber attack, and experts predict an exponential growth in the rate of cyber attacks by 2023. Cybercrimes are adopting Software-as-a-Service (SaaS) models and leveraging automation technology to scale their attacks. The availability of Malware as a Service in the cybercrime economy further strengthens their ability to carry out attacks at a larger volume and faster pace.

Generative AI is identified as a potential contributor to the intensification of the cyber attack situation. It is suggested that Generative AI could be used to create self-adaptive malwares and assemble knowledge useful for physical attacks. This raises concerns about the future impact of Generative AI on cybersecurity.

There are differing stances on the regulation of Generative AI. Some argue for limitations on its use, citing the belief that the rise of cyber attacks is due to the use of Generative AI. On the other hand, there are proponents of utilizing Generative AI for defense and combating its nefarious uses. They believe that considering threat actors and designing based on the attack surface can help leverage Generative AI for defensive purposes.

Disinformation is identified as a significant issue associated with Generative AI. The ability of Generative AI to generate realistic fake content raises concerns about the spread of disinformation and its potential consequences.

On a positive note, Generative AI can be used to analyze and respond to security alerts. It is suggested that employing Generative AI in this way can help speed up defensive measures to match the increasing speed of cyber attacks. Furthermore, it is argued that limiting the use of AI technology in cybersecurity would be counterproductive. Instead, AI can play a crucial role in fully analyzing security alerts and addressing the two-speed race in cybersecurity.

The analysis also highlights the incorporation of AI elements in emerging technologies. It is predicted that upcoming technologies will incorporate AI components, indicating the widespread influence of AI. However, there are concerns that fundamental threats associated with AI will also be present in these emerging technologies.

Understanding how AI models operate is emphasized as an important aspect in the field. The ability to explain AI models is crucial for addressing concerns and building trust in AI technology.

Watermarking on AI output is proposed as a potential solution to distinguish real content from fake. It is suggested that both AI companies and authorities should establish watermarking systems to ensure the reliability and authenticity of AI-generated content.

In conclusion, the analysis reveals the growing threat of cyber attacks and the need for stronger cybersecurity defenses. The impact of Generative AI on this situation is a subject of concern, with its potential to intensify attacks and contribute to the spread of disinformation. The regulation and use of Generative AI are topics of debate, with arguments made for limitations as well as for leveraging it in defense and combating nefarious activities. The incorporation of AI elements in emerging technologies raises both opportunities and concerns, while the understanding of AI models and the need for explainable AI should not be overlooked. Finally, watermarking on AI output has the potential to differentiate real content from fake and enhance reliability.

Dr. Victoria Baines

Data poisoning and technology evolution have emerged as significant concerns in the field of cybersecurity. Data poisoning refers to the deliberate manipulation of training data to generate outputs that deviate from the intended results. This form of attack can be insidious, as it slowly corrupts the learning process of machine learning models. Furthermore, influence operations have been conducted to spread discord and misinformation.

The rapid evolution of technology, particularly in artificial intelligence (AI), has created new opportunities for cybercriminals to exploit. AI has led to the replacement of humans with non-human agents in various domains, causing disruptions and potential threats. People have found ways to make bots go bad, and large language models have been repurposed for writing malware. This highlights the need for vigilance in harnessing technological advancements, as they can be exploited for malicious purposes.

The emergence of AI has also resulted in an evolution of cyber threats. Malware implementation has seen new methods and techniques, such as gaming AI models. The ecosystem of cybercriminals may undergo changes due to AI advancements, necessitating proactive measures to counter these evolving threats.

However, not all is bleak in the world of cybersecurity. AI and automation can play a vital role in alleviating the scale and stress issues faced by human operators. The current volume of alerts and red flags in cybersecurity is overwhelming for human teams. A 2019 survey revealed that 70% of cybersecurity executives experience moderate to high stress levels. AI can assist in scaling responses and relieving human operators from burnout, enabling them to focus on tasks they are proficient in, such as threat hunting.

It is worth noting that public perception of AI is often shaped by dystopian depictions in popular culture. The portrayal of AI in science fiction and dystopian narratives tends to create a negative perception. Interestingly, people are more inclined to show positivity towards “chatbots” rather than “Artificial Intelligence”. This demonstrates the influence of popular culture in shaping public opinion and highlights the need for accurate and balanced representation of AI in media.

In conclusion, data poisoning and technology evolution present significant challenges in the field of cybersecurity. The deliberate manipulation of training data and the exploitation of rapid technological advancements pose threats to the integrity and security of systems. However, AI and automation offer promising solutions to address scalability and stress-related issues, allowing human operators to focus on their core competencies. Moreover, it is important to educate the public about AI beyond dystopian depictions to foster a more balanced understanding of its potential and limitations.

Alexandra Topalian

A panel discussion was recently held to examine the cyber threats and opportunities presented by generative AI in the context of cybersecurity. The panel consisted of Richard Watson, a Global Cyber Security Leader at EY, Professor Victoria Baines, an Independent Cyber Security Researcher, Kevin Brown, the Chief Operating Officer at NCC Group, PLC, and Dr. Yazid Al Abdelkarim, the Chief Scientist of Emerging Technologies at CITE. Throughout the discussion, the participants highlighted the potential risks associated with the use of artificial intelligence (AI), specifically generative AI, in the cyber world.

One of the key points discussed during the panel was the emergence of new cyber threats arising from AI. Richard Watson, an EY consultant, stressed the importance of identifying these risks and provided examples of how generative AI can be employed to produce various types of content such as visuals, text, and audio. The panelists also acknowledged the potential danger of data poisoning in relation to generative AI.

Professor Baines echoed Watson’s concerns about data poisoning, emphasising its significance in her research. She also delved into the evolving nature of cyber crimes as new technologies, like generative AI, continue to advance. The panelists then proceeded to explore how cyber criminals can exploit generative AI to develop more sophisticated and elusive cyber threats. They highlighted the potential convergence of generative AI with social engineering tactics, such as phishing, and how this combination could amplify the effectiveness of manipulative attacks.

Dr. Yazid Al Abdelkarim shed light on the scale of cybersecurity attacks and the impact of generative AI. He stressed the need for regulation and shared insights on how SAIT advises organizations on staying ahead of cyber threats. The panelists discussed the challenges, including a talent gap, associated with implementing effective strategies for early detection and management of cyber threats. Kevin Brown shared real-life incidents to illustrate how organizations tackle these challenges.

The threat of deepfakes, where AI-generated content is used to manipulate or fabricate media, was another topic explored during the panel. The participants discussed strategies for addressing this type of threat, with a focus on early detection. They also touched on the ethical boundaries of retaliating against cyber attackers based on psychological profiling, highlighting the importance of complying with the law.

Regarding opportunities, the panelists agreed that generative AI offers benefits in the field of data protection and cybersecurity. Professor Baines emphasized the potential positive aspects of generative AI, highlighting opportunities for enhanced cybersecurity and protection of sensitive information.

In conclusion, the panelists acknowledged the lasting impact of generative AI on the landscape of emerging technologies and its growing influence on cybersecurity. They recognized the advantages and challenges brought about by generative AI in the field. The discussion underscored the need for effective regulations, risk management approaches, and cybersecurity strategies to address the evolving cyber threats posed by generative AI.

Kevin Brown

Generative AI, a powerful technology with various applications, is now being used for criminal activities, leading to concerns about its negative impacts on cybersecurity and criminal behavior. One key concern is that generative AI is lowering the barrier for criminals to exploit it. This means that criminals can easily leverage generative AI for illicit activities, making it more challenging for law enforcement agencies and organizations to prevent and mitigate cybercrime.

Another major concern is that criminals have an advantage over organizations when it comes to adopting new AI technologies. Criminals can quickly launch and utilize new AI technologies without having to consider the regulatory and legal aspects that organizations are bound by. This first-mover advantage allows criminals to stay one step ahead and exploit AI technologies for their nefarious activities.

The emergence of technologies like deepfakes has also brought in a new wave of potential cyber threats. Deepfakes, which are manipulated or fabricated videos or images, have become more accessible and can be utilized in harmful ways. This poses a significant risk to individuals and organizations, as deepfakes can be used for social engineering attacks and to manipulate public opinion or spread misinformation.

Moreover, the use of large language models in artificial intelligence has raised concerns about data poisoning. Large language models can be manipulated and poisoned, leading to a range of malicious motivations. This poses a threat to the integrity and reliability of AI systems, as attackers can exploit vulnerabilities in the data used to train these models.

Additionally, generative AI has the potential to amplify the effectiveness of phishing and manipulative attacks. By using generative AI, criminals can increase the volume and quality of phishing attempts. This allows them to create phishing messages that are highly professional, relevant, and tailored to the targeted individual or business. As a result, generative AI professionalizes phishing, making it more difficult for individuals and organizations to detect and protect themselves against such attacks.

In conclusion, the increased use of generative AI for criminal activities has raised significant concerns about cybersecurity and criminal behavior. The technology has lowered the barrier for criminals to exploit it, giving them an advantage over organizations in adopting new AI technologies. Furthermore, the accessibility of technologies like deepfakes and the potential for data poisoning in large language models have added to the complexity of the cybersecurity landscape. Additionally, generative AI has the potential to amplify the effectiveness of phishing and manipulative attacks, making it harder to detect and defend against such cyber threats. It is crucial for policymakers, law enforcement agencies, and organizations to address these concerns and develop strategies to mitigate the negative impacts of generative AI on cybersecurity.

Speakers