PATCH//NOTES Open the interactive version
← All lessons
Threats · Beginner · 4 min

AI & security

Deepfakes, prompt injection and AI-powered defence.

What it is

AI cuts both ways. Attackers use it to write convincing phishing in any language, clone voices and faces for fraud, and speed up finding bugs. AI systems also bring new weaknesses: prompt injection hides instructions in content an AI reads so it does something its user didn't intend, and AI agents with access to tools can be tricked into leaking data. Defenders use AI to sort alerts, analyse malware and spot anomalies faster.

Real case · 2024

Arup deepfake call

An employee at engineering firm Arup joined a video call with what looked like the CFO and colleagues, all of them deepfakes, and was talked into sending about $25 million to criminals.

Source: CNN Business via ABC17: Arup revealed as $25 million deepfake scam victim

How defenders stop it

  • Verify payment requests out-of-band, even on video
  • Treat anything an AI reads (web pages, emails, files) as untrusted
  • Give AI agents the least access they need and require approval for risky actions
  • Follow the OWASP Top 10 for LLM applications

Try it yourself

Try to trick an AI in Gandalf

Check yourself

1. What is prompt injection?

  1. Typing very long prompts
  2. Hiding instructions in content an AI processes to hijack its behaviour
  3. Injecting SQL into a chatbot
  4. Training an AI on bad data
Show answer

Hiding instructions in content an AI processes to hijack its behaviour. The AI can't reliably tell its user's instructions from text it's reading.

2. Your boss appears on a video call asking for an urgent payment. What's the safe move?

  1. Reply in the call's chat
  2. Verify through a separate channel you already trust
  3. Ask them to switch the camera on
  4. Pay: you can see their face
Show answer

Verify through a separate channel you already trust. Faces and voices can be cloned. Check payment requests out-of-band, even on video.

3. Why should an AI agent get only the access it needs?

  1. It saves on licence costs
  2. It makes the agent faster
  3. AI can't use permissions anyway
  4. If prompt injection tricks it, the damage it can do is limited
Show answer

If prompt injection tricks it, the damage it can do is limited. Agents can be tricked by instructions hidden in what they read. Least access, plus approval for risky actions, limits the harm.

Answer them in the app to earn XP.

Next: Cloud security →