Testing AI Models as Magic Boxes: Why Checks Fail - Michael Bolton
From Software Testing Unleashed - QA, DevEx & Quality Engineering · 28:07
Show by Richard Seidl | Software Development & Testing Expert
Testing AI models means testing a magic box: nobody has the source code, and the behavior is neither controlled nor understood. Michael Bolton calls that strongly magical, and with him I talk about what follows from it. His principles are simple: if something matters very much, don't depend on magic, and if you use it anyway, responsibility demands that you test it and maximize testability. Simple output checks don't stand a chance against a language model, so testing becomes experiencing, exploring and experimenting, like peeling an onion. And Penn and Teller's clear plastic cups show why we fall for these models in the first place: the insight we see in their text comes from us.
More episodes of Software Testing Unleashed - QA, DevEx & Quality Engineering
More episodes
- GenAI Evaluation: One Model, Three Goals - Deepshikha, Anamika Mukhopadhyay
- Contract-Driven Testing in Banking: Realistic Mocks - Phil Royston
- Building Trust in Teams: Start Small - Michael Kutz, Linda Van De Vooren
- Mental Skills: Testers Can't Outsource Their Brain to AI - Ard Kramer
- Dark Side of Testing: The Failures We Cause Ourselves - Bart Knaack
- Quality Hunting: Testing for Information Instead of Bugs - Rik Marselis
- On-Site Testing Conferences: Why Networking Needs Presence - Okan Çakmak
- You Need a Test Automation Strategy Before AI - Michaël Pilaeten
- Agentic Systems in Testing: Why Humans Stay in the Loop - Nishan Portoyan
- Welcome to Software Testing Unleashed
PodSearch is a free, independent podcast search engine. Read about PodSearch, the privacy policy or contact us.