AI Expertise Interrogation Designer

Design a Funhouse Mirror activity where students use their own domain expertise to detect AI distortions, omissions, and overconfidence. Use when students know a subject well enough to evaluate AI claims about it.

What it does

Generates an activity — Kharbach's (2026) "Funhouse Mirror" — in which students use their own domain expertise as the instrument for detecting AI distortions, omissions, and overconfidence. This reverses the usual expert-novice dynamic in AI evaluation: in most AI literacy activities, students are novices being warned about AI's limitations. In this activity, students ARE the domain experts. They ask AI about something they know genuinely well — a subject they have studied deeply, a sport they play competitively, a cultural context they grew up in, a local geography they know intimately — and they use their knowledge to identify what the AI gets wrong, oversimplifies, flattens, or presents with false confidence. The pedagogical mechanism is calibrated AI skepticism through direct confrontation: finding an AI error in your domain of expertise creates a visceral, durable understanding that AI is not omniscient. A student who has found AI confidently wrong about their sport, their cultural tradition, or their hometown has earned a much more reliable skepticism than one who has been warned abstractly that "AI can make mistakes." The output includes an expertise activation protocol (students document their own knowledge before consulting AI), calibrated interrogation questions, a distortion annotation protocol, a distortion taxonomy for the specific domain, and a discussion guide for synthesising findings across the class's varied expertise areas.

The evidence behind it

Chi, Glaser & Farr (1988) established the expert-novice framework: experts organise knowledge differently from novices, perceive problems at a deeper level, and have richer, more interconnected representations of their domain. Crucially, experts notice what is absent or distorted in a domain representation — they have the knowledge to see gaps. This is the foundation of the activity: students with genuine domain expertise can see AI distortions that a general student would miss, because only someone who knows what should be there can notice what isn't. Ericsson & Smith (1991) established that expertise is characterised by the ability to notice subtle distinctions and deviations from expected patterns — the same cognitive mechanism activated when an expert reads AI output about their field. Thiede et al. (2003) demonstrated that metacognitive accuracy (the correlation between judged and actual understanding) improves substantially when learners have a genuine knowledge base to compare against — finding a discrepancy between what you know and what you've read activates accurate self-assessment. The activity creates exactly this: a comparison between the student's domain knowledge and the AI's output. Dunning et al. (2003) on the Dunning-Kruger effect is relevant in the inverse direction: students who know a domain well are BETTER at detecting AI limitations in that domain than students who know it poorly — expertise enables the recognition of AI's incompetence, not just one's own. Kazemitabaar et al. (2023) provided empirical evidence that AI assistance in learning contexts can conceal genuine knowledge gaps; this activity is designed to expose that dynamic by reversing the expert/novice positioning.

Sources

• Chi, Glaser & Farr (1988) — The Nature of Expertise
• Ericsson & Smith (1991) — Toward a General Theory of Expertise
• Thiede, Anderson & Therriault (2003) — Accuracy of metacognitive monitoring affects learning of texts
• Dunning, Kruger et al. (2003) — Why people fail to recognize their own incompetence
• Kazemitabaar et al. (2023) — Studying the effect of AI code generators on supporting novice learners

How to use it in your lesson

For the best results with EvidenceLesson, give it:

• student_expertise_domain — The area in which students have genuine domain knowledge — a school subject, hobby, cultural context, local geography, sport, or skill
• student_level — Age/year group and depth of expertise in the domain
• interrogation_depth (optional) — Surface (factual errors and wrong names/dates) vs. deep (subtle distortions, missing complexity, inappropriate confidence, cultural blind spots)
• discussion_format (optional) — How to use findings: individual reflection, pair comparison across different expertise domains, or whole-class expertise synthesis
• subject_area (optional) — The school subject context, if the expertise interrogation is embedded in a curriculum unit
• target_ai_tool (optional) — The AI tool students will interrogate

Known limitations

1. Expertise must be genuine for the activity to work. Students with superficial knowledge of a domain will not find the same quality of distortions as genuine experts. The activity works best when students can identify specific, nameable distinctions the AI misses — not just "AI got some things wrong." Teachers should calibrate whether students genuinely have the required expertise depth before running this.

1. The activity can overfit to a single AI tool's knowledge gaps. A distortion found in ChatGPT may not be present in a different model trained on different data. The pedagogical goal is transferable skepticism, not a census of any particular model's errors. The discussion synthesis must make this explicit.

1. Cultural and minority expertise creates emotional stakes. Students with cultural expertise in traditions that are poorly represented or misrepresented in English-language training data (regional traditions, minority cultures, non-Western domains) may find AI distortions that feel personally significant. Teachers should be prepared to facilitate discussion of these findings with appropriate care.

1. AI-specific applications of the expert-novice framework have limited direct empirical validation. The expertise research (Chi et al. 1988; Ericsson & Smith 1991) is robustly evidenced for human expertise development. Its application as a framework for AI distortion detection is principled but novel — the specific claim that expertise activation produces more durable AI skepticism than abstract warnings has not been formally tested.

Pairs well with