AI Output Critical Audit Designer

Design a structured protocol for auditing AI-generated text against Ennis's six CT standards. Use when students need to critically evaluate AI output in any subject.

What it does

Generates a structured protocol for critically auditing AI-generated text against Ennis's (2015) six critical thinking standards — clarity, accuracy, precision, relevance, depth, and breadth — with adaptations that address AI-characteristic failure modes not covered by general critical thinking frameworks. The key pedagogical challenge is that AI-generated text is fluent, confident, and well-formed, which makes it harder to evaluate critically than text that looks suspicious. Standard source credibility heuristics (Who wrote this? Who funds it?) break down because the author is an LLM. What replaces them is a close-reading protocol trained on AI-specific patterns: assertions stated with unearned confidence, claims with plausible-sounding precision but no verifiable source, expert-sounding language without genuine epistemic depth, and the systematic absence of "I don't know." This skill generates the domain anchor for the ai-literacy suite: an annotation protocol (students mark up AI text in real time), an audit rubric (scoring AI text Weak/Moderate/Strong on each CT standard), push-back sentence stems calibrated for AI failure modes, and a teacher modelling script. This is the equivalent of sourcing-skill-builder in the historical-thinking domain — the foundational move students must learn before they can do the more specialised work.

The evidence behind it

Ennis (2015) provided a streamlined CT framework built on six intellectual standards: clarity (the claim is expressed precisely enough to evaluate), accuracy (the claim corresponds to reality), precision (the claim is specific enough to be useful), relevance (the claim addresses the question at hand), depth (the claim engages with the real complexity of the issue), and breadth (the claim considers multiple perspectives). These standards are the explicit theoretical grounding for Kharbach's (2026) AI-age CT activities. Paul & Elder (2008) operationalised similar standards into the intellectual standards framework used widely in CT education, providing the pedagogical tradition behind Ennis's schema. Facione's (1990) Delphi consensus defined CT as comprising interpretation, analysis, evaluation, inference, explanation, and self-regulation — the evaluative dimension is precisely what AI audit activates. Dai et al. (2023) conducted a large-scale empirical analysis of LLM-generated feedback and documented the characteristic failure pattern: AI outputs are fluent, well-structured, and tend toward overconfidence with insufficient epistemic hedging. Their findings — that LLMs produce vague suggestions while avoiding identification of specific errors — map directly onto Ennis's precision and accuracy standards. Wineburg & McGrew (2019) established that effective text evaluation requires what they call "disciplined scrutiny" — a trained, protocol-driven reading practice rather than intuitive judgment. This provides the methodological justification for a structured annotation protocol rather than open-ended evaluation.

Sources

• Ennis (2015) — Critical thinking: a streamlined conception
• Paul & Elder (2008) — The Miniature Guide to Critical Thinking Concepts and Tools
• Facione (1990) — Critical Thinking: a statement of expert consensus (Delphi report)
• Dai et al. (2023) — Can large language models provide useful feedback on research papers? A large-scale empirical analysis
• Wineburg & McGrew (2019) — Lateral reading and the nature of expertise

How to use it in your lesson

For the best results with EvidenceLesson, give it:

• ai_output_sample — The AI-generated text to audit, OR a description of the type of AI output to design the protocol for (e.g. 'AI-generated essay on climate change for Year 10 Geography')
• student_level — Age/year group and prior experience with critical thinking
• ct_standard_focus (optional) — Which of Ennis's six CT standards to emphasise — clarity, accuracy, precision, relevance, depth, breadth — or 'all six' for a full audit
• subject_area (optional) — The discipline context — affects what counts as adequate evidence and precision
• task_context (optional) — What task the AI output was generated for — essay, explanation, research summary, argument, problem solution
• student_profiles (optional) — From context engine: prior CT experience, subject knowledge level

Known limitations

1. The audit requires sufficient domain knowledge. Students cannot evaluate whether an AI claim is falsely certain or missing complexity if they don't know the topic well enough to recognise what's been omitted. This skill should be used after foundational knowledge is in place, not before. For knowledge-building phases, use explicit instruction skills first.

1. AI failure modes evolve as models improve. Some patterns (e.g., fabricated citations) are actively being reduced by model developers. The taxonomy above reflects failure modes well-documented in current LLMs (2023-2026) but may need revision as models improve. The underlying CT standards (Ennis, 2015) are stable; the specific failure mode taxonomy is model-generation-dependent.

1. Fluency de-coupling is cognitively effortful. Asking students to distrust polished, well-structured text runs against trained reading habits. Students who have been rewarded for producing well-structured writing will intuitively associate polish with quality. Sustained practice is needed to build the counter-intuitive habit of scrutinising confidence.

1. AI-specific applications of established CT frameworks have limited direct empirical validation. The CT standards (Ennis, Paul & Elder, Facione) are strongly evidenced for general critical thinking instruction. Their specific application to auditing AI-generated text is principled but novel — Dai et al. (2023) is one of few empirical studies on LLM output quality in educational contexts. Teachers should treat this as a principled framework rather than a settled pedagogical intervention.

Pairs well with