The AI coding tutor paradox grows as educators scramble to rethink how they test real skills

The AI coding tutor paradox grows as educators scramble to rethink how they test real skills

Key Points

- A survey of more than 700 computer science educators from 49 countries, conducted by an ACM task force, found that 69 percent believe AI has changed the skills needed for software development.

- Sixty-four percent have already shifted their teaching away from writing code from scratch toward code comprehension, debugging, and problem-solving.

- Assessment methods are changing even faster: 68 percent of respondents have adjusted how they test students, with more proctored in-person exams, oral exams, code defense sessions, and project-based work.

A survey of more than 700 educators worldwide finds that most have already changed their teaching and assessment methods because of AI. Oral exams, code comprehension, and project-based work are gaining ground. But nearly half of respondents say they lack proven examples for integrating AI into their courses.

AI coding tools like ChatGPT and GitHub Copilot can solve typical programming assignments in introductory courses, often scoring at the level of an average student. A new study now offers a broad look at how computer science educators around the world are responding. It was conducted by the "ACM Task Force on Generative AI and Programming Assessment," an expert group set up by the education committee of the international computing society ACM.

The task force surveyed 763 educators from 49 countries between May and October 2025. About 500 nearly complete questionnaires were analyzed. The findings show that 69 percent believe AI has changed the skills needed for software development. The top concern among educators is growing student dependency on technology (87 percent), followed by cheating and plagiarism (72 percent).

Teaching shifts from writing code to understanding it

Sixty-four percent of respondents have already changed how they teach. A thematic analysis of open-ended answers points in one direction: away from writing code from scratch, toward code comprehension, debugging, and problem-solving. Thirty-nine responses also mentioned teaching AI use and prompt engineering as explicit topics. Others demonstrate AI tools in class to show both their capabilities and their limits.

"We know that students will likely be developing with AI tools when they enter the workforce," said Steven Gordon, a professor at Ohio State University and lead author of the report. "The key is to produce graduates who are competent in programming and who understand both the capabilities and limitations of AI."

Oral exams and project-based assessments are gaining ground

Changes in assessment are even more striking. Sixty-eight percent of respondents have adjusted their testing methods. The most common shifts include more proctored in-person exams (56 mentions), reduced weight for homework (38), oral exams and code defense sessions (36), and paper-based tests (35). Project-based assessments (34) are also on the rise. Some educators now require students to disclose their AI use and submit logs of their interactions with AI tools.

Institutional policies remain a patchwork. Forty-five percent of respondents said their institution has guidelines on AI use, while 39 percent said it doesn't. Policies range from outright bans to explicit permission or encouragement, as long as usage is documented.

Nearly half of educators lack best-practice examples for AI integration

The biggest barrier, cited by 48 percent of respondents, is a lack of proven best practices. Twenty-eight percent say they don't have enough expertise with AI, while 20 percent see no need to integrate it into their teaching at all. When it comes to professional development, 74 percent want training on effective teaching methods and 66 percent want help redesigning assessments.

The results come with some caveats. Participants were recruited mainly through ACM and SIGCSE mailing lists and selected national educator directories. Respondents skewed heavily toward North America (227) and Europe (106), while Asia (57), South America (10), Oceania (9), and Africa (3) were far less represented. Among those who listed their institution type, 77 percent work at universities, with K-12 schools and vocational programs barely represented. The findings are also based on self-reported data.

The ACM study is freely available online. The task force also set up a website collecting practical examples and resources for educators.

Studies show AI use boosts grades but tanks actual learning

Several recent studies back up the concerns raised by respondents. An Anthropic study found that software developers who learned a new Python library with AI assistance scored 17 percent lower on a follow-up knowledge test than a control group without AI access. A small subset that fully delegated coding to AI finished tasks the fastest but averaged just 39 percent on the knowledge test.

Similar patterns show up beyond computer science. A Chinese longitudinal study tracking more than 26,000 students found that AI use boosted homework grades by 18 percent and cut completion time by 30 percent. But on closed-book exams after six months, scores dropped by 20 percent. On later entrance exams, the decline was 18 and 24 percent, depending on the test.

A UC Berkeley analysis covering more than 500,000 grades found a sharp rise in top marks after ChatGPT's release, especially in courses heavy on writing and programming assignments. The spike was most pronounced in courses where unproctored homework counted heavily toward the final grade. That's the exact type of assessment that many educators are now weighting less, according to the ACM report.

AI News Without the Hype – Curated by Humans

Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.

Subscribe now