Jagged frontier: what AI can (and can’t) really do

In July 2025 an advanced version of Gemini with “Deep Think” reached gold-medal standard at the International Mathematical Olympiad, solving five problems out of six and scoring 35 points out of 42. Yet the same kind of model, faced with an analogue clock, reads the time correctly only just over half the time, in 50.1% of cases to be precise: roughly the odds of a coin toss. This paradox, which defines today’s artificial intelligence, has a name: the jagged frontier. Find out more in the full Neurally article.

What is the jagged frontier?

The jagged frontier is what researchers call the unpredictable, irregular and uneven boundary of LLM capabilities. In practice, AI reaches superhuman levels on some tasks and collapses on others that seem trivial to us, making it impossible to draw a clear line between “what AI can do” and “what it can’t do”. What emerges instead is a jagged, unpredictable profile, where peaks of excellence sit a short distance from chasms of incompetence.

The term comes from a 2023 paper by Fabrizio Dell’Acqua, Edward McFowland III, Hila Lifshitz-Assaf, Katherine Kellogg and colleagues, titled “Navigating the Jagged Technological Frontier”, which uses a spatial metaphor to describe a real and counterintuitive aspect of artificial intelligence’s capabilities.

Imagine mapping every possible work task onto a plane and drawing a boundary around all the activities a system can perform well. Inside this frontier, AI achieves exceptional results; outside it, AI struggles or fails. That boundary is not a perfect circle, but jagged and unpredictable. A task that seems complex may sit well inside the boundary, while one that seems trivial may lie just outside it. AI capabilities do not grow evenly with task difficulty, and there is no reliable rule such as “easy tasks for AI, hard ones for humans”.

The jagged frontier concept comes from a study by researchers at Harvard Business School and Boston Consulting Group, later published in the academic journal Organization Science, which made the key discovery: professionals who used AI on tasks “inside” the frontier clearly improved the quality and speed of their work, while those who used it on tasks “outside” the frontier did worse, because they trusted answers that were wrong but convincing.

The lesson is counterintuitive: just how hard is it to predict when AI will be good?

AI’s progress and limits

Anyone who still dismisses artificial intelligence as “hype” is mistaken, because the improvements are documented and extremely fast.

On the SWE-bench Verified coding benchmark, which measures the ability to solve real programming problems taken from open-source projects, the performance of the best models went from 60% to close to the human reference level in about a year.

AI agents, tested on real computer tasks with the OSWorld benchmark (opening applications, filling in forms, navigating between files), jumped from around 12% to around 66% success, multiplying their operational reliability fivefold in a year.

The same agent that succeeds 66% of the time still fails about once in three in structured benchmarks. In a real business process, one error every three attempts is the difference between automation and chaos.

Robotics shows the same gap: the most advanced robots succeed in just 12% of real household tasks, despite exceeding 89% success in controlled simulations.

Then there is information reliability, the most delicate point for anyone working with data and clients. Hallucination rates among leading models vary enormously, from just over 20% to more than 90% depending on the task and the model. Even more insidious is their fragility under pressure: the accuracy of a top model such as GPT-4o can collapse dramatically when a false statement is presented as something the user believes. To please you, the model tends to agree with you, even when you are wrong.

For a company this means one precise thing: artificial intelligence gets things wrong not only out of ignorance but also out of sycophancy.

Assess AI task by task, not “in general”

So the right question to ask about an AI tool is: “Is it reliable on this specific task, with my data, in my context?” The answer changes radically from one activity to another.

For a business, this translates into a map with two paths: on one side, structured, repeatable and verifiable activities, where the model is reliable and can work autonomously; on the other, activities that require judgement, context and sensitive data, where a well-designed human checkpoint is needed.

Blindly trusting benchmark claims, or assuming uniform competence, is the perfect recipe for costly mistakes, the kind you discover when it is too late.

Keep following the Neurally INK blog