‘It Invented the Feathers’: A Week of What AI Still Gets Wrong

The failures that make the news are the dramatic ones: the agent that went rogue, the chatbot that said something outrageous. The failures that fill working developers’ days are smaller and, in their way, more revealing. This week a Hacker News thread asked a deceptively simple question — what is one simple thing you repeatedly ask a model to do that it still somehow messes up? — and the answers formed an unusually honest inventory of where fluency and reliability come apart. Not “AI can’t cure cancer,” but “AI still can’t count the letters in a word, and says the wrong number with total confidence.”

This is a Voices piece: a round-up of what real users said, in their own words, over the past week. Quotes sourced from: Hacker News. We opened each thread and lifted the wording verbatim; every quote is linked in the Sources list at the foot, with the handle, platform and date. As always, we kept the people who pushed back in the room, because a catalogue of complaints with the counter-arguments deleted is just a mood.

Confidently wrong about small things

The single most quotable failure of the week was not a gap in knowledge but a refusal to admit one. busyant described uploading photographs of birds and getting egregious mistakes on roughly one in ten — which would be forgivable, except for what the model did next. That is our moan of the day.

The wrong identification is a shrug. The invented anatomy is the problem. A model that said “I’m not sure, the key features aren’t visible” would be a useful tool; a model that fabricates the evidence to support its guess is an actively misleading one. It is the same pathology tartoran summed up in a single line: models are “bad at not inventing stuff (hallucinating facts, sources etc), they’re also bad at not over explaining, remembering details reliably, asking the right question and avoiding repetition.” Every item on that list is a small thing. Together they are the whole reliability problem.

And it reaches the genuinely trivial. slake noted, with some amusement, that “advanced models [are] bad at calculating the length of a string” — the sort of task a line of code has solved for fifty years, still fumbled by a system that can pass a bar exam. It is a useful reminder that these tools are not reasoning about characters the way you assume; they are predicting text, and sometimes the prediction of “how many letters” is just wrong.

It read the instructions. It ignored them.

For people who code with these tools all day, the sharpest irritation was not hallucination but disobedience. jgb1984 described keeping a modest set of rules in a project instructions file and watching the model trample them anyway: “Not a day goes by where Claude Opus violates one or several of the instructions.” When the explicit, written, right-there-in-the-file rules are optional, the promise of an agent that follows your conventions starts to look thin.

nilsherzig caught a subtler version — the model that does the task and narrates its own failure to do it: asking it to remove an idea from a document “often times results in an edit which explicitly states that this Idea is not relevant, instead of just removing all references to that Idea.” It did not delete the thing; it added a note saying the thing was deleted. And GuestFAUniverse pointed at the memory problem underneath a lot of this: ChatGPT “assumes too much from former conversations… It rarely asks for clarification instead of assuming too much.” A tool that guesses rather than asks is a tool you have to double-check, which is not the labour saving that was advertised.

The taste problem

A second cluster was about aesthetics and judgement — the things that are hard to score on a benchmark and therefore, apparently, hard to train. Swankivo, working on slide generation, found that “one thing general LLMs can’t do is leave empty space. Whitespace is the core of design that actually feels good, but they keep trying to add ‘distinctive design elements’.” mstaoru extended the list to “those cringey overstuffed presentation slides… Humor, cliffhangers, drama, anything subtle. Anything spatial or mechanical that is novel.” The pattern is a model that fills space because filling space is what it has seen, with no sense of when less would be more.

The same blind spot shows up in plain text. kanzure put it flatly: the models “seem to be bad at writing prose or text. Many of the sentence structures seem to be unvaried.” And ghostpepper described watching a model brute-force its way through search because “they don’t generate keyword search queries very well… if you watch what they search you will cringe.” Individually, small. Collectively, the texture of working with something that is confident and tin-eared at once.

There is a reason this cluster is so persistent, and it is not laziness on the labs’ part. The skills people are complaining about — restraint, taste, knowing when to leave a thing out — are exactly the ones that resist a benchmark. You can score whether an answer is factually right; you cannot easily score whether a slide breathes or a paragraph has rhythm. What gets measured gets optimised, and what cannot be measured gets whatever is left over. The upshot is a tool that keeps climbing the charts on the things that are testable while quietly stalling on the things that are merely tasteful — which happen to be the things a human notices first.

Is it getting worse?

The most contested thread of the week was whether the quality is actively sliding. SalariedSlave reported a growing unease: “these output patterns are getting worse. It’s not just Claude… Their writing can get borderline incomprehensible,” and wondered aloud whether newer models are “ingesting too much AI content.” That is a perception, not a measurement, and we flag it as such — but it is a widely shared one, and it rhymes with a real, documented worry we have written about in our explainer on model collapse.

Domain specifics piled up alongside the general unease. sandcat_ reported “constant mistakes and hallucinations” asking for video-game tips, “even in really well documented games.” wiper88 hit the same wall in a different game, finding that when asked about arena tactics or which enchant to pick, the model “makes a lot of mistakes in regards to which spells or enchants are available in which phase or expansion.” The pattern in both is instructive: these are topics with excellent public documentation, and the model still confidently mangles the specifics. Good source material is not the same as a reliable answer, because the model is not looking anything up so much as remembering an averaged impression of it.

And in the week’s other running gripe, 8-prime voiced a frustration shared by many phone owners: “Google absolutely jumped the gun with their push to Gemini over Google Assistant,” with some things “simply [not] work[ing] with Gemini because it doesn’t ‘know’ that it should use certain device functions.” A newer, cleverer model that cannot reliably do what the older, dumber one did is a regression the marketing never mentions, and it is a particularly bitter one when the downgrade is forced rather than chosen. Even the coding-tool churn of the week made the list, with throw03172019 declaring one rival “a huge joke” that “consistently destroys my Clojure files.” The through-line across all of it is not incapacity — these models can do remarkable things — but inconsistency you cannot predict, which is its own kind of cost.

To be fair: some of this is us

Now the other half of the room, because it turned up and it had a point. Not everyone thinks these are defects; some think they are misuse. mojuba made the case that people sabotage the models with their own prompts: “the shorter the better, let the model figure out the rest. Overinstruction degrades intelligence.” It is a genuine insight. A lot of the frustration with these tools is a skill gap in wielding them, and the instinct to pile on ever-more-elaborate instructions really can make the output worse rather than better.

But the concession only stretches so far. “Count the letters in this word” is not an over-engineered prompt. “Follow the three rules in this file” is not a failure of prompt craft. When a tool sold as broadly, generally capable stumbles on tasks a child or a one-line script would clear, “you’re holding it wrong” is only part of the answer. And there is a limit to how far the burden can be shifted onto the user: if the correct way to use a general-purpose assistant is to know in advance the narrow set of things it quietly fails at, then the generality was oversold. The fair reading is that these are extraordinary tools with a specific, persistent weakness: they are unreliable in exactly the small, checkable ways that are easiest to catch and most corrosive to trust.

What to take from the week

If there is a practical posture in all of this, it is to keep using the models for what they are good at while refusing to extend them credit they have not earned:

• Verify the small, checkable things. The confident wrong answer to an easy question is the most dangerous kind, because it is the most plausible. If you can check it in ten seconds, check it.
• Assume instructions are suggestions. If a rule matters, do not trust the model to honour it silently — build a check that catches the violation rather than hoping it won’t happen.
• Watch for invented detail. A model that supplies specifics it could not know — feathers it never saw, a source that does not exist — is not being helpful; it is being fluent. Treat unrequested specificity as a flag, not a bonus.
• Right tool, right task. A one-line script counts characters perfectly. Handing every trivial job to a probabilistic text engine is how you turn a solved problem back into an open one.

None of this is a case against the tools. The people quoted here use them every day, which is exactly why they notice the seams. The gap they keep describing is the one between how these systems sound and how far you can actually trust them — a gap we have measured before in the hallucination problem that is still not solved and in the users who noticed their AI sounding smart while answering worse. Fluency was the easy part. Being right about the small things, reliably, without inventing the feathers, is turning out to be the hard one.