The defenses to this like, "Yo, it's natural language it's not a calculator."

It's running on a computer. It's not like it can't hand off a calculation. And if it can't, why isn't that built in? Or else, why isn't it going, "Sorry, that's a math problem and I can't do math. I am, alas, only a poor computer who can't do math."

You literally don't have to invent or hallucinate math at all. It doesn't have to engage higher pattern-matching functions other than to deduce, "Oh shit, you're asking a math question, let's do 10.12 < 10.6 and see if that's true or false."

This is the answer machine that is supposed to replace us and take our jobs, so we can absolutely criticize it when it confidently declares utterly and confidently wrong answers to stuff its "unintelligent" predecessors did in calculator form just fine.

@pjakobs Well …
A NUMBER 9.9 (read as 9.90) is bigger as 9.11

When it’s version Numbers? Sure 9.11 is the higher version compared to 9.9

If it’s a date?
In USA it would be September 9th vs 11th so 9.11 is higher.
In Germany it’s 9th of September vs 9th of November so 9.11 is higher.

LLMs are good in some things but not the answer to everything. Don’t expect it to do Math right without context. 🤷‍♂️

I'm probably trying to approach this the wrong way (trying to understand the cause of this error)

I don't get where the 0.21 result is coming from 🤯

Just for fun i asked ChatGPT the same question and now the answer is "correct" (it was wrong but it "corrected" itself)

Funny enough, when pressing it that it was wrong and the right answer was 0.21 I got this

I can see how a *language* machine can conclude that nine point *eleven* is bigger than nine point *nine*.

But not that substracting leads to 9.21. Why not 9.2?

I assume the guy who came up with the stochastic parrot metaphor is very embarrassed by it by now. I would be.

(Completely ignoring the deep concept building that those multi-layered networks do when learning from vast datasets, so they stochastically work on complex concepts that we may not even understand, but yes, parrot.)

I'm confused.
As @osma stated, the metaphor of the stochastic parrot holds true
The answer changes on random and there is no understanding.

Why should anyone be embarrassed of being right?

But you're evidently gullible enough to have fallen for the grifter's proposition that the text strings emerging from a stochastic parrot relate to anything other than the text strings that went into it in the first place: we've successfully implemented Searle's Chinese Room, not an embodied intelligence.

en.wikipedia.org/wiki/Chinese_…

(To clarify: I think that a general artificial intelligence might be possible in principle: but this ain't it.)

Agree. I'm more and more convinced that today's chatbots are just an advanced version of ELIZA, fooling the users and just appearing intelligent

en.wikipedia.org/wiki/ELIZA

I wrote a thread about it infosec.exchange/@realn2s/1117…

where @dentangle fooled me using the ELIZA technics

I'm not sure of the "difference"
Different in pure dimension for sure (molehill vs mountain).

On a higher level:

ELIZA used keywords with a rank which, together with the relations to the output sequences were hardcoded in the source.

LLM use tokens with a probability which, together with the relations to the output tokes sequences are determined though training data

Closing with a anecdote from the wiki page:

Weizenbaum's own secretary reportedly asked Weizenbaum to leave the room so that she and ELIZA could have a real conversation. Weizenbaum was surprised by this, later writing: "I had not realized ... that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people."

It absolutely does!

Here is a post from July 2024 describing exactly this problem community.openai.com/t/why-9-1…

I fail to be astonished or call something intelligent if fails to do correct math in the numerical range up to 10 (even after one year, many training cycles, ...)

I agree, the technics behind the training and the construction of the response are vastly different.

Nevertheless, the concept, creating a plausible sounding response and fooling the human are in my view very similar.

In a way that is very disappointing. 60 years later and an increase of computing power by a factor of billons the solution concept of "AI" is still pretending 😞

"maybe it's ok to polish a text that isn't too important" - My feeling is that if the text isn't too important, it doesn't need much polishing, and a human should do any polishing necessary anyway. Then later when the human has to polish text that is absolutely critical to get right, the human has had practice at polishing and does it well.

@airshipper @kevinriggle @oli

this is the use of generative ai that i have the most sympathy for, because ‘knowledge work’ in a second language is hard.

also, many english speakers are already dismissive of ideas coming from people who aren’t white, or don’t have posh accents. being able to write well is a good counter that.

Thought about this some more. No, I do not think that using the Chinese room thought experiment you will ever accept a computer program‘s behavior as intelligent, no matter how much evidence for its intelligent behavior you get. Because per definition there‘s an algorithm executing it.

I don’t agree, because I don‘t buy into the human exceptionalism that we meat machines have some magic inside of us that gives us intent the machines can‘t have.

I'm confused
I wrote that I "FAIL to be astonished"

You wrote about "astonishingly "intelligent" answers"

I just refuse to call a system AI or even just intelligent if it just a reproduction of patterns

people have been mistaking "statistics" and "algorithms" and "procedural generation" and "fuzzy logic" for intelligence for a long while, I guess!

For me a working definition is; humans are intelligent, humans can learn from each other. Animals even, have a level of intelligence, they learn from each other.

Large language models? They don't learn from each other. If you use the output of one model to train another, the new model gets *worse* not smarter, you get model collapse. Not intelligence, still just statistics.

Of the maybe 1500 lines of code, less then 10 were mine. Understanding a spec that it never has come across and turning it into good, working code is something I fail to attribute to anything but intelligence.

Knowledge representation: Okay, another personal story, sorry. Long ago when PC didn‘t mean „x86 architecture“, I read about statistical text generation and wrote a program that would take a longer text, …