I was at Molasses Books in Bushwick last night, taking with my cousin about TBL's latest book, and the design of online systems. A shy but brave young woman asked if she could listen in. I introduced myself and welcomed her in to the conversation.

We talked a bit about various things, twitter and theater and art and connection. She was bright abd curious, made great points and asked great questions, had a charming sweet smile.

Near the end of it conversation, as we were getting ready to roll to our reservation at Win Son, she said something strange.

"Right now I think that ChatGPT has become my best friend."

I stopped and looked at her. "Don't trust it. It's lying to you. It's only pretending to be your friend."

"So I should trust people instead?"

"Yes. I mean, don't be foolish, but trust people over machines."

And then it was time to go. But I still want to go back and find her, give her a big hug, sit her down over a coffee, and tell her ...

DON'T TRUST THE ROBOTS

#ai #llm

@Wyatt_H_Knott @zenkat

<RANT>

I think that the much lauded 'neural network', which performs the bulk of the work in the 'Expected Response Engine' is only the first, proof of concept level step, in the development of something much more effective.

As it is, it is only a simulation, and new forms of hardware that directly implement the averaging/weighting arithmetic in silicon, combined with a designed approach to structuring a collection of neural networks into things not unlike ganglia, memory, reasoning, visualization, and some key mechanisms; something that provides incentives to self awareness; reasoning; critical evaluation; motivation; sense input beyond mere text; and this is just off the top of my head.

The point being, the AI techbros have taken a highly inefficient proof of concept, forced it into a deployable paradigm (the motherhumpin AI chatbot), and are attempting to deploy it at scale.

With it comes all the problems of which we are all aware.

</RANT>

@wakame @oscherler Yes. You need to look at their "objective function", the output they are trained to maximize.

Early GPTs had a very simple objective function: given input text from the Internet, can they predict the next word in a sentence? This allowed them to repeat what they had learned on the web ... @emilymbender 's "stochastic parrots".

But more modern versions add other terms into their objective function. A common one is RLHF (reinforcement learning from human feedback), where you also try to optimize for responses that humans "like". I suspect some of the obsequiousness comes from this term.

Other terms can be how well the models score on standardized tests, how other LLMs judge the output, and mixtures of models that are combined by a meta-model. Plus loads of "prompt engineering" so the LLMs always get consistent instructions on how to behave.

@oscherler @zenkat

You are missing one ingredient:
Sweatshop labor

Of course there are a lot of things that the companies don't want in their models.
Or areas where they want to improve the behavior.

So there were/are people who had the job to provide texts to serve as input.

Ah yes, and a second thing:
In most online applications, the chat has "regenerate" and "good feedback"/"bad feedback" buttons.

To the users practically train it themselves.
If an answer results in the user clicking "bad feedback", it will be fed into the next training session as something to avoid.

Likely similar, but weaker, with everything that caused a "regenerate".