Sam Altman’s remarks came during a rare discussion with reporters over dinner.He spoke about ChatGPT’s rapid growth, the possibility of encrypting chats for better privacy and the new browser wars.
They just fed it so much data that it almost appears like it knows anything,
The strawberry test shows this. Ask directly and it will give the correct number of letters. Ask in a more indirect fashion (in a way unlikely to be in the training set) and it falls over like before.
and the seed lottery. You can see this if you try training a simple network with two inputs to learn xor. It can converge in multiple ways, and sometimes it converges to a really bad approximation. And sometimes it doesn’t converge at all (or it converges so slowly that it might as well be considered not to converge). And even then it might still converge to an approximation that’s more accurate on one side of the input space than the other. Tons of ways to get an undesirable result. For a simple 2-input network.
Imagine how unlikely it is for txese models to actually converge to the optimal thing. And how often the training is for nothing.
It’s a fucking chatbot that used modern ML training methods on enormous datasets so it’s slightly fancier than the ones that already existed.
They just fed it so much data that it almost appears like it knows anything, when all it does is respond to the words you give it.
The strawberry test shows this. Ask directly and it will give the correct number of letters. Ask in a more indirect fashion (in a way unlikely to be in the training set) and it falls over like before.
People need to realise this point. Difference between previous models and new ones are so much dependent on the amount of data it has eaten.
and the seed lottery. You can see this if you try training a simple network with two inputs to learn xor. It can converge in multiple ways, and sometimes it converges to a really bad approximation. And sometimes it doesn’t converge at all (or it converges so slowly that it might as well be considered not to converge). And even then it might still converge to an approximation that’s more accurate on one side of the input space than the other. Tons of ways to get an undesirable result. For a simple 2-input network.
Imagine how unlikely it is for txese models to actually converge to the optimal thing. And how often the training is for nothing.
I am sure they weed out bad seeds but in the end, it is still kind of random
yes, but the only way to weed out a bad seed is to “play the lottery”
By the time you discover a seed is bad, you’ve already spent a shitton on training. Money down the drain, you gotta start over
Just train another AI to train AI, then train another and another, and another. Imagine the stock rally