Francois Chollet, Mike Knoop - LLMs won’t lead to AGI - $1,000,000 Prize to find true solution

Speaker 0: Okay. Today, I have the pleasure to speak with Francois Chollet, who is a AI researcher at Google and creator of Keras. And he's launching a prize in collaboration with Mike Kanouf, the cofounder of Zapier, who we'll also be talking to in a second. A $1,000,000 prize to solve the ARC benchmark that he created. So first question, what is the ARC benchmark and why do you even need this prize?

Why won't the biggest l l m we have in a year be able to just saturate it?

Speaker 1: Sure. So ARC is intended as a kind of IQ test for machine intelligence. And what makes it different from, most LAM benchmarks out there is that it's designed to be resistant to memorization. So if you look at the way LAMs work, they are basically this, big interpretive memory. And the way you scale up their capabilities is by trying to cram as much, knowledge and patterns as possible into them.

And, by contrast, ARC does not require a lot of knowledge at all. It's designed to only require what's known as, core knowledge, which is, basic knowledge about things like elementary physics, objectness, counting, that sort of thing, the sort of knowledge that any 4 year old or 5 year old possesses. Right? But what's interesting is that each puzzle in ARC is novel, is something that you've probably not encountered before, even if you've memorized the entire Internet. And the sweat the sweat makes it sorry.

The sweat, makes makes Arc challenging for LMS. And so far, LMS have not, been doing very well on it. In fact, the approaches that are working well, are more towards, discrete program search, program synthesis.

Speaker 0: So first of all, I'll I'll make a comment that I'm glad that as a skeptic of LLM, you have put out yourself a benchmark that is it accurate to say that suppose that the biggest model we have in a year is able to get 80% on this, then your view would be we are on track to AGI with LLMs. How would you think about that?

Speaker 1: Right. I'm pretty skeptical that we're gonna see do 80% in a year. That said, if we do see it, you would also have to look at how this was achieved. If you just, train the model and millions or billions of puzzles similar to ARC so that you're relying on, the ability to have some overlap between the tasks that you train on and the task that you're gonna see at test time, then you're still using memorization. Right?

And maybe maybe it can work. You know, hopefully, Arc is

Speaker 2: gonna be good enough that