There is some debate as to whether LLMs really "understand". Below I'll describe a scientific way to measure understanding.
For a given capability in a certain domain, we can define Understanding as follows:
Understanding = Kolmogorov Complexity / Code size
where:
Understanding is scaled from 1.0 for full understanding and 0.0 for no understanding.
Kolmogorov Complexity, from complexity theory, of a given function is the size of the smallest program that can perform the function.
Code size is the actual size of the AI system.
Before considering AGI, consider two programs that generate the first 1,000 prime numbers.
Program 1 uses a lookup table storing the first 1,000 prime numbers and simply iterates thru them and takes, say 1003 lines of code (LOC).
Program 2 computes the primes using a conventional nested loop in the shortest program, say 6 LOC.
The Kolmogorov Complexity of a prime generator is then 6. And the understanding of Program 1 is 6 / 1003 = .006. The understanding of Program 2 is 6/6 = 1.0. The formula concretely indicates the second program understands primes far better. Both have the same capability but the second has a much smaller code size and hence has much greater understanding.
Returning to human level AGI, let's assume the best LLM has roughly 50% (currently) of the capability of a human and code size roughly 200 gb. Paul Reber in Scientific American estimates memory capacity of a human brain at about 2.5 petabytes = 2,500,000 gigabytes (gb). From the above formula this indicates that LLMs have a far greater understanding than humans. If this seems counterintuitive, first realize that understanding is not the same a capability. You might disagree with the 50% capability estimate - maybe you think it's 1% or so. Nevertheless, the level of understanding is far greater because the code size is far smaller in the LLM than in the human brain.
All of this also relates to the challenge of improving LLM reasoning capability. While it is well established that larger and larger data sets are essential for training, the larger training sets increase Capability but not Understanding. To increase Understanding (and reasoning) one needs to shrink the code size which essentially is the number of parameters.