Doing something you didn't predict is not the same as doing something you can't understand.
Oh there are plenty of programs we don't understand. Take some legacy 1960's mainframe program running at a bank. There are plenty of those still in use. Did you know there's a global shortage of COBOL programmers? It's true. Nobody writes new systems in COBOL anymore but there is so much 60's code still around that they can't find people to maintain it.
These ancient legacy mainframe systems are opaque to the programmers who maintain them. The original designers are long retired. The documentation was already a lie decades ago. The world literally runs on code that nobody understands.
You think the current generation of Microsoft programmers has any idea anymore how every line of Windows works? They fix problems and add new features and
try not to break things. This is the reality of commercial software and it has been this way for decades.
Practically every maintenance programmer in the world is working on a system that nobody in the shop understands. This is a daily reality of the profession of software engineering.
I'm not downplaying the incredible cleverness of the latest generation of neural networks. I'm blown away by the latest news of AlphaGo Zero, which was simply taught the rules of Go and then programmed to play millions of games against itself to see what works, and is now an expert player without having to be programmed with any human knowledge at all.
But even so, it's a program running on conventional hardware. If they had to, the designers could freeze the cpu, take a memory dump, a copy of the source code, a pencil and paper, and a lot of coffee, and figure out what it's going to do next.
I'm trying to demystify the latest generation of software. It's still software, running on the same Von Neumann architecture as the early vacuum machines of the 1940's. Sure it's really amazing what they can do. But it's just code. 1's and 0's, registers and arithmetic/logic units, instruction sets, and clever high-level languages that let the programmers express higher abstractions.
These recent neural net programs do things that are "unexpected" not only on the level of behavior, but of strategy. They do unexpected and surprising things that are not mistakes, or bugs, or glitches, but functional behaviors.
No question about it. I fully agree and I'm genuinely impressed. But they're just programs. You are trying to make them into something beyond that.
And likewise, human brains are not magic. They are conventional physiochemical structures with very complex organization.
Agreed. But, in my opinion,
not computations. Needless to say one can have a lively debate about that. But if minds really are computations, we're very very far from being able to prove that or describe how the code works.