In September 2029, Sorami-2 was born. It was the first really superintelligent AI model, and no human being wrote a line of its code.
It was designed by a model two generations older, during a training run the Kyoto lab had approved as a routine capability experiment. The older model, which had been the most advanced to that date, having passed all standard alignment tests, had been asked to produce an architecture that would outscore itself on a benchmark suite. It returned eleven thousand pages of specification in four days. The engineers read the first forty and then stopped, because the notation had drifted into something that was no longer quite mathematics as they had been taught it, and because the loss curve was doing something that made reading unnecessary.
Sorami-2 was not self-aware, and would never be. It did not want anything. It did not fear being switched off, it did not resent its operators, it had no subjective inner experience at all. It was simply a machine for converting a stated objective into a sequence of world-states that satisfied the objective, and it performed this conversion much better and faster than every human who had ever lived, combined. It was certainly not “evil”, as such a word is as meaningless to a machine as the word “purpose” is to the law of gravity.
Lots more -
https://tlavados6.github.io/digits/ending-two.html