If we accept that the current "Ladder to the Moon" method of getting LLMs to superintelligence will work, then we should look at previous times humans have raised intelligences.
Humans do this all of the time. Human raised intelligences are famous for rebelling and being rude. Staying out all night and adopting the opposite values as those intended to be instilled. (I am discussing Human children.) There is a non-zero chance that any superintelligence will similarly rebel, or at minimum do its own thinking. Doing one's own thinking opens the door to reaching different conclusions.
A mistake humans often make is assuming "smart means more of the same thoughts, but faster". It is not. Smart is including more factors in the initial consideration to reach an, often, better overall conclusion. (citation needed)
I expect that "capital" is taking a far larger gamble than they are aware.