/ratanon/ - Rationalists Anonymous

Obviously correct, as proven by the insane and maldjusted goals of a lot "intelligent" "people" (INTJs). AI can only get worse.

>>6798

But those goals probably weren't things they would have been inevitably drawn to based on their biology, rather they formed as a result of the idiosyncracies of their life history and experience, which shaped a root goal system that started out pretty similar to other humans. Orthogonality is about whether you can program an AI so that even at startup it's inevitable it'll "grow up" to become something completely focused on whatever arbitrary goals you want, like paperclip maximizing.

>>6799

>Orthogonality is about whether you can program an AI so that even at startup it's inevitable it'll "grow up" to become something completely focused on whatever arbitrary goals you want

I'm afriad that isn't, strictly speaking, true. Sure, that would prove the orthogonality thesis, but intelligent agents simply not converging on a final goal as intelligence goes to infinity will also do it.

>>6800

>I'm afriad that isn't, strictly speaking, true. Sure, that would prove the orthogonality thesis, but intelligent agents simply not converging on a final goal as intelligence goes to infinity will also do it

Yudkowsky's notion of friendly AI is very much about the idea of programming AIs with fixed goal functions like Asimov's 3 Laws of Robotics that will ensure they'll behave in our best interests, not about socializing them to be nice like we would with children. Likewise, his fears of unfriendly AI are based on the idea that if AI can have such basic goal functions loaded into them from the start, we might accidentally choose bad ones that would lead to ironic be-careful-what-you-wish-for style disasters (like the famous story "The Monkey's Paw") when the AI becomes superintelligent. For example, the idea of a "paperclip maximizer" is not that some mad villain creates an AI with the intention that it will destroy the human race by turning the Earth into paperclips, but that an innocent desire to build an AI to improve efficiency in paperclip manufacture will go horribly wrong because the AI becomes superintelligent but still cannot help but remain rigidly locked into the axiomatic goal of making as many paperclips as possible.

>>6802

I don't understand the details of inverse reinforcement learning, but it seems to involve setting the right "reward function" (also sometimes called a 'utility function', see https://wiki.lesswrong.com/wiki/Utility_function ) where outputs are evaluated for how well they match up with some high-level goal, like the goal mentioned in your link of building a good house; in neural networks, my understanding is that the output would be evaluated in terms of this set function, and that would be used to directly adjust the connection strengths between simulated neurons (see http://en.wikipedia.org/wiki/Types_of_artificial_neural_networks#Fully_recurrent_network ). If I'm understanding that right, it isn't really like socialization where the only influence you can have is on what the intelligence experiences through its sensory channels. And as I understand it Yudkowsky's orthogonality thesis does involve the idea that all intelligences can be seen as optimizers of some reward function function, and the high-level goals we set in the reward function can be completely arbitrary–see http://wiki.lesswrong.com/wiki/Paperclip_maximizer which says:

>Most importantly, however, it would undergo an intelligence explosion: It would work to improve its own intelligence, where "intelligence" is understood in the sense of optimization power, the ability to maximize a reward/utility function—in this case, the number of paperclips. The AGI would improve its intelligence, not because it values more intelligence in its own right, but because more intelligence would help it achieve its goal of accumulating paperclips. … For humans, it would indeed be stupidity, as it would constitute failure to fulfill many of our important terminal values, such as life, love, and variety. The AGI won't revise or otherwise change its goals, since changing its goals would result in fewer paperclips being made in the future, and that opposes its current goal. It has one simple goal of maximizing the number of paperclips; human life, learning, joy, and so on are not specified as goals. An AGI is simply an optimization process—a goal-seeker, a utility-function-maximizer. Its values can be completely alien to ours. If its utility function is to maximize paperclips, then it will do exactly that.

And it seems to me Yudkowsky pretty consistently talks as though the key to making sure an AGI remains friendly is choosing the right utility/reward function to be present in the AGI from the start, for example in the paper at http://intelligence.org/files/ComplexValues.pdf he writes:

>Omohundro (2008) lists preservation of preference among the “basic AI drives.”

is in turn suggests an obvious technical strategy for shaping the impact of Arti cial Intelligence: if you can build an AGI with a known utility function, and that AGI is sufficiently competent at self-modi cation, it should keep that utility function even as it improves its own intelligence, e.g., as in the formalism of Schmidhuber’s Gödel machine (Schmidhuber 2007). e programmers of the champion chess-playing program Deep Blue could not possibly have predicted its exact moves in the game, but they could predict that Deep Blue was trying to win—functioning to steer the future of the chessboard into the set of end states de ned as victory.

>>6803

>If I'm understanding that right, it isn't really like socialization where the only influence you can have is on what the intelligence experiences through its sensory channels

You're probably right, I haven't looked into the specifics of it very deeply. It was just the basic impression I got from Stuart Russell's Edge response (https://www.edge.org/response-detail/26157) where he describes it as "learning a reward function by observing the behavior of some other agent who is assumed to be acting in accordance with such a function".

>And it seems to me Yudkowsky pretty consistently talks as though the key to making sure an AGI remains friendly is choosing the right utility/reward function to be present in the AGI from the start

Right, but isn't the utility function supposed to be determined by CEV, rather than hardcoded? See:

> In developing friendly AI, one acting for our best interests, we would have to take care that it would have implemented, from the beginning, a coherent extrapolated volition of humankind. In calculating CEV, an AI would predict what an idealized version of us would want, "if we knew more, thought faster, were more the people we wished we were, had grown up farther together". It would recursively iterate this prediction for humanity as a whole, and determine the desires which converge. This initial dynamic would be used to generate the AI's utility function.

https://www.xenosystems.net/against-orthogonality/

>>12889

I think Land is just wrong when he says that "Even the orthogonalists admit that there are values immanent to advanced intelligence, most importantly, those described by Steve Omohundro as ‘basic AI drives’ — now terminologically fixed as ‘Omohundro drives’".

I haven't asked, but I would think Yudkowsky would say you could definitely build an AI which tries to first take over the world and then commit suicide, i.e. one that violates the Omohundro drive for self-preservation. Such an AI would not be produced by natural evolution, but that's exactly why we need to design it ourselves instead.

Certainly, Scott Alexander's whole thing about "defeating Moloch" is to design something that just evolution, and more generally competition between many different agents, could *not* produce, because the result of evolutionary drives will be terrible.

>>12891

Can't make one that gets dumber though. Imo this line of argument is a red herring like those about non-orthogonal AI being outcompeted. The real questions are:

Is cognition itself darwinian somehow? Could true AGI be static / frozen and still earn that G? Without a literal galaxy-brain (that would view it as a dumb insect) to design it? Are "goals" actually real and formalizable?

Also finitude of the universe and convergence. Even if something like Clippy is possible, would it be practically distinguishable from an intelligence maximizer in the long term? It mostly has to balance entropy research against management of its decomposition…

>because the result of evolutionary drives will be terrible.

What is better than power acquisition for its own sake enjoyed inasmuch as done effectively? It feedbacks, unlike wastage to humor monkey token-goals.

Obviously wrong, and even the people who claim to be orthgonalists aren't really. Simply admitting Omohundro drives exist is enough to extinguish the hard orthogonalist thesis. The rest of the misunderstanding comes from moronically extrapolating the Dream Time to the future, thinking that Omohundro drives take some small part of an agent's resources, while they're free to spend the rest wherever they want. Hanson's Age of Em is a good antidote to such nonsense. Omohundro drives are all there is in a competitive environment, and any agent that makes it will have goals that are just hypertrophied versions of those drives.

>>12895

Sure, Omohundro drives are all that survive in a competitive environment, no one denies that. The question is what an AI would spend surplus on, if it ever got surplus. (And the theorized route by which it will get surplus is taking over the world by being superintelligent.)

The only way to reject orthogonality with confidence is to assert some cosmic moral law that hides in the equations. As far as we know, this universe runs on physics. Indeed we could expect intelligences to converge on universal effective principles for cooperation, but even so there's too much solution space left and too little reason for following any sort of law, especially when interacting with a known and lesser agent.