I agree wholly. And I’d add this argument: Our sense of self is utterly contingent on continuity…memory. Say we achieve the powerful Singularity intelligences described in the interview, either all at once, or, as I anticipate, gradually, with better and better brain augments. What happens when we can remember, I mean REALLY remember in detail our lives before augments, during the gradual buildup of augmentation, and then with remarkably sharp, full memories as we enter the Singularity with our vast, upgraded intelligences?

I’m as certain as anything that we would experience a much sharper sense of self than we do now.

It might or might not be an individual in the sense that we currently use that term. The “phenomenal self” that each of us humans have (to use Metzinger’s term) might not be something that a superintelligent being would find a use for. But that wasn’t my point.

My point was that it might not recognizably have any of the self-structure of “Ben Goertzel” … even if it were an individual of some kind.

Maybe the human personality and mind architecture are simply not compatible with drastically transhuman levels of intelligence.

The analogies some commenters make to “30 year old humans versus 3 year old humans” are not really on point, because bot of these have human brain architecture and have intelligence within an order of magnitude of ordinary humans..

– Ben G

Well said.

I think of it in this fashion: There are probably several million characteristics that accurately describe me. There are probably only a tiny handful of those that I consider essential to my identity–the core of who I am. Changing any of the others no more changes who I am than does getting laser eye surgery or having wisdom teeth removed–or simply spending a year in college. (In fact, the latter is more likely to result in some material change to who I am.)

To the extent that superintelligence might change who we are, I would suggest that it would most likely turbocharge the process by which life experiences change who we are. Going back to my college example above: spending a year, or several, in college is certainly likely to change who you are. If future technology allows one to effectively experience a similar amount of “life” (virtual or otherwise) in less time, then it’s possible that we’ll morph from any given present self to that self’s future self more quickly than we otherwise would–but technology would at best facilitate those ways that we allow the world to shape who we are day in and day out now. The technology itself wouldn’t be changing who we are.