Reasons SIAI (and research generally) is not optimal?

[Alan Dawrst]

Among the utilitarians I've met over the years, a sizable fraction have come to the conclusion that the optimal destination for utilitarian funding is organizations that research speculative futuristic scenarios and the philosophical / scientific / methodological questions that such research requires. In particular, many of these utilitarians have named the Singularity Institute for Artificial Intelligence (SIAI) as a good example of such an organization, so I'll focus on it here, but the discussion can apply more broadly.

The claim that it's optimal for utilitarians to research speculative scenarios (including unsettled methodological problems like understanding qualia or evaluating our impacts in an infinite universe) derives from the observation that small changes to the quality of our understanding could drastically alter our conclusions about which courses of action are good and bad. For instance, suppose we discovered that entities we never thought conscious actually do experience qualia and, in fact, suffer greatly in a preventable way. (This isn't an absurd suggestion -- it happened to me several years ago when I realized that animals can feel pain. To the extent that the question of which animals can suffer remains open, such a discovery process is still going on right now.) If these entities outnumbered the sentient organisms we currently know about by orders of magnitude, the new optimal course of action could be dominated by doing what would prevent the most suffering on the part of those new entities.

As far as focusing on futuristic speculation, the argument is basically that there's a non-negligible chance that humans will have vast impacts on their future light cone, affecting many orders of magnitude more sentient organisms than have or will ever populate earth during the few billion years for which life exists there. The chance that humans do have such an astronomical impact is small, but the expected value is still likely enormous.

As a follow-up to my previous question about where to donate, I'll note that I'm currently leaning toward donating the money toward research at SIAI. While in general that organization's work is probably something that utilitarians would endorse, this particular project is one that I've coordinated to be of special interest for utilitarians concerned about preventing massive amounts of suffering in the universe -- possibly even outside our lightcone. In general, I recommend that utilitarians consider contacting SIAI to see if the group can arrange for research that may be of mutual interest.

The main objection I have to this strategy is the following. I am a total hedonistic utilitarian with an "exchange rate" between pleasure and pain that gives a significant weight to the badness of pain. In addition, I care more about animal suffering than I think most people do, in part because hedonism implies a lot more potential value and disvalue on the part of animals than do consequentialisms that value more abstract traits that seem to be possessed mainly by humans and their evolutionary kin. The number of people who hold my particular values is very small; the number who hold utilitarianism proper is somewhat bigger; and the number of rationalists who tend to hold some brand of consequentialism is larger still.

Now, knowledge is important, but so is ideology. For instance, I have concerns about what might result from a superintelligent friendly AI that -- perhaps influenced by deep ecology and impulses to propagate life, or perhaps just due to giving insufficient thought to animal suffering -- led to an increase in the number of wild animals throughout the universe, or perhaps in new universes. So there's a question: At what point is it better to promote your specific memes (hedonistic anti-speciesism, in my case) rather than general knowledge or AI that's generally "human-friendly" but perhaps not Benthamite? This might include, for instance, promoting concern about wild-animal suffering, so that -- if humans do have a huge impact on the future of the universe -- they do so in a positive rather than negative way. Sure, research on decision theory is important, but unless people use it to maximize the right things, it's to no benefit, and could even be harmful.

However, I should point out that while SIAI has no explicit ideology, several of its members do lean strongly utilitarian, and many more lean strongly toward some sort of rationalist consequentialism. So even on the question of ideology, SIAI may not be a bad choice for Benthamites, because the amount of philosophical overlap remains extremely high relative to the overlap with the general population. And if one arranges for specific research on a utilitarian-oriented project, the actual marginal impact of a utilitarian's donation can potentially be even better. But I still think contributing to SIAI's general funds is (probably, based on my current knowledge) an excellent choice.

What do others think here? Are there other reasons SIAI and the like are not optimal for utilitarians? For instance, perhaps the Singularity scenario is highly improbable. Or perhaps SIAI's ability to have an impact on it if it did occur would likely be minuscule. Or maybe real "friendly AI" is a utilitarian pipe dream that will almost certainly never amount to anything. While I agree with all of those statements, I still think the vast potential consequences of success here dominate the expected-value calculation.

But maybe there are other causes that would have higher chance of success? Or other organizations more qualified to address these matters? Or other donation strategies (e.g., funding research informally by coordinating with undergraduate students) that have higher leverage? In other words, tell me why SIAI is not an optimal recipient of charitable-donation dollars for expected-value maximizers?

[Arepo]

Toby Ord would probably be the best person to speak to about this (although if he responds I hope he'll do so in public).

My problem with these shot-to-nothing scenarios is that the arguments inevitably involve more wild guesswork than the arguers suggest. The typical reasoning goes that some event has a massive impact on net utility, which heavily outweighs its implausibility.

But we seem to have increasing difficulty calculating the odds of even day-to-day events to the kind of degree that would allow such precision, and these are predictions that involve countless factors that such arguments usually gloss over. (I find this with your arguments for Pascal's wager, for eg, which is a relatively simple case in that all you need to do is show that the Christian heaven is more plausible than any other infinite utility alternative). The writer is always willing to offer reasons why he/she thinks one should believe the numbers weigh what they say they do, but they're nowhere near being a mathematical proof. So they're ultimately gut instinct.

More immediate projects - your warm fuzzies ones - invoke far fewer variables, and even then if you were to try to calculate the hedons involved you'd never finish. One advantage they have, I suppose, is that results (or lack of them) can be turned around relatively quickly, so you can see whether they've met goals they were aiming for. Even if the SIAI's goals happen to be sensible ones, you'll never be able to evaluate how well they're meeting them.

An alternative issue, which I is perhaps less significant but worth considering, is simply the public perception of the charities you might give to. The average reasonably smart person is quite capable of understanding eg Toby's comparison of Fred Hollows to Seeing Eye, and might find it particularly inspiring that someone like you were exemplifying a new kind of 21st century human, who genuinely structures large parts of his life around helping others - inspiring enough to persuade her to increase her own donations. Such inspiration is likely to be much more powerful if the causes you're supporting are ones said average person a) intuitively supports and b) can see a powerful case that they're superior to some of the alternatives one might consider. The Fred Hollows vs Seeing Eye comparison holds for both points, the SIAI vs Fred Hollows comparison holds for neither.

You can potentially imagine the tabloids running a sympathetic story about someone who'd cured n people of blindness last week. If they ran a similar story about someone who'd reduced the chance of a lab universe by a factor of n, you can bet it would only be to mock them. The same probably goes, albeit slightly less, for the broadsheets - they're not exactly written exclusively by utilitarianism sympathisers.

[Alan Dawrst]

Thanks for the comments, Arepo.

(I find this with your arguments for Pascal's wager, for eg, which is a relatively simple case in that all you need to do is show that the Christian heaven is more plausible than any other infinite utility alternative).

I think it is the case that "all you need to do" is show that a particular possibility has sufficiently large potential consequences, not discounted by a correspondingly small probability, that a particular term dominates in the expected-value calculation. But showing that is the whole meat of the question and isn't trivial.

I guess you might say it's "close to impossible." Still, I try to be an expected-value maximizer, so any non-infinitesimal change in probabilities here is, in my view, highly valuable.

Your point about public sympathy is relevant. Indeed, when making the case for, say, living frugally in order to donate large amounts, it does make sense to use tangible causes for purposes of illustration. Singer has probably made more of a positive impact on the world by taking his example charity to be Oxfam than SIAI (or even perhaps GiveWell, arguably for the same reason). Still, if your donations are private, this needn't be a problem, unless you make public claims about actually donating to Oxfam. And being public about donations to more speculative projects could be a good idea among utilitarians (hence this post).

[CarlShulman]

"The chance that humans do have such an astronomical impact is small,"

Not that I necessarily disagree, but why?

In Nick Bostrom's framework, is this because of extinction (including simulations being turned off), stagnation, or posthumans that produce few sentient organisms?

http://www.nickbostrom.com/papers/future.pdf

[Alan Dawrst]

The main reason I had in mind was that such scenarios sound like science fiction. That is, they depend on certain technologies (superintelligence, space travel, perhaps nanotechnology, etc.) whose development is not certain. Some may even be close to impossible -- like space travel over long distances, given the huge energy requirements? (Of course, I suppose interstallar probes manned by non-biological controls are pretty feasible in principle.)

And then there are the more abstract reasons you point out, including anthropic ones.

[CarlShulman]

"That is, they depend on certain technologies (superintelligence, space travel, perhaps nanotechnology, etc.) whose development is not certain."

Betting odds, please.
http://lesswrong.com/lw/mp/0_and_1_are_ ... abilities/

[Alan Dawrst]

I like your style.

I'll give some off-the-cuff probabilities (rather than odds), but I'd be glad to revise them (perhaps until agreement). Without updating for anthropics -- just based on technological risks -- I might say

P(humans extinct in 50 years) = 0.3
P(humans extinct in 100 years) = 0.6
P(humans extinct in 1000 years) = 0.8.

Conditional on humans surviving long enough that they could develop the following technologies, here are some probabilities that they would:

P(artificial general intelligence) = 0.35
P(Drexler-style molecular nanotechnology that produces almost any manufactured goods) = 0.6
P(self-replicating space probes that could be successfully disbursed throughout the galaxy) = 0.3.

Conditioning on human survival makes things messy, because the fact that humans survive may give us information about how easy these technologies are. I may not have fully accounted for that above, but the numbers are completely rough anyway. Also, the technologies are interrelated. For instance, P(nanotech given AGI) would be more like 0.9.

What are your estimates? And what other technologies / scenarios should be considered?

[CarlShulman]

I won't get into the probabilities of extinction here (but check your email for a relevant decision aid), that's a much longer discussion, but your first and third conditional probabilities seem weirdly low.

You're saying there's only a 35% chance of artificial intelligence being developed with thousands of years to work on the problem (with IA, expanding population, etc), in a world with apparently computable materialist physics, where evolved human brains implement intelligence enough for civilization? That seems really hard to justify.

Likewise, you don't need to go very near the speed of light for astronomical waste concerns to come into play. Orion pulse drives and the like don't rely on wacky new physics.

Drexlerian nanotechnology may not work out as advertised, but automated manufacturing bases are a much broader class, and the class that's most relevant.

[Alan Dawrst]

You're saying there's only a 35% chance of artificial intelligence being developed with thousands of years to work on the problem (with IA, expanding population, etc), in a world with apparently computable materialist physics, where evolved human brains implement intelligence enough for civilization?

Yeah, maybe that's a little low. 50%?

I guess my intuition is that AGI might very well require a comprehensive understanding of how the human brain works, and it's quite possible to me that people just aren't smart enough to ever figure that out. Intelligence augmentation could help, but I'm also rather skeptical (probability 50%?) about whether those technologies will ever get to a stage where they work well enough to provide significant benefit.

[RyanCarey]

The average reasonably smart person is quite capable of understanding eg Toby's comparison of Fred Hollows to Seeing Eye, and might find it particularly inspiring that someone like you were exemplifying a new kind of 21st century human, who genuinely structures large parts of his life around helping others - inspiring enough to persuade her to increase her own donations. Such inspiration is likely to be much more powerful if the causes you're supporting are ones said average person a) intuitively supports and b) can see a powerful case that they're superior to some of the alternatives one might consider. The Fred Hollows vs Seeing Eye comparison holds for both points, the SIAI vs Fred Hollows comparison holds for neither.

I agree with Arepo, but I believe he hasn't criticised your idea harshly enough.
When you donate to Fred Hollows, you're an outstanding human being.
When you donate to SIAI, you're a philanthropist-maverick.
I think you're really bringing the team down here. I think that we have to recognise here that much more than we can get done alone, we can get done by inducing favourable behaviour in friends, colleagues, relatives, acquaintances, and others. To counter that you may achieve a maximal impact by inducing other philnathropic utilitarians to direct their donations towards SIAI, you're just delaying the fundamental question. Does donation to SIAI marginalise utilitarianism as a politically viable choice. And in my opinion, the answer here is definitely yes. I think that disregarding absurd targets of philanthropy will hardly compromise our integrity and it will clearly favour our public relations. I suppose some sociological & historical expertise might help. From what sociology I know, I would imagine we can get the most done by opposing those behaviours that have moderate prevalence at the time. For example, now that slavery is inexistent in Western Countries, we have shifted our attention to sex-based, race-based, disability-based, species-based discrimination. I fear that to promote the wellbeing of aliens is decades too far ahead of our time.

[EmbraceUnity]

The primary driver of humanity's immense power to create and destroy has been technology. Altering the course of it can have radical implications. A more relevant question would be what sort of technological advancement should be promoted.

It seems to me that Open Source modes of production are more equitable, just, and efficient. If we are to create a world free of artificial scarcity, we must collaborate. Luckily, the logic of the new communications technologies we have been inundated with inherently fosters decentralized, distributed innovation and collaboration.

We need wiki-science. We need Open Source biotechnology and nanotechnology. Patents in these areas are already showing themselves to have perverse consequences. The domain of life should not be patentable, it should remain in the Commons.

Considering it is bio and nanotechnologies which hold the potential to initiate mega-scale projects such as the elimination of wild animal suffering, we must be very concerned with their development. How is eliminating wild animal suffering profitable? It clearly is not. Only the open source mode of production can effectively mobilize people to tackle this issue, and other issues of similar scale and importance which have no profit motive to incentivize them.

I don't know enough to comment on AI specifically, but certainly its utility function would need to be coded to value all sentient life. There is no way to be certain of this without it being open source. However, I don't hold out any hope for the singularity since it is all over my head, and I cannot place any reasonable probability estimate upon it. It is like one big deus ex machina that many people invoke to tell people to forget about any immediate political concerns.

[Alan Dawrst]

I fear that to promote the wellbeing of aliens is decades too far ahead of our time.

I think that is a legitimate concern. Many people still aren't sure whether to care about, say, the chicken that went into the nuggets they're eating, so the suggestion that extraterrestrial-wild-animal suffering matters might give them fuel for a reductio against caring about animals at all.

Still, for someone who thinks extraterrestrial suffering is potentially orders of magnitude more important than chicken suffering, there's a risk that, if he doesn't promote the cause, no one will -- maybe society would never get to that stage of progress. In the case of SIAI, I think the main argument for action now is that we may simply not have time to wait for society to come around to thinking about these questions, because AGI might come first (possibly within a few decades).

EmbraceUnity, thanks for the comments. I'll defer to SIAI for answering the question of to what extent open-source AI is a good idea -- I'm not sure. As far as coding a value on sentience into the objective function of the AI, I agree with the sentiment, though I would just remark that we need to be extremely careful about how the AI determines what "happiness" and "suffering" are. We don't want an AI, wired to have a happy-face-expression detector, that turns the solar system into molecule-scale smiley-face pictures (p. 15 here). Working out such issues is one of the main projects that SIAI is tackling.

[Arepo]

Still, for someone who thinks extraterrestrial suffering is potentially orders of magnitude more important than chicken suffering, there's a risk that, if he doesn't promote the cause, no one will -- maybe society would never get to that stage of progress.

I think this is unlikely. The logic of caring about non-human consciousnesses seems to flow quite straightforwardly from any view beyond basic egoism. Society as a whole, and even philosophers seem to collectively take decades more than individual people to draw basic logical inferences, but it does seem to happen.

Another issue I have with preventing low-probability massive disasters is that it seems like focussing on them could potentially make util genuinely self-effacing.

If we imagine we live in a universe with infinite subtletly to its physical laws, for example, then at any given point in the lifespan of sentience we might be able to point to a remote probability R that we'll be able to achieve some breakthrough quantity of utility U which we could target T in preference to maximising short-term/likely utility M such that (I'm almost certainly going to screw up the notation, but hopefully you can decipher what I'm trying to get at) P(RU|T) > M. Ie that we should always suffer now to reduce maximise total utility in a future we never get to.

This seems especially problematic when you're trying to reduce the risk of extinction events (which I think is a large part of what the SIAI do) in a world where you're not very confident net utility is (and is expected to continue being) positive.

[Alan Dawrst]

If we imagine we live in a universe with infinite subtletly to its physical laws, for example, then at any given point in the lifespan of sentience we might be able to point to a remote probability R that we'll be able to achieve some breakthrough quantity of utility U which we could target T in preference to maximising short-term/likely utility M such that (I'm almost certainly going to screw up the notation, but hopefully you can decipher what I'm trying to get at) P(RU|T) > M. Ie that we should always suffer now to reduce maximise total utility in a future we never get to.

A good point. This general scenario has been brought up by SIAI supporters, actually. I'm not convinced that it's a bug; maybe it's just a feature of utilitarianism that we should support. Sometimes potential costs are too big to get wrong, and if we could potentially prevent extraordinary amounts of suffering by finding Pascal's button, maybe we ought to look for it.

Your particular illustration raises the concern that, at any given time, it may be optimal to postpone reward, leading to the reward never actually being achieved. This is indeed a concern, and it has been discussed some in the philosophical literature. Again, this is precisely the kind of problem that SIAI has and will research, hopefully before someone builds a naive AI that makes these kinds of mistakes.

This seems especially problematic when you're trying to reduce the risk of extinction events (which I think is a large part of what the SIAI do) in a world where you're not very confident net utility is (and is expected to continue being) positive.

You're right. This is my concern about making sure that the friendly AI actually would care about animal suffering and would be sufficiently utilitarian in its general goals that it wouldn't create massive amounts of uncompensated suffering, such as in new universes.

[DanielLC]

Although always suffering now to maximize the potential total utility would, indeed, be bad, we can't do it. At some point, people will stop supporting the idea and try to get utility now.

If you were capable of that sort of thing, the thing to do would be to look for as long as you can without looking forever. For example, you can't count to a googolplex, so if you tried to go for a googolplex years, you wouldn't know when to stop.

[Alan Dawrst]

At some point, people will stop supporting the idea and try to get utility now.

Keep in mind that we're not necessarily talking about people -- AIs can be very different. And even human descendants could look very strange to us.

[Arepo]

I had a few more thoughts against this, not particularly related:

1) The kind of AI-induced catastrophe you envision seems likely to wipe out local life pretty quickly if it happens. This seems likely to be a relatively clean cut compared to more familiar disasters like meteor strikes, global warfare etc. If it wipes us out, it will probably do so with much less suffering than they would.

2) The same kind of catastrophe would probably (the way you describe it) eliminate all sentient life on earth, rather than just all humans, as a major war/impact/plague etc might. If you believe that wild animal suffering outweighs wild animal happiness, then this is a much preferable result to one that left the biosphere going as a misery generator.

3) The fact that someone is asking a question doesn't mean they're answering it. One recurring objection I have to utilitarian arguments is that where precise data isn't available they fudge - fair enough in itself - but then act much too confident about their probability guesstimates, often relying on (often equally haphazard) large numbers to overwhelm the difference. I would be wary of funding any such group of people until they can show some sort of substantial evidence that their analysis of the world is more accurate than, for eg, a reasonably intelligent scientist's. Otherwise they seem as likely as the rest of us to suffer cognitive biases, not least self-preservation bias if honest enquiry would lead to the conclusion that their jobs are an inefficient use of money.

4) This is the most interesting thought I've had, specifically aimed at Alan and others with a fair amount of money to throw around and the willingness to throw it - if there are various causes which it's very hard to differentiate between with any degree of confidence, rather than picking one somewhat arbitrarily (see 3), you could do something more deliberate:

a) pick out those causes that are obviously more utilitarian than many others, and not obviously less so than any (eg curing cataracts, education for third world women, promoting veganism, universal welfare organisations in general, funding research into certain technologies and more, perhaps promoting political action on extremely cut and dried issues)

b) try to identify the most effective organisations in terms of the effectiveness of your dollar - again with the same elimination criteria (Givewell, Population Services International, Fred Hollows, SIAI etc). Probably aim to select an equal number of organisations in each category so they're equally weighted.

c) put together a list of the remaining organisations.

d) pledge a regular donation of $N to all of the organisations on the list, initially to be distributed evenly among them.

e) set up a scheme where you invite people to donate to one or more of the selected groups and offer to adjust your donation according to those given by people who sign up for it. Two possible egs:

i) first come first serve. You'll match all other donations D up to a total of N from the $N pool, distributing N-D (if >0) evenly among the list afterwards.

ii) proportionately. At the end of a specified time, you'll calculate how much of D has been given to each organisation, and divide N up with the same proportional distribution.

The obvious plus to this setup is that what you lose in slightly greater credence for the efficacy of one approach, you surely gain in giving people (both utilitarian and others) an incentive to give more - if I feel much more strongly about the risk of climate change than of short term poverty reduction, I can give pay for money to go my preferred cause at a drastically increased rate.

If you tried ii) there'd be the risk of richer philanthropists dominating the pool and disillusioning the poorer ones, but if that happened, you could always ask some of said philanthropists to donate their money to expanding N rather than D. i) might be a better way of winning people over though, since if they donated early enough they could almost guarantee close to doubling their contribution, whereas if it's proportional, some people might find the expected gain too nebulous to feel as motivated by.

[RyanCarey]

if I could try to recap your 4th point Arepo:
> one could match others donations to encourage contribution.
> this donation-matching could have a limit
> the donation-matching could apply to only organisations that have some claim to being the most cost-effective charity in the world.

Sounds very interesting!

[Arepo]

That's about the size of it, although I don't think I was proposing an artificial limit to the donation-matching. There's obviously a practical one in that even rich benefactors can't match more than they have, but otherwise it seems to me like the bigger N is, the more motivating it would be.

That said, I had a couple of further thoughts:

f) rather than selecting one particular way of apportioning N, you could divide it up into multiple pools. Then you could see which actually received the most contributions, and adjust the pool size/number accordingly.

g) As an alternative to i) and ii), I thought of the perverse sounding

iii) Absolute. Invest N in a relatively high interest (but probably low-risk) account, and don't donate any of it except to match contributions. If at the end of your assigned time limit any of N remains, you put in the next set of N, as you would in the other examples. But at any point, the sum of N-D is sitting somewhere gathering interest, but not going to any charity.

This one seems like it might have a really powerful motivating factor, especially for people who weren't confident enough of their views to make larger contributions than normal just to swing donations from one direction to another. In this case, those contributing to D know that (in a sense, at least) N is actually not going to go anywhere unless they give their money.

You'd obviously want to find a fine balance here so that the value of iii)N wasn't too far beyond the expected sum of iii)D. But you also wouldn't want to modify it so much that it ruined the sense that N is only going to charity if you pay for it to.

Anecdotally, I would find iii) extremely motivating. If someone like Alan were to set up eg ii) and i), I'd certainly give a few quid to both, but if he were to set up iii) and ii) I'd probably give a token sum to ii)D to test the waters, but I'd be keen to instantly give a large sum relative my income to iii)D.

[Alan Dawrst]

Arepo, all of your points 1-3 from the 04 Aug 2009 are excellent; in fact, they've been concerns of mine as well. I didn't happen to mention them in the original post, but they are important to think about. Regarding #2, one response that SIAI might make is that, say, a smiley-face AI with the goal of maximizing the number of smiley faces in the universe wouldn't hesitate to engage in ruthless behavior to accomplish its objectives. It might, for instance, run lots of universe simulations (producing lots of wild-animal suffering in the process), or use reinforcement-learning algorithms in a painful way, or threaten torture to coerce other civilizations into helping it out. I agree these may be tenuous scenarios, and it seems perhaps implausible to imagine that a poorly designed, kludgey AI would actually be rational enough to succeed at such plans. But it's hard to say -- I think the ideas need further exploration.

That brings up point #3, which I agree is a concern. Still, even if the SIAI community isn't more capable than your average group of philosophers, it remains the case that they're doing important philosophical work, and it seems worthwhile to increase the total size of the funding pie devoted to such research.

The various ideas under point #4 are interesting. Indeed, I think there are major philanthropists who do something like this through their matching-grant challenges: i.e., they donate an amount to one of their preferred causes that's somehow proportional to the amount others donate. (iii) is like a 2-for-1 match (or n-for-1, for some n), though the threat that the money wouldn't otherwise be donated is more credible in the case of (iii) -- usually, I suspect the philanthropists will donate anyway what they don't use for matching.

My main objection to the proposal is that I don't think there's a large number of almost equally valuable charitable causes, even within the fudge factors of our ignorance. I don't claim to know that cause X is very likely better than cause Y, but if you can make an argument that X might be 10,000 times better than Y, while it's somewhat less likely that Y is 10,000 times better than X, then I'll go with X over Y. As you suggested above, it's sort of like Pascal's wager: Yes, there are religions and anti-religions -- scenarios in which doing something will save you from hell, and others in which doing that same thing will send you to hell -- but unless the scenarios seem almost exactly symmetric, I'm going to treat them differently. A 50.0001% chance of avoiding hell is much, much better than a 49.9999% chance.