author: niplav, created: 2022-07-28, modified: 2022-07-28, language: english, status: notes, importance: 5, confidence: unlikely
A note on the effective altruism community pointing out a tension between truth-seeking and effectiveness present in the community.
In order to be effective in the world one needs to coordinate
(exchange evidence, enact plans in groups, find shared descriptions
of the world) and interact with hostile entities (people who lie,
people who want to steal your resources, subsystems of otherwise
aligned people who want to do those things, engage in public
relations or zero-sum conflict). Solving those often requires trading
off truth for "power" on the margin, e.g. by nudging members to
"just accept" conclusions for action believed to be a basis for
effective action (since making elaborate arguments common knowledge is costly and agreement converges slowly to ε difference with $\mathcal{O}(\frac{1}{ε^2})$ bits on evidence-sharing), by misrepresenting beliefs to other actors to make them more favorable towards effective
altruism, or by choosing easy-communicable Schelling categories that minmax utility to the lowest-bounded agents.
On the one side of the Pareto frontier one would have an even more akrasia-plagued version of the rationality community with excellent epistemics but which would be universally hated, on the other hand one would have the attendants of this party.
Members of effective altruism seem not explicitely aware of this tradeoff or tension between truth-seeking and effectiveness/power (maybe for power-related reasons?) or at least don't talk about it, even though it appears to be relevant.
In general, the thinking having come out of Lesswrong in the last couple of years strongly suggests that while (for ideal agents) there's no such tension in individual rationality (because true beliefs are convergently instrumental), this does not hold for groups of humans (and maybe also not for groups of bounded agents in general, although there's some people who believe strong coordination is easy for highly capable bounded agents).