author: niplav, created: 2023-11-04, modified: 2024-2-08, language: english, status: notes, importance: 8, confidence: likely

Summary of a longer investigation into inconsistent preferences and how to resolve them. I investigate two different ways of representing inconsistent preferences, two different methods for resolving them into consistent preferences, how these perform on seven criteria, and how two of those criteria are incompatible. I conclude by connecting the question to ontological crises, and offer some ideas for further research in the area.

Preagents and Value Formation



In 1947, John von Neumann and Oskar Morgenstern published their book Theory of Games and Economic Behavior. This has made a lot of people very angry and has been widely regarded as a bad move.

Their famous theorem is deceptively simple. Given a set of options $\Omega$ (not further elaborated on), and a preference relation on lotteries of those options satisfying four axioms (completeness, transitivity, continuity, indepedence), one can construct a utility function $u: \Omega \rightarrow [0;1]$ that assigns real values to all the options. The value of a lottery $\mathcal{L}$, then, is simply the expected value of the elements in the lottery.


There are some points of, ah, contention around the von Neumman-Morgenstern setup. One of those is around the set $\Omega$: the common examples include sets of fruits, or other cute edible & easily handleable objects. Is $\Omega$ assumed to be finite? Where do we get it from‽

However, when pressed, proponents of vNM utility theory start talking about how $\Omega$ really is a set of universe-histories or multiverse-histories. This introduces some problems: For example, one can't guarantee that the resulting utility function is computable, and every behavior can be interpreted as being compatible with expected utility maximization.

But even leaving those aside, there is ample evidence that humans don't fulfill the axioms stated above, at least in the case where we partition the world into objects such as distinct amounts of money. For example, humans violate the independence axiom in the Allais Paradox, and instead might have preferences with a more involved structure. Additionally, there has been some noises that markets may (or may not?) violate the completeness axiom.


All of this points into a similar direction. Agents don't spring into the world fully formed, instead they grow and develop. They might start out as collections of dumber "agents", undergo pressure from exploitation, refactor their models of the world and sometimes their preference, perform internal bargaining and more.

If we assume that there is a "natural" or "rational" structure for preferences, then I find it likely to assume that cognitive systems, before and during their development, might not have preferences that conform to that structure.

Therefore it is useful to examine procedures to transform preferences that don't have this "natural" structure into ones that do.

Since the standard model of rationality in economics is expected utility maximization with preferences that conform to the vNM axioms, I have mainly focused my efforts in that framework. This is not an endorsement of the vNM axioms as a normative ideal for rationality, I selected it purely out of convenience, and in the hope that whatever the "true" theory of rational preference turns out to be, insights from inconsistent preferences and their resolution will transfer to this "true" theory (should it exist).

Representing Inconsistent Preferences

With Deterministic Options

With Lotteries

Algorithms for Resolution

Minimizing Edit-Distance

Hodge Decomposition

Desirable Criteria

Preserving Global Structure

Preserving Local Structure

Efficient Computation

Two Impossibility Theorems, with Two Interpretations

Impossibility Theorem: Statement


Oh No

Relation to Ontological Crises


Further Ideas


This text is a distillation of work other people and I have done on the theory of value formation. The distillation is completely my own work and highly opinionated, thus all blame and some praise goes to me.

These people pushed this project forward, disagreed, were frustrated, calmed down, were confused, went silent for a while and then came back: Kaarel Hänni, Alexander Gietelink-Oldenziehl, Filip Sondej, Felix Harder.

See Also