[Posted to the CFAR mailing list]

[Somewhat experimental: Looking for thumbs up and thumbs down on this kind of writing. I’m trying to clarify some of the fuzziness around why we are calling the-thing-some-of-us-are-calling-rationality “rationality.”]

So what is this rationality thing anyway?

Simply stated, some behavior works better than other behavior for achieving a given goal. In fact, for formal and well defined environments, “games”, this is provably true. In the early to mid 20th century, academic mathematicians developed game theory and decision theory, mathematical formalizations of idealized decision algorithms that give provably optimal outcomes (in expectation).

One school of rationality (let’s call it “formal rationality”) is largely about learning and relying on these decision rules. For a rationalist of this type, progress in the field means doing more math, and discovering more theorems or decisions rules. Since most non-trivial decision problems involve dealing with uncertainty, and uncertainty in the real world is quantified using statistics, statistics is central to the practice of formal rationality.

MIRI does the sort of work that a formal rationalist would consider to be progress on rationality: trying to develop solutions to decision theory problems. (This is NOT to say that most, or even any of the actual people who work are MIRI are themselves of the “formal rationality” school as opposed to those to follow. In fact I have reason to think that NONE of them would identify as such.) The other large frontiers of “formal rationality” are mostly in economics. The economy can be thought of as a single gigantic game theoretic game.

For the formal rationalist, rationality is almost entirely solved. We have game theory. We have probability theory. We have decision theory. There may be edge-case scenarios that need to be solved (pascal’s mugging, for instance), but for the most part, the “art” has already been invented. Declaring oneself a rationalist in the formal sense is a statement of philosophy: it means you trust the approximations of the formal decision rules over intuition, common sense, tradition, or well, anything.  One doesn’t need to qualify with the word “aspiring.”

(There’s a framework nearby to formal rationality which is largely captured by the term “evidence-based.” This is the position that one should base one’s actions and beliefs on evidence, over intuition or superstition. We can call this traditional rationality.  Traditional rationality includes science, and evidence-seeking in general.)

If you have formalized decision rules that describe the behaviour of goal directed agents, you now have the affordance to check what humans are actually doing. Enter Kahneman and Tversky. Over the course of the 1970’s to 1990’s ,they do many experiments and determine that 1) most people are not optimal goal-directed agents, (i.e. they are “irrational”. Little surprise to anyone, I think), 2) that those with advanced knowledge of “formal rationality” (e.g. statistics, economics, probability theory, game theory, decision theory) also fail to be optimal goal-directed agents (WE’re irrational too), and 3) that humans tend to deviate from ideal behaviour in systematic, predictable ways.

Thus develops the Heuristics and Biases project in psychology, with gives to rise another approach to the project of rationality. If humans are intrinsically and systematically biased, and simply telling a person about the bais doesn’t fix it (as is often the case), then the greater share of rationality training involves coming up with methods to counteract native cognitive bias. We can call this approach to rationality “the debiasing approach.” It inherits many of the formalizations from formal rationality (which do reflect ideal behavior), but the emphasis is on dealing with the actual human mind and correcting it’s faults. The project of rationality involves math, but now it is mostly in the domain of psychology.

This is in large part, the approach to rationality that Eliezer took in the sequences (though the sequences are a philosophical treatise, and his aims went beyond debiasing), and it fairly well characterizes LessWrong.

In 2012, CFAR is founded, and initially takes the debiasing approach. But the organization pretty quickly pivots away from that sort of model (you’ll notice that there are no modules in the current workshop of the form “this is the X fallacy/bias and here is the technique that eliminates or mitigates it.”) Developing debiasing protocols proves to be difficult, but there’s a nearby thing which is very useful and and much more tractable. CFAR borrows the System 1 / System 2 framework from heuristics and biases and develops methods to get those facets of the mind to communicate with one another.

For instance, sometimes a person intellectually endorses an action but doesn’t feel emotionally motivated about it. Propagating urges (or aversion factoring) is a technique that facilitates the dialogue between those parts of the mind, such that one (or both) of them updates and they are both on the same page. Similarly, sometimes a part of the mind has information about how likely a project is to succeed, but that data needs to be queried to be useful. The Inner Simulator / Murphyjitsu is a technique that lets the conscious, verbal system query the part of the mind that automatically makes predictions of that sort.

This approach isn’t about mitigating specific biases, but rather propagating information that one already has in one part of his/her mind to other parts of the the cognitive system. We can call this approach the “parts-propagation approach.” It’s about unifying the mind (minding our way style, mostly, but not exclusively) such that all parts of the mind are on the same page and pulling in the same direction, so that we can be more “agenty ducks” (i.e., better approximations of the simplified goal-directed agents of formal rationality, with stable, ordered goals) that can get shit done in the world.

These are three rather different approaches to rationality, and each one entails very different ideas of what the “art of rationality” should look like and what directions research should take. I have thoughts about which of these approaches are most tractable and important, but my goal here is only to clarify some of the confusion about what is meant by “rationality” and why.

Thoughts? Are these categories good ones? Do they carve reality at the joints?