The top of the pyramid is constant

~ elityre ~ Leave a comment

[epistemic status: incomplete thought, perhaps to be followed up on in later posts]

I just read most of this article in the Atlantic, which points out that despite increasing investment (of both money and manpower) in science, the rate of scientific discovery is, at best commiserate with scientific progress in the 1930s, and may not even be meeting that bar.

(This basic idea is something that I’ve been familiar with for several years. Furthermore, this essay reminds me of something I read a few months ago: that the number of scientific discoveries named after their discovers (a baseline metric for importance?) is about the same decade to decade, despite vastly more scientists. [I know the source, but I can’t be bothered to cite it right now. Drop a message in the comments if you want it.]

When I read the headline of this article, my initial hypothesis was this:

Very few people in the world can do excellent groundbreaking science. Doing excellent scientific research requires both a very high intrinsic intelligence, and additionally, some other cognitive propensities and dispositions which are harder to pin down. In earlier decades science was a niche enterprise that attracted only these unusual people.

Today, science is a gigantic network of institutions that includes many times as many people. It still attracts the few individuals capable of being excellent scientists, but it also includes 10 to 1000 times as many people who don’t have the critical properties.

My posit: The great scientists do good work. Any additional manpower put into the scientific institutions is approximately useless. So the progress of science is constant.

(There’s probably a second order factor that all those extra people, and especially the bureaucracy that is required to manage and organize them all, get in way, and make it harder for the best scientists to do their work. (And in particular, it might dilute the attention of the best scientists in training their successors, which weakens the transmission of the cognitive-but-non-biological factors that contribute to “great-scientist-ness.”)

But I would guess that this is mostly a minor factor.)

But…

Between 1900 and 2015, the world population increased by close to 5 times. It seems like if my model was correct, the number of “great scientists” today would be higher than it was in 1930, if only because of population growth (ignoring things like the Flynn effect).

Why aren’t there 5 x as many great scientists? Maybe the bureaucracies getting in the way thing was bigger than I thought?

Maybe the “adjacent possible” of scientific discoveries increases linearly, for some reason, instead of exponentially, as one would expect?

Or maybe “discoveries named after their creators” is not a good proxy for “important discoveries”, because it’s a status symbol. And the number of people at the top of a status hierarchy is constant, even if the status hierarchy is much bigger.

Circling vs. Unrolling

~ elityre ~ Leave a comment

[Musing]

In reference to Critch’s post here.

I’m intrigued by the explicit unrolling in contrast to circling. I wonder how much circling an instance of developing overpowered tools on weird partly-orthagonal dimensions (like embodiment) because you haven’t yet discovered the basic simple structure of the domain.

Like, a person might have a bunch of cobbled together hacks and heuristics (including things about narrative, and chunking next actions, and discipline) for maintaining their productivity. But a crisp understanding of the relevant psychology makes “maintaining productivity” a simple and mostly effortless thing to do.

Or a person who spends years doing complicated math without paper. They will discover all kinds of tricks for doing mental computation, and they might get really good at these tricks, and building that skill might even have benefits in other domains. But at the end of the day, all of that training is blown out of the water as soon as they have paper. Paper makes the thing they were training hard to do easy.

To what extent is Circling working hard to train capacities that are being used as workarounds for limited working memory and insufficient theoretical understanding the structure of human interaction?

(This is a real question. My guess is, “some, but less than 30%”.)

A lot of my strategies for dealing with situations of this sort are circling-y, and feel like a lot of that is superfluous. If I had a better theoretical understanding, I could do the thing with much more efficiency.

For instance, I exert a lot of effort to be attuned to the other person in general and to be picking up subtle signs from them, and tracking where they’re at. If had a more correct theoretical understanding, a better ontology, I would only need to be tracking the few things that it turns out are actually relevant.

Since humans don’t know what those factors are, now, people are skilled at this sort of interaction insofar as they can track everything that’s happening with the other person, and as a result, also capture the few things that are relevant to the underlying structure.

I suspect that others disagree strongly with me here.

A mechanistic description of status

~ elityre ~ 1 Comment

[This is an essay that I’ve had bopping around in my head for a long time. I’m not sure if this says anything usefully new-but it might click with some folks. I think this is pretty bad and needs to be rewritten and maybe expanded substantially, but this blog is called “musings and rough drafts.”]

In this post, I’m going to outline how I think about status. In particular, I want to give a mechanistic account of how status necessarily arises, given some set of axioms, in much the same way one can show that evolution by natural selection must necessarily occur given the axioms of 1) inheritance of traits 2) variance in reproductive success based on variance in traits and 3) mutation.

(I am not claiming any particular skill at navigating status relationships, any more than a student of sports-biology is necessarily a skilled basketball player.)

By “status” I mean prestige-status.

Axiom 1: People have goals.

That is, for any given human, there are some things that they want. This can include just about anything. You might want more money, more sex, a ninja-turtles lunchbox, a new car, to have interesting conversations, to become an expert tennis player, to move to New York etc.

Axiom 2: There are people who control resources relevant to other people achieving their goals.

The kinds of resources are as varied as the goals one can have.

Thinking about status dynamics and the like, people often focus on the particularly convergent resources, like money. But resources that are only relevant to a specific goal are just as much a part of the dynamics I’m about to describe.

Knowing a bunch about late 16th century Swedish architecture is controlling a goal relevant-resource, if someone has the goal of learning more about 16th century Swedish architecture.

Just being a fun person to spend time with (due to being particularly attractive, or funny, or interesting to talk to, or whatever) is a resource relevant to other people’s goals.

Axiom 3: People are more willing to help (offer favors to) a person who can help them achieve their goals.

Simply stated, you’re apt to offer to help a person with their goals if it seems like they can help you with yours, because you hope they’ll reciprocate. You’re willing to make a trade with, or ally with such people, because it seems likely to be beneficial to you. At minimum, you don’t want to get on their bad side.

(Notably, there are two factors that go into one’s assessment of another person’s usefulness: if they control a resource relevant to one of your goals, and if you expect them to reciprocate.

This produces a dynamic where by A’s willingness to ally with B is determined by something like the product of

  • A’s assessment of B’s power (as relevant to A’s goals), and
  • A’s assessment of B’s probability of helping (which might translate into integrity, niceness, etc.)

If a person is a jerk, they need to be very powerful-relative-to-your-goals to make allying with them worthwhile.)

All of this seems good so far, but notice that we have up to this point only described individual pairwise transactions and pairwise relationships. People speak about “status” as a attribute that someone can possess or lack. How does the dynamic of a person being “high status” arise from the flux of individual transactions?

Lemma 1: One of the resources that a person can control is other people’s willingness to offer them favors

With this lemma, the system folds in on itself, and the individual transactions cohere into a mostly-stable status hierarchy.

Given lemma 1, a person doesn’t need to personally control resources relevant to your goals, they just need to be in a position such that someone who is relevant to your goals will privilege them.

As an example, suppose that you’re introduced to someone who is very well respected in your local social group: Wendy. Your assessment might be that Wendy, directly, doesn’t have anything that you need. But because Wendy is well-respected by others in your social group, they are likely to offer favors to her. Therefore, it’s useful for Wendy to like you, because then they are more apt to call on other people’s favors on your behalf.

(All the usual caveats about has this is subconscious, and humans are adaption-executors and don’t do explicit verbal assessments of how useful a person will be to them, but rely on emotional heuristics that approximate explicit assessment.)

This causes the mess of status transactions to reinforce and stabilize into a mostly-static hierarchy. The mass of individual A-privileges-B-on-the-basis-of-A’s-goals flattens out, into each person having a single “score” which determines to what degree each other person privileges them.

(It’s a little more complicated than that because people who have access to their own resources have less need of help from other. So a person’s effective status (the status-level at which you treat them is closer to their status minus your status. But this is complicated again because people are motivated not to be dicks (that’s bad for business), and respecting other people’s status is important to not being a dick.)

Initial thoughts about the early history of Circling

~ elityre ~ Leave a comment

I spent a couple of hours over the past week looking into the origins and early history of Circling, as part of a larger research project.

If you want to read some original sources, this was the most useful and informative post on the topic that I found.

You can also read my curated notes (only the things that were most interesting to me), including my thinking about the Rationality Community.

A surprising amount of the original work was done while people were in college. Notably, Bryan, Decker, and Sarah, all taught and developed Circling / AR in the living spaces of their colleges:

“Even before this, Bryan Bayer and Decker Cunov had independently discovered the practice as a tool to resolve conflicts in their shared college household in Missouri,”

“Sara had been a college student, had discovered Authentic Relating Games, had introduced them into her college dorm with great success”

It reminds me that a lot the existence and growth of EA was driven by student groups. I wonder if most movements are seeded by people in their early 20s, and therefore college campuses have been the background for the origins of most movements throughout the past century.

There’s in  way in which the teaching of Circling spread, the way the teaching of rationality didn’t.

It sounds like many of the people who frequently attended the early weekend programs that Guy and Jerry (and others) were putting on, had ambitions to develop and run similar programs of their own one day. And to a large degree, they did. There’s been something like 10 to 15 for pay circling-based programs, across at least 4 organizations. In contrast Rationality has one CFAR, that primarily runs a single program.

I wonder what accounts for the difference?

Hypotheses:

  • Circlers tend to be poor, where rationalist tend to be software engineers. Circlers could dream of doing Circling full time, but there’s not much appeal for rationalists to be teaching rationality full time. (That would be a pay cut, and there’s no “activity” that rationalist love and that they would get to do as their job.)
  • Rationality is too discrete and explicit. Once you’ve taught the rationality techniques you know, you’re done (or you have to be in the business of inventing new ones), whereas teaching Circling is more like a service: there’s not a distinct point when the student “has it” and doesn’t need your teaching, but a gradual apprenticeship.
  • Relatedly, maybe there’s just not enough demand for rationality training. A CFAR workshop is, for most rationalists, is a thing that you do once, whereas Circlers might attend several Circling immersion or trainings in a year. Rationality can become a culture and a way of life, but CFAR workshops are not. As a result, the demand for rationality training amounts to 1 workshop per community member, instead of something like 50 events per community member.
    • Notably, if CFAR had a slightly different model, this feature could change.
  • Rationality is less of concrete thing, separate from the CFAR or LessWrong brands.
    • Because of this, I think most people don’t feel enough ownership of “Rationality” as an independent thing. It’s Eliezer’s thing or CFAR’s thing. Not something that is separate from either of them.
    • Actually, the war between the founders might be relevant here. That Guy and Decker were both teaching Circling highlighted that is was separate from any one brand.
    • I wonder what the world would look like if Eliezer coined a new term for the thing we call rationality, instead of taking on a word that already has meaning in the wider world. I expect there would be less potential for a mass movement, but more and affordance to teach the thing, a feeling that one could be expert at it.
  • Maybe the fact the Circling was independently discovered by Guy and Jerry, and Decker and Bryan, made it obvious that no one owned it.
    • If we caused a second rationality-training organization to crop up, would that cause a profusion of rationality orgs?
  • Circling people acquired enough confidence in their own skills that they felt comfortable charging for them, rationalist don’t.
    • It is more obvious who the people who are skilled in circling is, because you can see it in a Circle.
    • Circling has an activity that is engaging to spend many an hour at and includes a feedback loop, so people became skilled at it in a way that rationalists don’t.

There aren’t people who are trying to build Rationality empires the way Jordan is trying to build a Circling empire.

I get the sense that a surprising number of the core people of circling are what I would call “jocks.” (Though my actual sample is pretty limited)

  • Guy originally worked as a personal trainer.
  • Sean Wilkinson and John Thompson ran a personal tennis academy before teaching Circling.
  • Jordan was a model.

“Many of us lived together in communal houses and/or were in relationships with other community members.”

They had group houses and called themselves “the community”. I wonder how common those threads are, in subcultures across time (or at least across the past century).

Goal-factoring as a tool for noticing narrative-reality disconnect

~ elityre ~ Leave a comment

[The idea of this post, as well as the opening example, were relayed to me by Ben Hoffman, who mentioned it as a thing that Michael Vassar understands well. This was written with Ben’s blessing.]

Suppose you give someone an option of one of three fruits: a radish, a carrot, and and apple. The person chooses the carrot. When you ask them why, they reply “because it’s sweet.”

Clearly, there’s something funny going on here. While the carrot is sweeter than the radish, the apple is sweeter than the carrot. So sweetness must not be the only criterion your fruit-picker is using to make his decision. He/she might be choosing partially on that basis, but there must also be some other, unmentioned factor, that is guiding his/her choice.

Now imagine someone is describing the project that they’re working on (project X). They explain their reasoning for undertaking this project, the good outcomes that will result from it: reasons a, b, and c.

When someone is presenting their reasoning like this, it can be useful to take a, be and c as premises, and try and project what seems to you like the best course of action that optimizes for those goals. That is, do a quick goal-factoring, to see if you can discover a y, that seems to fulfill goals a, b, and c, better than X does.

If you can come up with such a Y, this is suggestive of some unmentioned factor in your interlocutor’s reasoning, just as there was in the choice of your fruit-picker.

Of course this could be innocuous. Maybe Y has some drawback you’re unaware of, and so actually X is the better plan. Maybe the person you’re speaking with just hadn’t thought of Y.

But but it also might be he/she’s lying outright about why he/she’s doing X. Or maybe he/she has some motive that he/she’s not even admitting to him/herself.

Whatever the case, the procedure of taking someone else’s stated reasons as axioms and then trying to build out the best plan that satisfies them is a useful procedure for drawing out dynamics that are driving situations under the surface.

I’ve long used this technique effectively on myself, but I sugest that it might be an important lens for viewing the actions of institutions and other people. It’s often useful to tease out exactly how their declared stories about themselves deviate from their revealed agency, and this is one way of doing that.

 

 

When do you need traditions? – A hypothesis

~ elityre ~ Leave a comment

[epistemic status: speculation about domains I have little contact with, and know little about]

I’m rereading Samo Burja’s draft, Great Founder Theory. In particular, I spent some time today thinking about living, dead, and lost traditions and chains of Master-Apprenticeship relationships.

It seems like these chains often form the critical backbone of a continuing tradition (and when they fail, the tradition starts to die). Half of Nobel winners are the students of other Noble winners.

But it also seems like there are domains that don’t rely, or at least don’t need to rely on the conveyance of tacit knowledge via Master-Appreticeship relationships.

For instance, many excellent programmers are self-taught. It doesn’t seem like our civilization’s collective skill in programming depends on current experts passing on their knowledge to the next generation via close in-person contact. As a thought experiment, if all current programers disappeared today, but the computers and educational materials remained, I expect we would return to our current level of collective programing skill within a few decades.

In contrast, consider math. I know almost nothing about higher mathematics, but I would guess that if all now-living mathematicians disappeared, they’ed leave a lot of math, but progress on the frontiers of mathematics would halt, and it would take many years, maybe centuries, for mathematical progress to catch up to that frontier again. I make this bold posit on the basis of the advice I’ve heard (and I’ve personally verified) that learning from tutors is way more effective than learning just from textbooks, and that mathematicians do track their lineages.

In any case, it doesn’t seem like great programers run in lineages the way that Nobel Laureates do.

This is in part because programming in particular has some features that lends itself to autodidactictry: in particular, a novice programer gets clear and immediate feedback: his/her code either compiles or it doesn’t. But I don’t think this is the full story.

Samo discusses some of the factors that determine this difference in his document: for instance, traditions in domains that provide easy affordance for “checking work” against the territory  (such as programming) tend to be more resilient.

But I want to dig into a more specific difference.

Theory:

A domain of skill entails some process that when applied, produces some output.

Gardening is the process, fruits are the output. Carpentry (or some specific construction procedure) is the process, the resulting chair is the output.  Painting is the process, the painting is the output.

To the degree that the output is or embodies the generating process, master-apprenticeship relationships are less necessary.

It’s a well trodden trope that a program is the programmer’s thinking about a problem. (Paul Graham in Holding a Program in One’s Head: “Your code is your understanding of the problem you’re exploring.“) A comparatively large portion of a programmer’s thought process is represented in his/her program (including the comments). A novice programer, looking at a program written by a master, can see not just what a well-written program looks like, but also, to a large degree, what sort of thinking produces a well-writen program. Much of the tacit knowledge is directly expressed in the final product.

Compare this to say, a revolutionary scientist. A novice scientist might read the papers of elite groundbreaking science, and the novice might learn something, but so much of the process – the intuition that the topic in question was worth investigating, the subtle thought process that led to the hypothesis, the insight of what experiment would elegantly investigate that hypothesis – are not encoded in the paper, and are not legible to the reader.

I think that this is a general feature of domains. And this feature is predictive of the degree to which skill in a given domain relies strongly on traditions of Master- Apprenticeship.

Other examples:

I have the intuition, perhaps false (are there linages of award-winning novelist the way there are linages of Nobel laureates?), that novelists mostly do not learn their craft in apprenticeships to other writers. I suggest that writing is like programing: largely self-taught, except in the sense that one ingests and internalizes large numbers of masterful works. But enough of the skill of writing great novels is contained in the finished work that new novelists can be “trained” this way.

What about Japanese wood-block printing? From the linked video, it seems as if David Bull received about an hour of instruction in wood carving once every seven years or so. But those hours were enormously productive for him. Notably, this sort of wood-carving is a step removed from the final product: one carves the printing block, and then uses the block to make a print. Looking at the finished block, it seems, does not sufficiently convey the techniques used for creating the block. But on top of that the block is not the final product, only an intermediate step. The novice outside of an apprenticeship may only ever see the prints of a master-piece, not the blocks that make the prints.

Does this hold up at all?

That’s the theory. However, I can come up with at least a few counter proposals and confounding factors:

Countertheory: The dominating factor is the age of the tradition. Computer Science is only a few decades old, so recreating it can’t take more than a few decades. Let it develop for a few more centuries (without the advent of machine intelligence or other transformative technology), and the Art of Programming will have progressed so far that it does depend on Master/Apprentice relationships, and the loss of all living programers would be as much as a hit as the loss of all living mathematicians.

This doesn’t seem like it explains novelists, but maybe “good writing” is mostly a matter of fad? (I expect some literary connoisseurs would leap down my throat at that. In any case, it doesn’t seem correct to me.)

Confounder: economic incentive: If we lost all masters of Japanese wood-carving, but there was as much economic incentive for the civilization to remaster it as there would be for remastering programming, would it take any longer? I find that dubious.

Why does this matter? 

Well for one thing, if you’re in the business of building traditions to last more than a few decades, it’s pretty important to know when you will need to institute close-contact lineages.

Separately, this seem relevant whenever one is hoping to learn from dead masters.

Darwin surely counts among the great scientific-thinkers. He successfully abstracted out a fundamental structuring principle of the natural world. As someone interested in epistemology, it seems promising to read Darwin, in order to tease out how he was thinking. I was previously planning to read the Origins of Species. Now, it seems much more fruitful to read Darwin’s notebooks, which I expect to contain more of his process than his finished works do.

 

 

 

Initial Comparison between RAND and the Rationality Cluster

~ elityre ~ 2 Comments

I’m currently reading The Doomsday Machine: Confessions of a Nuclear War Planner by Daniel Ellsberg (the man who leaked the Pentagon Papers), on the suggestion of Anna Salamon.

I’m interested in the cold war planning communities because they might be relevant to the sort of thinking that is happening, or needs to happen, around AI x-risk, today. And indeed, there are substantial resemblances between the RAND corporation and at least some of the orgs that form the core of the contemporary x-risk ecosystem.

For instance…

A narrative of “saving the world”:

[M]y colleagues were driven men. They shared a feeling—soon transmitted to me—that we were in the most literal sense working to save the world. A successful Soviet nuclear attack on the United States would be a catastrophe, and not only for America.

A perception of the inadequacy of the official people in power:

But above all, precisely in my early missile-gap years at RAND and as a consultant in Washington, there was our sense of mission, the burden of believing we knew more about the dangers ahead, and what might be done about them, than did the generals in the Pentagon or SAC, or Congress or the public, or even the president. It was an enlivening burden.

We were rescuing the world from our Soviet counterparts as well as from the possibly fatal lethargy and bureaucratic inertia of the Eisenhower administration and our sponsors in the Air Force.

Furthermore, a major theme of the book is the insanity of US Nuclear Command and Control polices.  Ellsberg points repeatedly at the failures of decision-making and morality amongst the US government.

A sense of intellectual camaraderie:

In the middle of the first session, I ventured—though I was the youngest, assigned to be taking notes, and obviously a total novice on the issues—to express an opinion. (I don’t remember what it was.) Rather than showing irritation or ignoring my comment, Herman Kahn, brilliant and enormously fat, sitting directly across the table from me, looked at me soberly and said, “You’re absolutely wrong.” A warm glow spread throughout my body. This was the way my undergraduate fellows on the editorial board of the Harvard Crimson (mostly Jewish, like Herman and me) had routinely spoken to each other; I hadn’t experienced anything like it for six years. At King’s College, Cambridge, or in the Society of Fellows, arguments didn’t remotely take this gloves-off, take-no-prisoners form. I thought, “I’ve found a home.”

Visceral awareness of existential failure:

At least some of the folks at RAND had a visceral sense of the impending end of the world. They didn’t feel like they were just playing intellectual games.

I couldn’t believe that the world would long escape nuclear holocaust. Alain Enthoven and I were the youngest members of the department. Neither of us joined the extremely generous retirement plan RAND offered. Neither of us believed, in our late twenties, we had a chance of collecting on it.

That last point seems particularly relevant. Folks in our cluster invest in the development and practice of tools like IDC in part because of the psychological pressures that accompany the huge stakes of x-risk.

At least some of the “defense intellectuals” of the Cold War were under similar pressures.[1]

For this reason, the social and intellectual climate around RAND and similar organizations during the Cold War represents an important case study, a second data point for comparison to our contemporaries working on existential risk.

How did RAND employees handle the psychological pressures? Did they spontaneously invent strategies for thinking clearly in the face of the magnitude of the stakes? If so, can we emulate those strategies? If not, does that imply that their thinking about their work was compromised? Or does it suggest that our emphasis on psychological integration methods are misplaced?

And perhaps most importantly, what mistakes did they make? Can we use their example to foresee similar mistakes of our own and avoid them?

[1] – Indeed, it seems like they were under greater pressures. There’s a sense of franticness and urgency that I feel in Ellsberg’s description that I don’t feel around MIRI. But I think that this is due to the time horizons that RAND and co. were operating under compared to the those that MIRI is operating under. I expect that as we near the arrival of AGI, there will be a sense of urgency and psychological pressure that is just as great and greater than that of the cold war planners.

End note: In addition to all these more concrete correlations, there’s also the intriguing intertwining of existential risk and decision theory in both of the data points of nuclear war planning and AI safety. I wonder if that is merely coincidence or represents some deeper connection.

Something simple to try in conversations

~ elityre ~ 2 Comments

Last night, I was outlining conversational techniques. By “conversational technique” I mean things like “ask for an example/ generate a hypothesis”, “repeat back, in your own words, what the other person just said”, “consider what would make you change my mind”, etc. and the times when it would be useful to use them, so that I could make more specific Trigger Action Plans. I quickly noticed a way of carving up the space which seemed useful to me and potentially interesting to my (currently non-existent) readers.

In a conversation, second to second, you may be trying to understand what another person is saying, or you may be trying to help them understand what you are trying to convey. There are perhaps some other possibilities, such as trying to figure out a new domain together, but even then, at any given moment one of you is likely to be explaining and the other listening.

It seems quite useful to me, to be tracking which you are aiming to do at any given moment, understand – or get someone to understand.

Being aware of who is doing what allows two conversationalists to coordinate, verbally and explicitly, if need be. A conversation is apt to go better when participant A is focused on listening while participant B is focused on explicating, and vise versa. Discussions often become less manageable when both parties are too busy explaining to listen.

Before I start training more specific conversational TAPs, I’ve started paying attention to which of the two I’m doing second to second.

Approaches to this thing called “Rationality” (or alternatively, a history of our lineage)

~ elityre ~ Leave a comment

[Posted to the CFAR mailing list]

[Somewhat experimental: Looking for thumbs up and thumbs down on this kind of writing. I’m trying to clarify some of the fuzziness around why we are calling the-thing-some-of-us-are-calling-rationality “rationality.”]

So what is this rationality thing anyway?

Simply stated, some behavior works better than other behavior for achieving a given goal. In fact, for formal and well defined environments, “games”, this is provably true. In the early to mid 20th century, academic mathematicians developed game theory and decision theory, mathematical formalizations of idealized decision algorithms that give provably optimal outcomes (in expectation).

One school of rationality (let’s call it “formal rationality”) is largely about learning and relying on these decision rules. For a rationalist of this type, progress in the field means doing more math, and discovering more theorems or decisions rules. Since most non-trivial decision problems involve dealing with uncertainty, and uncertainty in the real world is quantified using statistics, statistics is central to the practice of formal rationality.

MIRI does the sort of work that a formal rationalist would consider to be progress on rationality: trying to develop solutions to decision theory problems. (This is NOT to say that most, or even any of the actual people who work are MIRI are themselves of the “formal rationality” school as opposed to those to follow. In fact I have reason to think that NONE of them would identify as such.) The other large frontiers of “formal rationality” are mostly in economics. The economy can be thought of as a single gigantic game theoretic game.

For the formal rationalist, rationality is almost entirely solved. We have game theory. We have probability theory. We have decision theory. There may be edge-case scenarios that need to be solved (pascal’s mugging, for instance), but for the most part, the “art” has already been invented. Declaring oneself a rationalist in the formal sense is a statement of philosophy: it means you trust the approximations of the formal decision rules over intuition, common sense, tradition, or well, anything.  One doesn’t need to qualify with the word “aspiring.”

(There’s a framework nearby to formal rationality which is largely captured by the term “evidence-based.” This is the position that one should base one’s actions and beliefs on evidence, over intuition or superstition. We can call this traditional rationality.  Traditional rationality includes science, and evidence-seeking in general.)

If you have formalized decision rules that describe the behaviour of goal directed agents, you now have the affordance to check what humans are actually doing. Enter Kahneman and Tversky. Over the course of the 1970’s to 1990’s ,they do many experiments and determine that 1) most people are not optimal goal-directed agents, (i.e. they are “irrational”. Little surprise to anyone, I think), 2) that those with advanced knowledge of “formal rationality” (e.g. statistics, economics, probability theory, game theory, decision theory) also fail to be optimal goal-directed agents (WE’re irrational too), and 3) that humans tend to deviate from ideal behaviour in systematic, predictable ways.

Thus develops the Heuristics and Biases project in psychology, with gives to rise another approach to the project of rationality. If humans are intrinsically and systematically biased, and simply telling a person about the bais doesn’t fix it (as is often the case), then the greater share of rationality training involves coming up with methods to counteract native cognitive bias. We can call this approach to rationality “the debiasing approach.” It inherits many of the formalizations from formal rationality (which do reflect ideal behavior), but the emphasis is on dealing with the actual human mind and correcting it’s faults. The project of rationality involves math, but now it is mostly in the domain of psychology.

This is in large part, the approach to rationality that Eliezer took in the sequences (though the sequences are a philosophical treatise, and his aims went beyond debiasing), and it fairly well characterizes LessWrong.

In 2012, CFAR is founded, and initially takes the debiasing approach. But the organization pretty quickly pivots away from that sort of model (you’ll notice that there are no modules in the current workshop of the form “this is the X fallacy/bias and here is the technique that eliminates or mitigates it.”) Developing debiasing protocols proves to be difficult, but there’s a nearby thing which is very useful and and much more tractable. CFAR borrows the System 1 / System 2 framework from heuristics and biases and develops methods to get those facets of the mind to communicate with one another.

For instance, sometimes a person intellectually endorses an action but doesn’t feel emotionally motivated about it. Propagating urges (or aversion factoring) is a technique that facilitates the dialogue between those parts of the mind, such that one (or both) of them updates and they are both on the same page. Similarly, sometimes a part of the mind has information about how likely a project is to succeed, but that data needs to be queried to be useful. The Inner Simulator / Murphyjitsu is a technique that lets the conscious, verbal system query the part of the mind that automatically makes predictions of that sort.

This approach isn’t about mitigating specific biases, but rather propagating information that one already has in one part of his/her mind to other parts of the the cognitive system. We can call this approach the “parts-propagation approach.” It’s about unifying the mind (minding our way style, mostly, but not exclusively) such that all parts of the mind are on the same page and pulling in the same direction, so that we can be more “agenty ducks” (i.e., better approximations of the simplified goal-directed agents of formal rationality, with stable, ordered goals) that can get shit done in the world.

These are three rather different approaches to rationality, and each one entails very different ideas of what the “art of rationality” should look like and what directions research should take. I have thoughts about which of these approaches are most tractable and important, but my goal here is only to clarify some of the confusion about what is meant by “rationality” and why.

Thoughts? Are these categories good ones? Do they carve reality at the joints?