Tag: skill lineages

Culture vs. Mental Habits

~ elityre ~ Leave a comment

[epistemic status: personal view of the rationality community.]

In this “post”, I’m going to outline two dimensions on which one could assess the rationality community and the success of the rationality project. This is hardly the only possible break-down, but it is one that underlies a lot of my thinking about rationality community building, and what I would do, if I decided rationality community building were a strong priority.

I’m going to call those two dimensions Culture and Mental Habits. As we’ll see these are not cleanly distinct categories, and they tend to bleed into each other. But they have separate enough focuses that one can meaningfully talk about the differences between them.

Culture

By “culture” I mean something like…

  • Which good things are prioritized?
  • Which actions and behaviors are socially rewarded?
  • Which concepts and ideas are in common parlance?

Culture is about groups of people, what those groups share and what they value.

My perception is that on this dimension, the Bay area rationality community has done extraordinarily well.

Truth-seeking is seen as paramount: individuals are socially rewarded for admitting ignorance and changing their minds. Good faith and curiosity about other people’s beliefs is common.

Analytical and quantitative reasoning is highly respected, and increasingly, so is embodied intuition.

People get status for doing good scholarship (e.g. Sarah Constantin), for insightful analysis of complicated situations (e.g. Scott Alexander, for instance), or for otherwise producing good or interesting intellectual content (e.g. Eliezer).

Betting (putting your money where your mouth is) is socially-encouraged. Concepts like “crux” and “rationalist taboo” are well known enough to be frequently invoked in conversation.

Compared to the backdrop of mainline American culture, where admitting that you were wrong means losing face, and trying to figure out what’s true is secondary (if not outright suspicious, since it suggests political non-allegiance), the rationalist bubble’s culture of truth seeking is an impressive accomplishment.

Mental habits

For lack of a better term, I’m going to call this second dimension “mental habits” (or perhaps to borrow Leverage’s term “IPs”).

The thing that I care about in this category is “does a given individual reliably execute some specific cognitive move, when the situation calls for it?” or “does a given individual systematically avoid a given cognitive error?”

Some examples, to gesture at what I mean

  • Never falling prey to the planning fallacy
  • Never falling prey to sunk costs
  • Systematically noticing defensiveness and deflinching or a similar move
  • Systematically noticing and responding to rationalization phenomenology
  • Implementing the “say oops” skill, when new evidence comes to light that overthrows an important position of yours
  • Systematic avoidance of the sorts of errors I outline my Cold War Cognitive Errors investigation (this is the only version that is available at this time).

The element of reliability is crucial. There’s a way that culture is about “counting up” (some people know concept X, and use it sometimes) and mental habits is about “counting down” (each person rarely fails to execute relevant mental process Y).

The reliability of mental habits (in contrast with some mental motion that you know how to do and have done once or twice), is crucial, because it puts one in a relevantly different paradigm.

For one thing, there’s a frame under which rationality is about avoiding failure modes: how to succeed in a given domain depends on the domain, but rationality is about how not to fail, generally. Under that frame, executing the correct mental motion 10% of the time is much less interesting and impressive than executing it everytime (or even 90% of the time).

If the goal is to avoid the sorts of errors in my cold war post, then it is not even remotely sufficient for individuals to be familiar with the patches: they have to reliably notice the moments of intervention and execute the patches, almost every time, in order to avoid the error in the crucial moment.

Furthermore, systematic execution of a mental TAP allows for more complicated cognitive machines. Lots of complex skills depend on all of the pieces of the skills working.

It seems to me, that along this dimension, the rationality community has done dismally.

Eliezer wrote about Mental Habits of this sort in the sequences and in his other writing, but when I consider even very advanced members of my community, I think very few of them systematically notice rationalization, or will reliably avoid sunk costs, or consistently respond to their own defensiveness.

I see very few people around me who explicitly attempt to train 5-second or smaller rationality skills. (Anna and Matt Fallshaw are exceptions who come to mind).

Anna gave a talk at the CFAR alumni reunion this year, in which she presented two low-level cognitive skills of that sort. There were about 40 people in the room watching the lecture, but I would be mildly surprised if even 2 of those people reliably execute the skills described, in the relevant-trigger situation, 6 months from that talk.

But I can imagine a nearby world, where the rationality community was more clearly a community of practice, and most of the the people in that room, would watch that talk and then train the cognitive habit to that level of reliability.

This is not to say that fast cognitive skills of this sort are what we should be focusing on. I can see arguments that culture really is the core thing. But nevertheless, it seems to me that the rationality community is not excelling on the dimension of training it’s members in mental TAPs.

[Added note: Brienne’s Tortoise skills is nearly archetypal of what I mean by “mental habits”.]

When do you need traditions? – A hypothesis

~ elityre ~ Leave a comment

[epistemic status: speculation about domains I have little contact with, and know little about]

I’m rereading Samo Burja’s draft, Great Founder Theory. In particular, I spent some time today thinking about living, dead, and lost traditions and chains of Master-Apprenticeship relationships.

It seems like these chains often form the critical backbone of a continuing tradition (and when they fail, the tradition starts to die). Half of Nobel winners are the students of other Noble winners.

But it also seems like there are domains that don’t rely, or at least don’t need to rely on the conveyance of tacit knowledge via Master-Appreticeship relationships.

For instance, many excellent programmers are self-taught. It doesn’t seem like our civilization’s collective skill in programming depends on current experts passing on their knowledge to the next generation via close in-person contact. As a thought experiment, if all current programers disappeared today, but the computers and educational materials remained, I expect we would return to our current level of collective programing skill within a few decades.

In contrast, consider math. I know almost nothing about higher mathematics, but I would guess that if all now-living mathematicians disappeared, they’ed leave a lot of math, but progress on the frontiers of mathematics would halt, and it would take many years, maybe centuries, for mathematical progress to catch up to that frontier again. I make this bold posit on the basis of the advice I’ve heard (and I’ve personally verified) that learning from tutors is way more effective than learning just from textbooks, and that mathematicians do track their lineages.

In any case, it doesn’t seem like great programers run in lineages the way that Nobel Laureates do.

This is in part because programming in particular has some features that lends itself to autodidactictry: in particular, a novice programer gets clear and immediate feedback: his/her code either compiles or it doesn’t. But I don’t think this is the full story.

Samo discusses some of the factors that determine this difference in his document: for instance, traditions in domains that provide easy affordance for “checking work” against the territory  (such as programming) tend to be more resilient.

But I want to dig into a more specific difference.

Theory:

A domain of skill entails some process that when applied, produces some output.

Gardening is the process, fruits are the output. Carpentry (or some specific construction procedure) is the process, the resulting chair is the output.  Painting is the process, the painting is the output.

To the degree that the output is or embodies the generating process, master-apprenticeship relationships are less necessary.

It’s a well trodden trope that a program is the programmer’s thinking about a problem. (Paul Graham in Holding a Program in One’s Head: “Your code is your understanding of the problem you’re exploring.“) A comparatively large portion of a programmer’s thought process is represented in his/her program (including the comments). A novice programer, looking at a program written by a master, can see not just what a well-written program looks like, but also, to a large degree, what sort of thinking produces a well-writen program. Much of the tacit knowledge is directly expressed in the final product.

Compare this to say, a revolutionary scientist. A novice scientist might read the papers of elite groundbreaking science, and the novice might learn something, but so much of the process – the intuition that the topic in question was worth investigating, the subtle thought process that led to the hypothesis, the insight of what experiment would elegantly investigate that hypothesis – are not encoded in the paper, and are not legible to the reader.

I think that this is a general feature of domains. And this feature is predictive of the degree to which skill in a given domain relies strongly on traditions of Master- Apprenticeship.

Other examples:

I have the intuition, perhaps false (are there linages of award-winning novelist the way there are linages of Nobel laureates?), that novelists mostly do not learn their craft in apprenticeships to other writers. I suggest that writing is like programing: largely self-taught, except in the sense that one ingests and internalizes large numbers of masterful works. But enough of the skill of writing great novels is contained in the finished work that new novelists can be “trained” this way.

What about Japanese wood-block printing? From the linked video, it seems as if David Bull received about an hour of instruction in wood carving once every seven years or so. But those hours were enormously productive for him. Notably, this sort of wood-carving is a step removed from the final product: one carves the printing block, and then uses the block to make a print. Looking at the finished block, it seems, does not sufficiently convey the techniques used for creating the block. But on top of that the block is not the final product, only an intermediate step. The novice outside of an apprenticeship may only ever see the prints of a master-piece, not the blocks that make the prints.

Does this hold up at all?

That’s the theory. However, I can come up with at least a few counter proposals and confounding factors:

Countertheory: The dominating factor is the age of the tradition. Computer Science is only a few decades old, so recreating it can’t take more than a few decades. Let it develop for a few more centuries (without the advent of machine intelligence or other transformative technology), and the Art of Programming will have progressed so far that it does depend on Master/Apprentice relationships, and the loss of all living programers would be as much as a hit as the loss of all living mathematicians.

This doesn’t seem like it explains novelists, but maybe “good writing” is mostly a matter of fad? (I expect some literary connoisseurs would leap down my throat at that. In any case, it doesn’t seem correct to me.)

Confounder: economic incentive: If we lost all masters of Japanese wood-carving, but there was as much economic incentive for the civilization to remaster it as there would be for remastering programming, would it take any longer? I find that dubious.

Why does this matter? 

Well for one thing, if you’re in the business of building traditions to last more than a few decades, it’s pretty important to know when you will need to institute close-contact lineages.

Separately, this seem relevant whenever one is hoping to learn from dead masters.

Darwin surely counts among the great scientific-thinkers. He successfully abstracted out a fundamental structuring principle of the natural world. As someone interested in epistemology, it seems promising to read Darwin, in order to tease out how he was thinking. I was previously planning to read the Origins of Species. Now, it seems much more fruitful to read Darwin’s notebooks, which I expect to contain more of his process than his finished works do.