When do you need traditions? – A hypothesis

~ elityre ~ Leave a comment

[epistemic status: speculation about domains I have little contact with, and know little about]

I’m rereading Samo Burja’s draft, Great Founder Theory. In particular, I spent some time today thinking about living, dead, and lost traditions and chains of Master-Apprenticeship relationships.

It seems like these chains often form the critical backbone of a continuing tradition (and when they fail, the tradition starts to die). Half of Nobel winners are the students of other Noble winners.

But it also seems like there are domains that don’t rely, or at least don’t need to rely on the conveyance of tacit knowledge via Master-Appreticeship relationships.

For instance, many excellent programmers are self-taught. It doesn’t seem like our civilization’s collective skill in programming depends on current experts passing on their knowledge to the next generation via close in-person contact. As a thought experiment, if all current programers disappeared today, but the computers and educational materials remained, I expect we would return to our current level of collective programing skill within a few decades.

In contrast, consider math. I know almost nothing about higher mathematics, but I would guess that if all now-living mathematicians disappeared, they’ed leave a lot of math, but progress on the frontiers of mathematics would halt, and it would take many years, maybe centuries, for mathematical progress to catch up to that frontier again. I make this bold posit on the basis of the advice I’ve heard (and I’ve personally verified) that learning from tutors is way more effective than learning just from textbooks, and that mathematicians do track their lineages.

In any case, it doesn’t seem like great programers run in lineages the way that Nobel Laureates do.

This is in part because programming in particular has some features that lends itself to autodidactictry: in particular, a novice programer gets clear and immediate feedback: his/her code either compiles or it doesn’t. But I don’t think this is the full story.

Samo discusses some of the factors that determine this difference in his document: for instance, traditions in domains that provide easy affordance for “checking work” against the territory  (such as programming) tend to be more resilient.

But I want to dig into a more specific difference.

Theory:

A domain of skill entails some process that when applied, produces some output.

Gardening is the process, fruits are the output. Carpentry (or some specific construction procedure) is the process, the resulting chair is the output.  Painting is the process, the painting is the output.

To the degree that the output is or embodies the generating process, master-apprenticeship relationships are less necessary.

It’s a well trodden trope that a program is the programmer’s thinking about a problem. (Paul Graham in Holding a Program in One’s Head: “Your code is your understanding of the problem you’re exploring.“) A comparatively large portion of a programmer’s thought process is represented in his/her program (including the comments). A novice programer, looking at a program written by a master, can see not just what a well-written program looks like, but also, to a large degree, what sort of thinking produces a well-writen program. Much of the tacit knowledge is directly expressed in the final product.

Compare this to say, a revolutionary scientist. A novice scientist might read the papers of elite groundbreaking science, and the novice might learn something, but so much of the process – the intuition that the topic in question was worth investigating, the subtle thought process that led to the hypothesis, the insight of what experiment would elegantly investigate that hypothesis – are not encoded in the paper, and are not legible to the reader.

I think that this is a general feature of domains. And this feature is predictive of the degree to which skill in a given domain relies strongly on traditions of Master- Apprenticeship.

Other examples:

I have the intuition, perhaps false (are there linages of award-winning novelist the way there are linages of Nobel laureates?), that novelists mostly do not learn their craft in apprenticeships to other writers. I suggest that writing is like programing: largely self-taught, except in the sense that one ingests and internalizes large numbers of masterful works. But enough of the skill of writing great novels is contained in the finished work that new novelists can be “trained” this way.

What about Japanese wood-block printing? From the linked video, it seems as if David Bull received about an hour of instruction in wood carving once every seven years or so. But those hours were enormously productive for him. Notably, this sort of wood-carving is a step removed from the final product: one carves the printing block, and then uses the block to make a print. Looking at the finished block, it seems, does not sufficiently convey the techniques used for creating the block. But on top of that the block is not the final product, only an intermediate step. The novice outside of an apprenticeship may only ever see the prints of a master-piece, not the blocks that make the prints.

Does this hold up at all?

That’s the theory. However, I can come up with at least a few counter proposals and confounding factors:

Countertheory: The dominating factor is the age of the tradition. Computer Science is only a few decades old, so recreating it can’t take more than a few decades. Let it develop for a few more centuries (without the advent of machine intelligence or other transformative technology), and the Art of Programming will have progressed so far that it does depend on Master/Apprentice relationships, and the loss of all living programers would be as much as a hit as the loss of all living mathematicians.

This doesn’t seem like it explains novelists, but maybe “good writing” is mostly a matter of fad? (I expect some literary connoisseurs would leap down my throat at that. In any case, it doesn’t seem correct to me.)

Confounder: economic incentive: If we lost all masters of Japanese wood-carving, but there was as much economic incentive for the civilization to remaster it as there would be for remastering programming, would it take any longer? I find that dubious.

Why does this matter? 

Well for one thing, if you’re in the business of building traditions to last more than a few decades, it’s pretty important to know when you will need to institute close-contact lineages.

Separately, this seem relevant whenever one is hoping to learn from dead masters.

Darwin surely counts among the great scientific-thinkers. He successfully abstracted out a fundamental structuring principle of the natural world. As someone interested in epistemology, it seems promising to read Darwin, in order to tease out how he was thinking. I was previously planning to read the Origins of Species. Now, it seems much more fruitful to read Darwin’s notebooks, which I expect to contain more of his process than his finished works do.

 

 

 

Initial Comparison between RAND and the Rationality Cluster

~ elityre ~ 2 Comments

I’m currently reading The Doomsday Machine: Confessions of a Nuclear War Planner by Daniel Ellsberg (the man who leaked the Pentagon Papers), on the suggestion of Anna Salamon.

I’m interested in the cold war planning communities because they might be relevant to the sort of thinking that is happening, or needs to happen, around AI x-risk, today. And indeed, there are substantial resemblances between the RAND corporation and at least some of the orgs that form the core of the contemporary x-risk ecosystem.

For instance…

A narrative of “saving the world”:

[M]y colleagues were driven men. They shared a feeling—soon transmitted to me—that we were in the most literal sense working to save the world. A successful Soviet nuclear attack on the United States would be a catastrophe, and not only for America.

A perception of the inadequacy of the official people in power:

But above all, precisely in my early missile-gap years at RAND and as a consultant in Washington, there was our sense of mission, the burden of believing we knew more about the dangers ahead, and what might be done about them, than did the generals in the Pentagon or SAC, or Congress or the public, or even the president. It was an enlivening burden.

We were rescuing the world from our Soviet counterparts as well as from the possibly fatal lethargy and bureaucratic inertia of the Eisenhower administration and our sponsors in the Air Force.

Furthermore, a major theme of the book is the insanity of US Nuclear Command and Control polices.  Ellsberg points repeatedly at the failures of decision-making and morality amongst the US government.

A sense of intellectual camaraderie:

In the middle of the first session, I ventured—though I was the youngest, assigned to be taking notes, and obviously a total novice on the issues—to express an opinion. (I don’t remember what it was.) Rather than showing irritation or ignoring my comment, Herman Kahn, brilliant and enormously fat, sitting directly across the table from me, looked at me soberly and said, “You’re absolutely wrong.” A warm glow spread throughout my body. This was the way my undergraduate fellows on the editorial board of the Harvard Crimson (mostly Jewish, like Herman and me) had routinely spoken to each other; I hadn’t experienced anything like it for six years. At King’s College, Cambridge, or in the Society of Fellows, arguments didn’t remotely take this gloves-off, take-no-prisoners form. I thought, “I’ve found a home.”

Visceral awareness of existential failure:

At least some of the folks at RAND had a visceral sense of the impending end of the world. They didn’t feel like they were just playing intellectual games.

I couldn’t believe that the world would long escape nuclear holocaust. Alain Enthoven and I were the youngest members of the department. Neither of us joined the extremely generous retirement plan RAND offered. Neither of us believed, in our late twenties, we had a chance of collecting on it.

That last point seems particularly relevant. Folks in our cluster invest in the development and practice of tools like IDC in part because of the psychological pressures that accompany the huge stakes of x-risk.

At least some of the “defense intellectuals” of the Cold War were under similar pressures.[1]

For this reason, the social and intellectual climate around RAND and similar organizations during the Cold War represents an important case study, a second data point for comparison to our contemporaries working on existential risk.

How did RAND employees handle the psychological pressures? Did they spontaneously invent strategies for thinking clearly in the face of the magnitude of the stakes? If so, can we emulate those strategies? If not, does that imply that their thinking about their work was compromised? Or does it suggest that our emphasis on psychological integration methods are misplaced?

And perhaps most importantly, what mistakes did they make? Can we use their example to foresee similar mistakes of our own and avoid them?

[1] – Indeed, it seems like they were under greater pressures. There’s a sense of franticness and urgency that I feel in Ellsberg’s description that I don’t feel around MIRI. But I think that this is due to the time horizons that RAND and co. were operating under compared to the those that MIRI is operating under. I expect that as we near the arrival of AGI, there will be a sense of urgency and psychological pressure that is just as great and greater than that of the cold war planners.

End note: In addition to all these more concrete correlations, there’s also the intriguing intertwining of existential risk and decision theory in both of the data points of nuclear war planning and AI safety. I wonder if that is merely coincidence or represents some deeper connection.

Something simple to try in conversations

~ elityre ~ 2 Comments

Last night, I was outlining conversational techniques. By “conversational technique” I mean things like “ask for an example/ generate a hypothesis”, “repeat back, in your own words, what the other person just said”, “consider what would make you change my mind”, etc. and the times when it would be useful to use them, so that I could make more specific Trigger Action Plans. I quickly noticed a way of carving up the space which seemed useful to me and potentially interesting to my (currently non-existent) readers.

In a conversation, second to second, you may be trying to understand what another person is saying, or you may be trying to help them understand what you are trying to convey. There are perhaps some other possibilities, such as trying to figure out a new domain together, but even then, at any given moment one of you is likely to be explaining and the other listening.

It seems quite useful to me, to be tracking which you are aiming to do at any given moment, understand – or get someone to understand.

Being aware of who is doing what allows two conversationalists to coordinate, verbally and explicitly, if need be. A conversation is apt to go better when participant A is focused on listening while participant B is focused on explicating, and vise versa. Discussions often become less manageable when both parties are too busy explaining to listen.

Before I start training more specific conversational TAPs, I’ve started paying attention to which of the two I’m doing second to second.

Approaches to this thing called “Rationality” (or alternatively, a history of our lineage)

~ elityre ~ Leave a comment

[Posted to the CFAR mailing list]

[Somewhat experimental: Looking for thumbs up and thumbs down on this kind of writing. I’m trying to clarify some of the fuzziness around why we are calling the-thing-some-of-us-are-calling-rationality “rationality.”]

So what is this rationality thing anyway?

Simply stated, some behavior works better than other behavior for achieving a given goal. In fact, for formal and well defined environments, “games”, this is provably true. In the early to mid 20th century, academic mathematicians developed game theory and decision theory, mathematical formalizations of idealized decision algorithms that give provably optimal outcomes (in expectation).

One school of rationality (let’s call it “formal rationality”) is largely about learning and relying on these decision rules. For a rationalist of this type, progress in the field means doing more math, and discovering more theorems or decisions rules. Since most non-trivial decision problems involve dealing with uncertainty, and uncertainty in the real world is quantified using statistics, statistics is central to the practice of formal rationality.

MIRI does the sort of work that a formal rationalist would consider to be progress on rationality: trying to develop solutions to decision theory problems. (This is NOT to say that most, or even any of the actual people who work are MIRI are themselves of the “formal rationality” school as opposed to those to follow. In fact I have reason to think that NONE of them would identify as such.) The other large frontiers of “formal rationality” are mostly in economics. The economy can be thought of as a single gigantic game theoretic game.

For the formal rationalist, rationality is almost entirely solved. We have game theory. We have probability theory. We have decision theory. There may be edge-case scenarios that need to be solved (pascal’s mugging, for instance), but for the most part, the “art” has already been invented. Declaring oneself a rationalist in the formal sense is a statement of philosophy: it means you trust the approximations of the formal decision rules over intuition, common sense, tradition, or well, anything.  One doesn’t need to qualify with the word “aspiring.”

(There’s a framework nearby to formal rationality which is largely captured by the term “evidence-based.” This is the position that one should base one’s actions and beliefs on evidence, over intuition or superstition. We can call this traditional rationality.  Traditional rationality includes science, and evidence-seeking in general.)

If you have formalized decision rules that describe the behaviour of goal directed agents, you now have the affordance to check what humans are actually doing. Enter Kahneman and Tversky. Over the course of the 1970’s to 1990’s ,they do many experiments and determine that 1) most people are not optimal goal-directed agents, (i.e. they are “irrational”. Little surprise to anyone, I think), 2) that those with advanced knowledge of “formal rationality” (e.g. statistics, economics, probability theory, game theory, decision theory) also fail to be optimal goal-directed agents (WE’re irrational too), and 3) that humans tend to deviate from ideal behaviour in systematic, predictable ways.

Thus develops the Heuristics and Biases project in psychology, with gives to rise another approach to the project of rationality. If humans are intrinsically and systematically biased, and simply telling a person about the bais doesn’t fix it (as is often the case), then the greater share of rationality training involves coming up with methods to counteract native cognitive bias. We can call this approach to rationality “the debiasing approach.” It inherits many of the formalizations from formal rationality (which do reflect ideal behavior), but the emphasis is on dealing with the actual human mind and correcting it’s faults. The project of rationality involves math, but now it is mostly in the domain of psychology.

This is in large part, the approach to rationality that Eliezer took in the sequences (though the sequences are a philosophical treatise, and his aims went beyond debiasing), and it fairly well characterizes LessWrong.

In 2012, CFAR is founded, and initially takes the debiasing approach. But the organization pretty quickly pivots away from that sort of model (you’ll notice that there are no modules in the current workshop of the form “this is the X fallacy/bias and here is the technique that eliminates or mitigates it.”) Developing debiasing protocols proves to be difficult, but there’s a nearby thing which is very useful and and much more tractable. CFAR borrows the System 1 / System 2 framework from heuristics and biases and develops methods to get those facets of the mind to communicate with one another.

For instance, sometimes a person intellectually endorses an action but doesn’t feel emotionally motivated about it. Propagating urges (or aversion factoring) is a technique that facilitates the dialogue between those parts of the mind, such that one (or both) of them updates and they are both on the same page. Similarly, sometimes a part of the mind has information about how likely a project is to succeed, but that data needs to be queried to be useful. The Inner Simulator / Murphyjitsu is a technique that lets the conscious, verbal system query the part of the mind that automatically makes predictions of that sort.

This approach isn’t about mitigating specific biases, but rather propagating information that one already has in one part of his/her mind to other parts of the the cognitive system. We can call this approach the “parts-propagation approach.” It’s about unifying the mind (minding our way style, mostly, but not exclusively) such that all parts of the mind are on the same page and pulling in the same direction, so that we can be more “agenty ducks” (i.e., better approximations of the simplified goal-directed agents of formal rationality, with stable, ordered goals) that can get shit done in the world.

These are three rather different approaches to rationality, and each one entails very different ideas of what the “art of rationality” should look like and what directions research should take. I have thoughts about which of these approaches are most tractable and important, but my goal here is only to clarify some of the confusion about what is meant by “rationality” and why.

Thoughts? Are these categories good ones? Do they carve reality at the joints?

 

Progress on brokenness

~ elityre ~ Leave a comment

[This post is about me.]

Relating to a brief interaction between Mark, Andrew, and Eli, yesterday.

Over the past few months, my sense of myself and my oddities has gone through some significant changes and sophistication.

I’ve long known that my internal processes differed in many respects from those of most people, and while there are clear drawbacks to my cognitive style, there are also clear advantages. It wasn’t necessarily clear that the advantages outweighed the drawbacks in the abstract, but I happened to like my cognitive style, and it seemed useful for reasons of variety is nothing else. In any case there was a sense of “that’s just the way Eli is.”

In my interactions with the cooler people of rationalist community, my oddities started to “come into focus” more. Instead of being black boxes of “the ways Eli is weird” I began developing much deeper causal models of what I was doing differently and why. It wasn’t an either-or-proposition: I could figure out how to refine the things that I was doing to avoid the drawbacks without blunting the advantages.

Then more recently, largely as a result of my interactions with this group, I’ve gotten a fuller sense of what my mind is doing, moment-to-moment, on a phenomenological level, and I’m excited because I think I have (or am close to having) the tools, the phenomenology, and the community to flatly resolve issues that have plagued me for my whole life, and which pretty much everyone assumed were un-fixable and would just need to be accommodated.

An anecdote: When I was in second grade, in the winter, I would wear long-sleeved shirts. I would also wash my hands a lot (due to OCD-like tendencies). However, I wouldn’t roll up my sleeves, so I would walk around with damp sleeves and even if I dried my hands, it was as if they were  constantly being submerged in water. This was in Arizona, a desert. My hands would get so dry, that they had the texture of sandpaper and they were constantly cracked to the point of bleeding. This was uncomfortable, to say the least.

My mom asked me, with exasperation, “why don’t you just roll up your sleeves?” and I responded, “because no one told me too.”

I said that a lot growing up. It was (and is) a common pattern for me. There have been many things that most kids just pick up, that most people find obvious, that I had to be told or taught explicitly.

There are a lot of things going wrong in the story above. There’s the thing that I felt like I needed to keep washing my hands. There’s clearly some sort of loop missing that does error-correction or automatic hypothesis generation or something.

If I needed to be told to roll up my sleeves, how much more am I missing subtle and implicit processes that most people don’t even have the words to describe. If that wasn’t wasn’t clear to me, how much worse am I at all the other things that people do automatically without even realizing that they are doing it?

But, this anecdote makes me optimistic. Because when someone finally told me to roll up my sleeves before washing my hands, I did. The hard part was the figuring out what to do, not the doing it. There may be (there likely are) simple things that I could be taught to do, explicitly, that are not actually much harder for me to do than for everyone else (but which I just don’t come to do automatically), which would close-to solve the cognitive bottlenecks. I suspect that there is enormously low-hanging fruit, if someone can just figure out how to point it out to me in a way I can understand.

I’m tinkering a lot, moving towards fuller abstract models and finding phenomenological levers. I’m figuring out what the pieces are. I’m trying stuff. And you guys are the best people in the world to help me figure out what those processes are.

Now, I shouldn’t be too excited. There may be thousands of implicit micro-protocols that I’m missing. I’m not sure that there are, but it seems possible. But even if that’s the case, if I can fill the most important holes that I’m missing, maybe I can bootstrap. If I can learn the process that others use to learn implicitly, explicitly, I can solve the rest myself and go FOOM.

This is super interesting to me, because 1) It would help me a bunch practically, 2) because I’m super curious about what my mind does and why, and this is an abstractly interesting project, and 3) because if many of these things are in fact hard for me to learn, but easy for me to do, then I could fill some long-standing holes very rapidly, which means that I could approximately know how to be a “normal human”, while still having access to all the exceptional competencies that I’ve been forced to develop as workarounds over the course of my 22 years missing parts of my brain. I mean, I’m a little crippled, and I’m still largely functional. What does the non-crippled version look like?
Optimism bias and stuff. Everything is harder than it seems. But I am actually making progress and I’m excited about what comes next.

My hamming problem: Making dealing with overwhelm automatic?

~ elityre ~ 1 Comment
[I posted this to a neurophenomenology mailing list, here, on February 11, 2016]
Most of my wasted time (and most of my wasted potential value), is lost in “procrastination.” But “procrastination” isn’t reductionistic enough: the phenomenon has parts.

In particular, this is due to a particular kind of aversion to a particular kind of sensation of overwellm. This overwhelm has certain characteristics. For instance, I have never (I think) experienced it as resulting from some task that didn’t have a deadline.

I think this overwhelm is the result of system 1 not believing that it can accomplish a task or several tasks. That fear is anxiety provoking. It causes my mind to glance away or to become absorbed in something that I consider to be much less important (or even just not important at all).

There’s more though. I can notice this sensation and then use some process (Aversion factoring, or Focusing) to pay attention to the anxiety. In this self-dialog I am “forced” to come to terms with the fact, that I, on reflection, want to accomplish the task (or at least want the task accomplished). [Because of the deadline] simply not doing the thing in question is not a live option, once I have stared it in the face. I’ll then, typically, negotiate between the parts of myself. “What would it take for me to want to do the thing?” I might do just one bite-sized unit as a CoZE. Usually this dialog ends with some sort of “just start”, which I then proceed to do, unless I have an available affordence which I can bullshit myself into believing takes priority over doing the aversive thing.

But, when I do, “just start”, most of the time it goes well. In one of two ways.

Sometimes, I start, and I make progress and it isn’t as bad as I thought, or as overwhelming as I thought, and I relax.

Sometimes, I start, and I make progress, and I’m feeling confused, and it still feels overwhelming, but now the next steps are clear, and the fear propels me instead of paralyzes me. I’m still aroused, but now my arousal has an outlet, and this also feels damn good, and even though I’m feeling pressured, I’m no longer aversive to looking at, thinking about, the task to be accomplished.

First of all, this phenomenon seems to indicate that my anxiety and overwhelm is the result of not knowing how to do a thing. Once the path forward is clear, I feel pressured to take it, but not knowing what the path is, I’m anxious and will take any opportunity to be distracted from my fearful thought.

(I think this is a special case of the more general “my system 1 doesn’t believe that it/I can accomplish a given task, and so doesn’t want to think about it.” But with the exception of brute physical skills, if I don’t believe I can do a thing, it’s because I don’t know how.)

I can force myself to stare a scary thing in the face, and come to grips with it, but this is really will-power-y, and hence unreliable. I want to figure out how to make this process perfectly automatic.

This is my hamming problem, and I think it is the key bottleneck on the productivity of most people. If I had a technique that would reliably and efficiently cause me to flinch towards the things I don’t know how to do, and consequently scare me, this would be an God-damn superpower. This is the main difference between my most productive days, when I typically work ten hours at clip, and then rest (because I need it and not because I’m avoiding something), and then go back for more, and most of my days, which are flowing and efficient until I hit something aversive and grind to a halt.

There are a bunch of ideas in this space, but this is MY HAMMING PROBLEM. I’m not looking for some idea that helps a little. I want this problem solved. I want to be at the point where I never have this problem again and I just churn through would-be aversions of this type, effortlessly, everyday. I want the bottleneck on my productivity to be my time and my physical needs, not my micro-hedonic fears.

I think this is possible, and I’m determined to figure out how.

Click here to Reply