Tag: human rationality

Committed Engagement and the Critical Importance of Ambiguity

~ elityre ~ 3 Comments

[epistemic status: the basic idea has been validated by at least my experience, and it seems to resonate with others. But I’m not confident that I have the right framing or am using the right concepts.]

[Part of my Psychological Principles of Productivity drafts.]

In this essay, I want to point out a fact about human psychology, and some interventions  based on that fact.

First, an example. There’s a rule that my mom taught me for cleaning my room, when I was growing up: never pick up an object more than once. Once you have an item in your hand, you must put it where it goes, never put it back down where you found it. The reason for this is that you otherwise tend to get stuck in a loop: where you pick up a thing, are not quite sure where it goes, and so pick up another thing. Finding yourself in the same situation, you pick up the first thing again.

In my adult life, I sometimes find myself in a similar situation when processing email. I’m going through my inbox, and I get to an email that I’m not quite sure how to respond to, and I notice myself flicking back to my inbox without having made a decision about how to reply.

There’s an important truth about human psychology in this phenomenon: ambiguity, that is unclarity about specific next actions, is micro-hedonically aversive, and the human mind tends to flinch away from it.

Productivity

In fact, I think that ambiguity is the primary cause of ugh fields that can curtail my (your?) productivity.

Committed engagement

That’s because resolving ambiguity, clarifying what your options are, and choosing which one to commit to, is hard work. It requires conscious, System-2 style, effort. For most of us, being so called “knowledge workers”, resolving ambiguity is the bulk of our work. The hard part is figuring out what to do. Doing it is often comparatively easy.

Often, when Aversion Factoring, I find that the only reason why I don’t feel like doing something, is the effort of chunking out what exactly the next actions are. After I’ve done that I have no aversion at all.

Accordingly, I now think of processing my various inboxes (and particularly the inbox of reminders that I leave for myself), not as a low-energy, time-limited [as opposed to energy-limited] task, but as a key component of the work that I do.

And when I’m processing inboxes, I step into a mode that I call committed engagement: I make it my intention to plow through and empty the inbox. Given that I’m going to get to and deal with every item, there’s no incentive to look at a thing and put it back. In Committed engagement, the natural thing to do with an item is figure out what needs to be done with it. (Committed engagement is an energized state, with some pressure to get through the task rapidly.)

This is contrast to a sort of “shallow engagement” in which I skim over the inbox, clicking on things that seem quick or interesting, and then marking them as unread again, if they require even a little bit of thought.

Simulation for resolving ambiguity

I have a variety of useful TAPs based on this principle that my mind avoids ambiguity. When I feel averse to a thing in a way that has the flavor of ambiguity (which I do have specific phenomenology for), I visualize the very first smallest steps of the action in my Inner Simulator, which often lowers the activation energy so substantially that it becomes basically easy to take the action.

For instance, Trigger: “I should start writing, but I don’t feel like it” -> Action: “Visualize opening up my laptop” tends to automatically lead to opening up my laptop and begin writing. 

If know that I should strength train, but I don’t feel like it, I’ll simulate concretely standing up, walking to the elevator, and pushing the button. Which in most cases, is sufficient to cause me to get up, walk over, and push the button. And once I’m in the elevator, I’m on my way to the gym.

I think of this as taking advantage of “the smallest atomic action” principle of setting good TAPs. But instead of setting a plan for the future, you’re “setting a plan” for the very next moment. It’s almost humorous how much motivation cascades from merely imagining a simple atomic action.

Similarly, if I’m lost in working on a problem, I might write down the first step, or the main blocker, just to make it clear to me what that is. From there, the next actions are often clear and I can make progress.

Epistemology

This psychological fact is extremely important for productivity, but it is also relevant to epistemology. Your mind is averse to ambiguity. when considering a problem, you have a tendency to deflect away from the parts that are non-concrete: which are often where the important thinking is to be done.

This is at least a part of the reason why “rubber ducking” or talking with a friend is often helpful: stating your problem out loud forces you to clarify the points where you have ambiguity, which you might otherwise skim over.

A shout out

I think my mom probably learned that rule from David Allen (who she met in person), or at least his excellent book, Getting Things Done. He says:

You may find you have a tendency, while processing your in-basket, to pick something up, not know exactly what you want to do about, and then let your eyes wander onto another item farther down the stack and get engaged with it. That item may be more attractive to your psyche because you know right away what to do with it – and you don’t feel like thinking about what’s in your hand. This is dangerous territory. What’s in your hand is likely to land on a “hmppphhh” stack on the side of your desk because you become distracted by something easier, more important, or more interesting below it.

Furthermore, this idea that clarifying your work, and resolving your “stuff” into next actions is the bulk of one’s intellectual labor, is an important theme of the book.

. . .

Keep this in mind: Your mind flinches away from ambiguity. But you can learn to notice, and counter-flinch.

 

Related: Microhedonics, Attention, Visualization

References: Getting Things Done: the Art of Stress Free Productivity

My current model of Anxiety

~ elityre ~ 5 Comments

[epistemic status: untested first draft model

Part of my Psychological Principles of Productivity series]

This is a brief post on my current working model of what “anxiety” is. (More specifically, this is my current model of what’s going on when I experience a state characterized by high energy, distraction, and a kind of “jittery-ness”/ agitation. I think other people may use the handle “anxiety” for other different states.)

I came up with this a few weeks ago, durring that period of anxiety and procrastination. (It was at least partial inspired by my reading a draft of Kaj’s recent post on IFS. I don’t usually have “pain” as an element of my psychological theorizing.)

The model

Basically, the state that I’m calling anxiety is characterized by two responses moving “perpendicular” to each other: increased physiological arousal, mobilizing for action, and a flinch response redirecting attention to decrease pain.

Here’s the causal diagram:

 

IMG_2554.JPG

The parts of the model

It starts with some fear or belief about the state of the world. Specially, this fear is an alief about an outcome that 1) would be bad and 2) is uncertain.

For instance:

  • Maybe I’ve waited too late to start, and I won’t be able to get the paper in by the deadline.
  • Maybe this workshop won’t be good and I’m going to make a fool of myself.
  • Maybe this post doesn’t make as much sense as I thought.

(I’m not sure about this, but I think that the uncertainty is crucial. At least in my experience, at least some of the time, if there’s certainty about the bad outcome, my resources are mobilized to deal with it. This “mobilization and action” has an intensity to it, but it isn’t anxiety.)

This fear is painful, insofar as it represents the possibility of something bad happening to you or your goals.

The fear triggers physiological arousal, or SNS activation. You become “energized”. This is part of your mind getting you ready to act, activating the fight-or-flight response, to deal with the possible bad-thing.

(Note: I originally drew the diagram with the pain causing the arousal. My current guess is that it makes more sense to talk about the fear causing the arousal directly. Pain doesn’t trigger fight-or-flight responses (think about being stabbed, or having a stomach ache). It’s when their’s danger, but not certain harm, that we get ready to move.)

However, because the fear includes pain, there are other parts of the mind that have a flinch response. There’s a sub-verbal reflex away from the painful fear-thought.

In particular, there’s often an urge towards distraction. Distractions like…

  • Flipping to facebook
  • Flipping to LessWrong
  • Flipping to Youtube
  • Flipping to [webcomic of your choice]
  • Flipping over to look at your finances
  • Going to get something to eat
  • Going to the bathroom
  • Walking around “thinking about something”

This is often accompanied by rationalization thought, that is justifying the distraction behavior to yourself.

So we end up with the fear causing both high levels of physiological SNS activation, and distraction behaviors.

Consequences

The distraction-seeking is what gives rise to the “reactivity” (I should write about this sometime) of anxiety, and the heightened SNS gives rise to the jittery “high energy” of anxiety.

Of course, these responses work at cross purposes: the SNS energy is mobilizing for action, (and will be released when action has been taken and the situation is improved) and and the flinch is trying not to think the bad possibility.

I think the heightened physiological arousal might be part of why  anxiety is hard to dialogue with. Doing focusing requires (? Is helped by?) calm and relaxation.

I think this might also explain a phenomenon that I’ve observed in myself: both watching TV and masturbating defuse anxiety. (That is, I can be highly anxious and unproductive, but if if I watch youtube clips for and hour and a half, or masturbate, I’ll feel more settled and able to focus afterwards).

This might be because both of these activities can grab my attention so that I loose track of the originating fear thought, but I don’t think that’s right. I think that these activities just defuse the heightened SNS, which clears space so that I can orient on making progress.

This suggests that any activity that reduces my SNS activation will be similarly effective. That matches my experience (exercise, for instance, is a standard excellent response to anxiety), but I’ll want to play with modulating my physiological arousal a bit and see.

Note for application

In case this isn’t obvious from the post, this model suggests that you want to learn to notice your flinches and (the easier one) your distraction behaviors, so that they can be triggers for self-dialogue. If you’re looking to increase your productivity, this is one of the huge improvements that is on the table for many people. (I’ll maybe say more about this sometime.)

What to do with should/flinches: TDT-stable internal incentives

~ elityre ~ 5 Comments

[epistemic status: current hypothesis, backed by some simple theory, and virtually no empirical evidence yet]

{Part of the thinking going into my Psychology and Phenomenology of Productivity sequence}

The situation

I’ve been having an anxious and unproductive week. There’s a project that I intend to be working on, but I’ve been watching myself procrastinate (fairly rare for me these days), and work inefficiently.

More specifically this situation occurs:

I’m sitting down to start working. Or I am working, and I encounter a point of ambiguity, and my attention flinches away. Or I’m taking a break and am intended to start working again.

At that moment, I feel the pressure of the “should”, the knowing that I’m supposed to/I reflectively want to be making progress, and also feel the inclination to flinch away, to distract myself with something, to flick to LessWrong (it used to be youtube, or SMBC, but I blocked those) or to get something to eat. This comes along with a clench in my belly.

The Opportunity and Obligation

This is a moment of awareness. At that moment, I am conscious of my state, I’m conscious of my desire to make progress on the project. If I do flick to LessWrong, or otherwise distract myself, I will loose that conscious awareness. I’ll still feel bad, still have the clench in my belly, but I won’t be consciously aware of the thing I’m avoiding (at least until the next moment like this one). At that moment, I’m at choice about what to do (or at least more at choice). In the next moment, if the default trajectory is followed, I won’t be.

Realizing this put’s a different flavor on procrastination. Typically, if I’m procrastinating, I have a vague “just one more” justification. It’s ok to watch just one more youtube clip, I can quit after that one. I can stay in bed for another five minutes, and then get up. But if my level of consciousness of my situation fluctuates, that justification is flatly not true.

I have the opportunity right now, to choose something different. I, in actual fact, will not have that opportunity in five minutes.

That me, right then, in that timeslice, has a specific obligation to the world. [I should maybe write a post about how my morality cashes out to different timeslices having different highly-specific obligations to serve the Good.] In that moment, I, the me that is conscious of the should, have the obligation to seize the opportunity of that increased consciousness and use it to put myself on a trajectory such that the next timeslice can effectively pursue a project that will be a tick of the world iterating to a good, safe, future.

The problem

The naive way to seize on that opportunity is to force myself do the task.

There’s a problem with that solution, aside even from the fact that it doesn’t seem like it will work (it’s typically a red flag when one’s plan is “I’ll just use will power”). Even if I could reliably seize on my moment of awareness to force myself to overcome the aversion of my flinch response, doing so would disincentivize me from noticing in the first place.

Doing that would be to install a TAP: whenever I notice myself with a should/flinch, I’ll immediately grit my teeth and preform an effortful and painful mental action. This is conditioning my brain to NOT notice such experiences.

Which is to say, the “just do it” policy is not stable. If I successfully implemented it, I would end up strictly worse off, because I’d still be procrastinating, but I would be much less likely to notice my procrastination.

A guess at a solution

After having noticed this dynamic this week, this is the approach that I’m trying: when I notice the experience of an entangled “should” and the flinch away from it, I orient to hold both of them. More specifically, I move into facilitation mode, where my goal is to make sure that the concerns of both parts are heard and taken into account. Not to force any action, but to mediate between the two conflicting threads.

(Taking advantage of fleeting moments of increased consciousness to hold the concerns of two inchoate and conflicting things at once, is a bit tricky, but I bet I’ll aquire skill with practice.)

If I were to generalize this goal it is something like: when I have a moment of unusual awareness of a conflict, I move to in the direction of increased awareness.

I’ve only been doing this for a few days, so my n is super small, and full of confounds, but this seems to have lead to more time spent dialoguing parts, and days this week have been increasingly focused and productive.

 

Culture vs. Mental Habits

~ elityre ~ Leave a comment

[epistemic status: personal view of the rationality community.]

In this “post”, I’m going to outline two dimensions on which one could assess the rationality community and the success of the rationality project. This is hardly the only possible break-down, but it is one that underlies a lot of my thinking about rationality community building, and what I would do, if I decided rationality community building were a strong priority.

I’m going to call those two dimensions Culture and Mental Habits. As we’ll see these are not cleanly distinct categories, and they tend to bleed into each other. But they have separate enough focuses that one can meaningfully talk about the differences between them.

Culture

By “culture” I mean something like…

  • Which good things are prioritized?
  • Which actions and behaviors are socially rewarded?
  • Which concepts and ideas are in common parlance?

Culture is about groups of people, what those groups share and what they value.

My perception is that on this dimension, the Bay area rationality community has done extraordinarily well.

Truth-seeking is seen as paramount: individuals are socially rewarded for admitting ignorance and changing their minds. Good faith and curiosity about other people’s beliefs is common.

Analytical and quantitative reasoning is highly respected, and increasingly, so is embodied intuition.

People get status for doing good scholarship (e.g. Sarah Constantin), for insightful analysis of complicated situations (e.g. Scott Alexander, for instance), or for otherwise producing good or interesting intellectual content (e.g. Eliezer).

Betting (putting your money where your mouth is) is socially-encouraged. Concepts like “crux” and “rationalist taboo” are well known enough to be frequently invoked in conversation.

Compared to the backdrop of mainline American culture, where admitting that you were wrong means losing face, and trying to figure out what’s true is secondary (if not outright suspicious, since it suggests political non-allegiance), the rationalist bubble’s culture of truth seeking is an impressive accomplishment.

Mental habits

For lack of a better term, I’m going to call this second dimension “mental habits” (or perhaps to borrow Leverage’s term “IPs”).

The thing that I care about in this category is “does a given individual reliably execute some specific cognitive move, when the situation calls for it?” or “does a given individual systematically avoid a given cognitive error?”

Some examples, to gesture at what I mean

  • Never falling prey to the planning fallacy
  • Never falling prey to sunk costs
  • Systematically noticing defensiveness and deflinching or a similar move
  • Systematically noticing and responding to rationalization phenomenology
  • Implementing the “say oops” skill, when new evidence comes to light that overthrows an important position of yours
  • Systematic avoidance of the sorts of errors I outline my Cold War Cognitive Errors investigation (this is the only version that is available at this time).

The element of reliability is crucial. There’s a way that culture is about “counting up” (some people know concept X, and use it sometimes) and mental habits is about “counting down” (each person rarely fails to execute relevant mental process Y).

The reliability of mental habits (in contrast with some mental motion that you know how to do and have done once or twice), is crucial, because it puts one in a relevantly different paradigm.

For one thing, there’s a frame under which rationality is about avoiding failure modes: how to succeed in a given domain depends on the domain, but rationality is about how not to fail, generally. Under that frame, executing the correct mental motion 10% of the time is much less interesting and impressive than executing it everytime (or even 90% of the time).

If the goal is to avoid the sorts of errors in my cold war post, then it is not even remotely sufficient for individuals to be familiar with the patches: they have to reliably notice the moments of intervention and execute the patches, almost every time, in order to avoid the error in the crucial moment.

Furthermore, systematic execution of a mental TAP allows for more complicated cognitive machines. Lots of complex skills depend on all of the pieces of the skills working.

It seems to me, that along this dimension, the rationality community has done dismally.

Eliezer wrote about Mental Habits of this sort in the sequences and in his other writing, but when I consider even very advanced members of my community, I think very few of them systematically notice rationalization, or will reliably avoid sunk costs, or consistently respond to their own defensiveness.

I see very few people around me who explicitly attempt to train 5-second or smaller rationality skills. (Anna and Matt Fallshaw are exceptions who come to mind).

Anna gave a talk at the CFAR alumni reunion this year, in which she presented two low-level cognitive skills of that sort. There were about 40 people in the room watching the lecture, but I would be mildly surprised if even 2 of those people reliably execute the skills described, in the relevant-trigger situation, 6 months from that talk.

But I can imagine a nearby world, where the rationality community was more clearly a community of practice, and most of the the people in that room, would watch that talk and then train the cognitive habit to that level of reliability.

This is not to say that fast cognitive skills of this sort are what we should be focusing on. I can see arguments that culture really is the core thing. But nevertheless, it seems to me that the rationality community is not excelling on the dimension of training it’s members in mental TAPs.

[Added note: Brienne’s Tortoise skills is nearly archetypal of what I mean by “mental habits”.]

Using the facilitator to make sure that each person’s point is held

~ elityre ~ 3 Comments

[Epistemic status: This is a strategy that I know works well from my own experience, but also depends on some prereqs.

I guess this is a draft for my Double Crux Facilitation sequence.]

Followup to: Something simple to try in conversations

Related to: Politics is the Mind Killer, Against Disclaimers

Here’s a simple model that is extremely important to making difficult conversations go well:

Sometimes, when a person is participating in a conversation, or an argument, he or she will be holding onto a “point”, that he/she wants to convey.

For instance…

  • A group is deciding which kind of air conditioner to get, and you understand that one brand is much more efficient than the others, for the same price.
  • You’re listening to a discussion between two intellectuals who you can tell are talking past eachother, and you have the perfect metaphor that will clarify things for both of them.
  • Your startup is deciding how to respond to an embarrassing product failure, one of the cofounders wants to release a statement that you think will be off-putting to many of your customers.

As a rule, when a person is “holding onto” a point that they want to make, they are unable to listen well.

The point that a person wants to make relates to something that’s important to them. If it seems that their conversational-partners are not going to understand or incorporate that point, that important value is likely going to be lost. Reasonably, this entails a kind of anxiety.

So, to the extent that it seems to you that your point won’t be heard or incorporated, you’ll agitatedly push for airtime, at the expense of good listening. Which, unfortunately, results in a coordination problem of each person pushing to get their point heard and no one listening. Which, of course, makes it more likely that any given point won’t be heard, triggering a positive feedback loop.

In general, this means that conversations are harder to the degree that…

  1. The topic matters to the participants.
  2. The participant’s visceral expectation is that they won’t be heard.

(Which is a large part of the reason why difficult conversations get harder as the number of participants increases. More people means more points competing to be heard, which exacerbates the death spiral.)

Digression

I think this goes a long way towards explicating why politics is a mind killer. Political discourse is a domain which…

  1. Matters personally to many participants, and
  2. Includes a vast number of “conversational participants”,
  3. Who might take unilateral action, on the basis of whatever arguments they hear, good or bad.

Given that setup, it is quite reasonable to treat arguments as soldiers. When one sees someone supporting, or even appearing to support a policy or ideology that you consider abhorrent or dangerous, there is a natural and reasonable anxiety that the value you’re protecting will be lost. And there is a natural (if usually poorly executed) desire to correct the misconception in the common knowledge before it gets away from you. Or failing that, to tear down the offending argument / discredit the person making it.

(To see an example of the thing that one is viscerally fearing, see the history of Eric Drexler’s promotion of nanotechnology. Drexler made arguments about Nanotech, which he hoped would direct resources in such a way that the future could be made much better. His opponents attacked strawmen of those arguments. The conversation “got away” from Drexler, and the whole audience discounted the ideas he supported, thus preventing any progress towards the potential future that Drexler was hoping to help bring into being.

I think the visceral fear of something like this happening to you is what motivates “treating arguments as soldiers“)

End digression

Given this, one of the main thing that needs to happen to make a conversation go well, is for each participant to (epistemically!) aleive that their point will be gotten to and heard. Otherwise, they can’t be expected to put it aside (even for a moment) in order to listen carefully to their interlocutor (because doing so would increase the risk of their point in fact not being heard).

When I’m mediating conversations, one strategy that I employ to facilitate this is to use my role as the facilitator to “hold” the points of both sides. That is (sometimes before the participants even start talking to each-other), I’ll first have each one (one at a time) convey their point to me. And I don’t go on until I can pass the ITT of that person’s point, to their (and my) satisfaction.

Usually, when I’m able to pass the ITT, there’s a sense of relief from that participant. They now know that I understand their point, so whatever happens in the conversation, it won’t get lost or neglected. Now, they can relax and focus on understanding what the other person has to say.

Of course, with sufficient skill, one of the participants can put aside their point (before it’s been heard by anyone) in order to listen. But that is often asking too much of your interlocutors, because doing the “putting aside” motion, even for a moment is hard, especially when what’s at stake is important. (I can’t always do it.)

Outsourcing the this step to the facilitator, is much easier, because the facilitator has less that is viscerally at stake for them (and has more metacognition to track the meta-level of the conversation).

I’m curious if this is new to folks or not. Give me feedback.

 

RAND needed the “say oops” skill

~ elityre ~ 1 Comment

[Epistemic status: a middling argument]

A few months ago, I wrote about how RAND, and the “Defense Intellectuals” of the cold war represent another precious datapoint of “very smart people, trying to prevent the destruction of the world, in a civilization that they acknowledge to be inadequate to dealing sanely with x-risk.”

Since then I spent some time doing additional research into what cognitive errors and mistakes  those consultants, military officials, and politicians made that endangered the world. The idea being that if we could diagnose which specific irrationalities they were subject to, that this would suggest errors that might also be relevant to contemporary x-risk mitigators, and might point out some specific areas where development of rationality training is needed.

However, this proved somewhat less fruitful than I was hoping, and I’ve put it aside for the time being. I might come back to it in the coming months.

It does seem worth sharing at least one relevant anecdote, from Daniel Ellsberg’s excellent book, the Doomsday Machine, and analysis, given that I’ve already written it up.

The missile gap

In the late nineteen-fifties it was widely understood that there was a “missile gap”: that the soviets had many more ICBM (“intercontinental ballistic missiles” armed with nuclear warheads) than the US.

Estimates varied widely on how many missiles the soviets had. The Army and the Navy gave estimates of about 40 missiles, which was about at parity with the the US’s strategic nuclear force. The Air Force and the Strategic Air Command, in contrast, gave estimates of as many as 1000 soviet missiles, 20 times more than the US’s count.

(The Air Force and SAC were incentivized to inflate their estimates of the Russian nuclear arsenal, because a large missile gap strongly necessitated the creation of more nuclear weapons, which would be under SAC control and entail increases in the Air Force budget. Similarly, the Army and Navy were incentivized to lowball their estimates, because a comparatively weaker soviet nuclear force made conventional military forces more relevant and implied allocating budget-resources to the Army and Navy.)

So there was some dispute about the size of the missile gap, including an unlikely possibility of nuclear parity with the Soviet Union. Nevertheless, the Soviet’s nuclear superiority was the basis for all planning and diplomacy at the time.

Kennedy campaigned on the basis of correcting the missile gap. Perhaps more critically, all of RAND’s planning and analysis was concerned with the possibility of the Russians launching a nearly-or-actually debilitating first or second strike.

The revelation

In 1961 it came to light, on the basis of new satellite photos, that all of these estimates were dead wrong. It turned out the the Soviets had only 4 nuclear ICBMs, one tenth as many as the US controlled.

The importance of this development should be emphasized. It meant that several of the fundamental assumptions of US nuclear planners were in error.

First of all, it meant that the Soviets were not bent on world domination (as had been assumed). Ellsberg says…

Since it seemed clear that the Soviets could have produced and deployed many, many more missiles in the three years since their first ICBM test, it put in question—it virtually demolished—the fundamental premise that the Soviets were pursuing a program of world conquest like Hitler’s.

…

That pursuit of world domination would have given them an enormous incentive to acquire at the earliest possible moment the capability to disarm their chief obstacle to this aim, the United States and its SAC. [That] assumption of Soviet aims was shared, as far as I knew, by all my RAND colleagues and with everyone I’d encountered in the Pentagon:

The Assistant Chief of Staff, Intelligence, USAF, believes that Soviet determination to achieve world domination has fostered recognition of the fact that the ultimate elimination of the US, as the chief obstacle to the achievement of their objective, cannot be accomplished without a clear preponderance of military capability.

If that was their intention, they really would have had to seek this capability before 1963. The 1959–62 period was their only opportunity to have such a disarming capability with missiles, either for blackmail purposes or an actual attack. After that, we were programmed to have increasing numbers of Atlas and Minuteman missiles in hard silos and Polaris sub-launched missiles. Even moderate confidence of disarming us so thoroughly as to escape catastrophic damage from our response would elude them indefinitely.

Four missiles in 1960–61 was strategically equivalent to zero, in terms of such an aim.

This revelation about soviet goals was not only of obvious strategic importance, it also took the wind out of the ideological motivation for this sort of nuclear planning. As Ellsberg relays early in his book, many, if not most, RAND employees were explicitly attempting to defend US and the world from what was presumed to be an aggressive communist state, bent on conquest. This just wasn’t true.

But it had even more practical consequences: this revelation meant that the Russians had no first strike (or for that matter, second strike) capability. They could launch their ICBMs at American cities or military bases, but such an attack had no chance of debilitating US second strike capacity. It would unquestionably trigger a nuclear counterattack from the US who, with their 40 missiles, would be able to utterly annihilate the Soviet Union. The only effect of a Russian nuclear attack would be to doom their own country.

[Eli’s research note: What about all the Russian planes and bombs? ICBMs aren’t the the only way of attacking the US, right?]

This means that the primary consideration in US nuclear war planning at RAND and elsewhere, was fallacious. The Soviet’s could not meaningfully destroy the US.

…the estimate contradicted and essentially invalidated the key RAND studies on SAC vulnerability since 1956. Those studies had explicitly assumed a range of uncertainty about the size of the Soviet ICBM force that might play a crucial role in combination with bomber attacks. Ever since the term “missile gap” had come into widespread use after 1957, Albert Wohlstetter had deprecated that description of his key findings. He emphasized that those were premised on the possibility of clever Soviet bomber and sub-launched attacks in combination with missiles or, earlier, even without them. He preferred the term “deterrent gap.” But there was no deterrent gap either. Never had been, never would be.

To recognize that was to face the conclusion that RAND had, in all good faith, been working obsessively and with a sense of frantic urgency on a wrong set of problems, an irrelevant pursuit in respect to national security.

This realization invalidated virtually all of RAND’s work to date. Virtually every, analysis, study, and strategy, had been useless, at best.

The reaction to the revelation

How did RAND employees respond to this reveal, that their work had been completely off base?

That is not a recognition that most humans in an institution are quick to accept. It was to take months, if not years, for RAND to accept it, if it ever did in those terms. To some degree, it’s my impression that it never recovered its former prestige or sense of mission, though both its building and its budget eventually became much larger. For some time most of my former colleagues continued their focus on the vulnerability of SAC, much the same as before, while questioning the reliability of the new estimate and its relevance to the years ahead. [Emphasis mine]

…

For years the specter of a “missile gap” had been haunting my colleagues at RAND and in the Defense Department. The revelation that this had been illusory cast a new perspective on everything. It might have occasioned a complete reassessment of our own plans for a massive buildup of strategic weapons, thus averting an otherwise inevitable and disastrous arms race. It did not; no one known to me considered that for a moment. [Emphasis mine]

According to Ellsberg, many at RAND were unable to adapt to the new reality and continued (fruitlessly) to continue with what they were doing, as if by inertia, when the thing that they needed to do (to use Eliezer’s turn of phrase) is “halt, melt, and catch fire.”

This suggests that one failure of this ecosystem, that was working in the domain of existential risk, was a failure to “say oops“: to notice a mistaken belief, concretely acknowledge that is was mistaken, and to reconstruct one’s plans and world views.

Relevance to people working on AI safety

This seems to be at least some evidence (though, only weak evidence, I think), that we should be cautious of this particular cognitive failure ourselves.

It may be worth rehearsing the motion in advance: how will you respond, when you discover that a foundational crux of your planning is actually mirage, and the world is actually different than it seems?

What if you discovered that your overall approach to making the world better was badly mistaken?

What if you received a strong argument against the orthogonality thesis?

What about a strong argument for negative utilitarianism?

I think that many of the people around me have effectively absorbed the impact of a major update at least once in their life, on a variety of issues (religion, x-risk, average vs. total utilitarianism, etc), so I’m not that worried about us. But it seems worth pointing out the importance of this error mode.

A note: Ellsberg relays later in the book that, durring the Cuban missile crisis, he perceived Kennedy as offering baffling terms to the soviets: terms that didn’t make sense in light of the actual strategic situation, but might have been sensible under the premiss of a soviet missile gap. Ellsberg wondered, at the time, if Kennedy had also failed to propagate the update regarding the actual strategic situation.

I believed it very unlikely that the Soviets would risk hitting our missiles in Turkey even if we attacked theirs in Cuba. We couldn’t understand why Kennedy thought otherwise. Why did he seem sure that the Soviets would respond to an attack on their missiles in Cuba by armed moves against Turkey or Berlin? We wondered if—after his campaigning in 1960 against a supposed “missile gap”—Kennedy had never really absorbed what the strategic balance actually was, or its implications.

I mention this because additional research suggests that this is implausible: that Kennedy and his staff were aware of the true strategic situation, and that their planning was based on that premise.

Circling vs. Unrolling

~ elityre ~ Leave a comment

[Musing]

In reference to Critch’s post here.

I’m intrigued by the explicit unrolling in contrast to circling. I wonder how much circling an instance of developing overpowered tools on weird partly-orthagonal dimensions (like embodiment) because you haven’t yet discovered the basic simple structure of the domain.

Like, a person might have a bunch of cobbled together hacks and heuristics (including things about narrative, and chunking next actions, and discipline) for maintaining their productivity. But a crisp understanding of the relevant psychology makes “maintaining productivity” a simple and mostly effortless thing to do.

Or a person who spends years doing complicated math without paper. They will discover all kinds of tricks for doing mental computation, and they might get really good at these tricks, and building that skill might even have benefits in other domains. But at the end of the day, all of that training is blown out of the water as soon as they have paper. Paper makes the thing they were training hard to do easy.

To what extent is Circling working hard to train capacities that are being used as workarounds for limited working memory and insufficient theoretical understanding the structure of human interaction?

(This is a real question. My guess is, “some, but less than 30%”.)

A lot of my strategies for dealing with situations of this sort are circling-y, and feel like a lot of that is superfluous. If I had a better theoretical understanding, I could do the thing with much more efficiency.

For instance, I exert a lot of effort to be attuned to the other person in general and to be picking up subtle signs from them, and tracking where they’re at. If had a more correct theoretical understanding, a better ontology, I would only need to be tracking the few things that it turns out are actually relevant.

Since humans don’t know what those factors are, now, people are skilled at this sort of interaction insofar as they can track everything that’s happening with the other person, and as a result, also capture the few things that are relevant to the underlying structure.

I suspect that others disagree strongly with me here.

Goal-factoring as a tool for noticing narrative-reality disconnect

~ elityre ~ Leave a comment

[The idea of this post, as well as the opening example, were relayed to me by Ben Hoffman, who mentioned it as a thing that Michael Vassar understands well. This was written with Ben’s blessing.]

Suppose you give someone an option of one of three fruits: a radish, a carrot, and and apple. The person chooses the carrot. When you ask them why, they reply “because it’s sweet.”

Clearly, there’s something funny going on here. While the carrot is sweeter than the radish, the apple is sweeter than the carrot. So sweetness must not be the only criterion your fruit-picker is using to make his decision. He/she might be choosing partially on that basis, but there must also be some other, unmentioned factor, that is guiding his/her choice.

Now imagine someone is describing the project that they’re working on (project X). They explain their reasoning for undertaking this project, the good outcomes that will result from it: reasons a, b, and c.

When someone is presenting their reasoning like this, it can be useful to take a, be and c as premises, and try and project what seems to you like the best course of action that optimizes for those goals. That is, do a quick goal-factoring, to see if you can discover a y, that seems to fulfill goals a, b, and c, better than X does.

If you can come up with such a Y, this is suggestive of some unmentioned factor in your interlocutor’s reasoning, just as there was in the choice of your fruit-picker.

Of course this could be innocuous. Maybe Y has some drawback you’re unaware of, and so actually X is the better plan. Maybe the person you’re speaking with just hadn’t thought of Y.

But but it also might be he/she’s lying outright about why he/she’s doing X. Or maybe he/she has some motive that he/she’s not even admitting to him/herself.

Whatever the case, the procedure of taking someone else’s stated reasons as axioms and then trying to build out the best plan that satisfies them is a useful procedure for drawing out dynamics that are driving situations under the surface.

I’ve long used this technique effectively on myself, but I sugest that it might be an important lens for viewing the actions of institutions and other people. It’s often useful to tease out exactly how their declared stories about themselves deviate from their revealed agency, and this is one way of doing that.

 

 

Approaches to this thing called “Rationality” (or alternatively, a history of our lineage)

~ elityre ~ Leave a comment

[Posted to the CFAR mailing list]

[Somewhat experimental: Looking for thumbs up and thumbs down on this kind of writing. I’m trying to clarify some of the fuzziness around why we are calling the-thing-some-of-us-are-calling-rationality “rationality.”]

So what is this rationality thing anyway?

Simply stated, some behavior works better than other behavior for achieving a given goal. In fact, for formal and well defined environments, “games”, this is provably true. In the early to mid 20th century, academic mathematicians developed game theory and decision theory, mathematical formalizations of idealized decision algorithms that give provably optimal outcomes (in expectation).

One school of rationality (let’s call it “formal rationality”) is largely about learning and relying on these decision rules. For a rationalist of this type, progress in the field means doing more math, and discovering more theorems or decisions rules. Since most non-trivial decision problems involve dealing with uncertainty, and uncertainty in the real world is quantified using statistics, statistics is central to the practice of formal rationality.

MIRI does the sort of work that a formal rationalist would consider to be progress on rationality: trying to develop solutions to decision theory problems. (This is NOT to say that most, or even any of the actual people who work are MIRI are themselves of the “formal rationality” school as opposed to those to follow. In fact I have reason to think that NONE of them would identify as such.) The other large frontiers of “formal rationality” are mostly in economics. The economy can be thought of as a single gigantic game theoretic game.

For the formal rationalist, rationality is almost entirely solved. We have game theory. We have probability theory. We have decision theory. There may be edge-case scenarios that need to be solved (pascal’s mugging, for instance), but for the most part, the “art” has already been invented. Declaring oneself a rationalist in the formal sense is a statement of philosophy: it means you trust the approximations of the formal decision rules over intuition, common sense, tradition, or well, anything.  One doesn’t need to qualify with the word “aspiring.”

(There’s a framework nearby to formal rationality which is largely captured by the term “evidence-based.” This is the position that one should base one’s actions and beliefs on evidence, over intuition or superstition. We can call this traditional rationality.  Traditional rationality includes science, and evidence-seeking in general.)

If you have formalized decision rules that describe the behaviour of goal directed agents, you now have the affordance to check what humans are actually doing. Enter Kahneman and Tversky. Over the course of the 1970’s to 1990’s ,they do many experiments and determine that 1) most people are not optimal goal-directed agents, (i.e. they are “irrational”. Little surprise to anyone, I think), 2) that those with advanced knowledge of “formal rationality” (e.g. statistics, economics, probability theory, game theory, decision theory) also fail to be optimal goal-directed agents (WE’re irrational too), and 3) that humans tend to deviate from ideal behaviour in systematic, predictable ways.

Thus develops the Heuristics and Biases project in psychology, with gives to rise another approach to the project of rationality. If humans are intrinsically and systematically biased, and simply telling a person about the bais doesn’t fix it (as is often the case), then the greater share of rationality training involves coming up with methods to counteract native cognitive bias. We can call this approach to rationality “the debiasing approach.” It inherits many of the formalizations from formal rationality (which do reflect ideal behavior), but the emphasis is on dealing with the actual human mind and correcting it’s faults. The project of rationality involves math, but now it is mostly in the domain of psychology.

This is in large part, the approach to rationality that Eliezer took in the sequences (though the sequences are a philosophical treatise, and his aims went beyond debiasing), and it fairly well characterizes LessWrong.

In 2012, CFAR is founded, and initially takes the debiasing approach. But the organization pretty quickly pivots away from that sort of model (you’ll notice that there are no modules in the current workshop of the form “this is the X fallacy/bias and here is the technique that eliminates or mitigates it.”) Developing debiasing protocols proves to be difficult, but there’s a nearby thing which is very useful and and much more tractable. CFAR borrows the System 1 / System 2 framework from heuristics and biases and develops methods to get those facets of the mind to communicate with one another.

For instance, sometimes a person intellectually endorses an action but doesn’t feel emotionally motivated about it. Propagating urges (or aversion factoring) is a technique that facilitates the dialogue between those parts of the mind, such that one (or both) of them updates and they are both on the same page. Similarly, sometimes a part of the mind has information about how likely a project is to succeed, but that data needs to be queried to be useful. The Inner Simulator / Murphyjitsu is a technique that lets the conscious, verbal system query the part of the mind that automatically makes predictions of that sort.

This approach isn’t about mitigating specific biases, but rather propagating information that one already has in one part of his/her mind to other parts of the the cognitive system. We can call this approach the “parts-propagation approach.” It’s about unifying the mind (minding our way style, mostly, but not exclusively) such that all parts of the mind are on the same page and pulling in the same direction, so that we can be more “agenty ducks” (i.e., better approximations of the simplified goal-directed agents of formal rationality, with stable, ordered goals) that can get shit done in the world.

These are three rather different approaches to rationality, and each one entails very different ideas of what the “art of rationality” should look like and what directions research should take. I have thoughts about which of these approaches are most tractable and important, but my goal here is only to clarify some of the confusion about what is meant by “rationality” and why.

Thoughts? Are these categories good ones? Do they carve reality at the joints?