Tag: intellectual traditions and communities

Moldbug’s insight

~ elityre ~ 1 Comment

I’ve been reading some of Curtis Yarvin’s work lately.

For the most part, he seems like a blowhard, and an incorrect blowhard at that. His general rhetorical approach seems to be to make bold assertions, dressed up in flowery and bombastic language, and then to flatter his reader for being in on the secret. When he’s on podcast interviews, mostly the hosts will agree with his premises, but occasionally he’ll make a claim that they reject and push back against. Then Yarvin is forced to defend his bold claims instead of just insinuating them, and often his actual argumentation comes off as pretty weak.

I get the feeling sometimes when reading his work of reading a high school essay, of the author reaching for arguments to defend a bottom line, decided for other reasons, rather than reporting the arguments and evidence that lead the author to believe the conclusion.1

He admits directly that he’s writing for fun, and occasionally talks about writing to troll people. I get the impression that his views were arrived at in part by a sincere intellectual investigation of history and political philosophy, and in part because they were fun (ie shocking) to advocate for in 2008. But now they’re a key part of Yarvin’s brand and he’s kind of stuck with them. As in academic philosophy, his incentives are towards doubling down on his distinctive ideas, regardless of their truth.)

His rhetorical style reminds me of that of Eliezer Yudkowsky and Nassim Taleb. All three of them have a deep knowledge of their subject matter and each writes with an arrogance / confidence in the correctness of his view and an insinuation that the reader, like him, understands some important truths not grasped by the masses of humanity. This style makes these authors fun to read, for some people, and insufferably annoying for other people.

My read, so far, is that if you don’t already buy into his basically aesthetic premises, his disgust for modernity and for progressivism in particular, he doesn’t have much in the way of good arguments for persuading you of his views. Perhaps the main thing that he does is open people’s eyes, allowing them to see through a hitherto completely unknown perspective that pierces through the civic propaganda of our time. Having seen through that perspective, perhaps some parts of the the world makes more sense. But that’s not because Moldbug made a strong case for his claims, so much as his rhetoric ensnared you in his wake, and pulled you along for a bit. (I’m very interested in Moldbug fans who disagree—especially those who’s mind was changed by some of his posts.)

That said, he does have a few important and novel-to-me analytical points.

Today, I think I grasped an important core of Yarvin’s political philosophy which I hadn’t previously understood, and which, not having understood, made many of his claims seem bizarre in their not-even-wrongness.

All of the following is a compression of my understanding of his view, and is not to be taken as an endorsement of that view.

Claim 1: Sovereignty is Conserved

This is almost a catchphrase for Yarvin. He uses it all over the place.

There is always some force or entity outside of and above the law. Every law is enforced by some process (otherwise it’s hardly a law). And the process that enforces the law must, necessarily, have the power to exempt itself from that law. If not, it wasn’t actually the system ultimately doing the enforcing. Sovereignty is “above-the-law-ness”, and it’s always conserved.2

As an intuition pump: there exists someone in the US government, who, if they decided to, could “disappear” you a (more or less) ordinary US citizen. Possibly the president could detain or assassinate a US citizen for no legible cause, and face no consequences. Possibly some specific people in the intelligences services, as well. If there’s no one person who could do it, there’s surely a consortium of people that, working in concert, could. (That sovereignty is conserved doesn’t mean that it’s always concentrated). In the limit, the whole of the US military must be above the law, because if it decided to, in a coordinated way, it could trivially overturn any law, or the whole governmental system for that matter. [More on that possibility later.]

Even if no specific individual is above the law, the government as a whole sure as hell is. “The government” can, fundamentally, do whatever it “wants”.

This is explicitly counter to an ideal enlightenment philosophy—that of equality before the law. That no person, no mater how powerful, is exempt from the same basic legal standards.

Moldbug asserts that any claim to equality above the law is horseshit. Sovereignty is conserved. Power is real, and it bottoms out somewhere, and wherever it bottoms out is always going to be above the law.

This isn’t a law of physics, but it is a law of nature—at least as inescapable as the logic of supply and demand, or natural selection. 3

Because of his rhetoric and politics, it’s easy to read Moldbug as not caring at all about the inequities of power. This is somewhat of a misunderstanding. It’s a non-question for Yarvin whether it’s good or desirable that sovereignty is conserved. It’s just a fact of life that power is going to ground out somewhere. Whether we consider that inhumane or prefer that it was otherwise is of no more relevance that if we wished perpetual motion was possible. It’s not possible, and it’s not possible for a pretty fundamental reason.4

But as a society, we’re are intent on deluding ourselves about the nature of power. That might cause problems, in roughly the way it might if we insisted on deluding ourselves about the efficacy of perpetual motion machines.

Claim 2: The profit motive + competition is a stronger guarantee than ideology

So there’s always some entity outside the law. But, one might think, given that sad reality, that its better to divide up that power as much as possible so that as few people as possible, and ideally no one, can unilaterally disappear people. Checks and balances, and limited powers, and so on, to prevent any individual or group in government, and the government as a whole from being too powerful. Perhaps we can’t abolish sovereignty, but dividing it up as much as possible and spreading it around seems like the the most humane way to deal with the unfortunate situation, right?

Yarvin is in favor of monarchy, so he says “no”. Why not?

Because, in practice, the less concentrated power is, the more it is effectively controlled by ideology rather than rational optimization for anyone’s interests.

This is the basic problem of voter incentives: The odds of any individual person’s vote shifting policy, and impacting that person’s life directly are so minuscule as to be irrelevant. The main impact that your voting behavior has on your life is through signaling: signaling to your peers and to yourself what kind of person you are. If your vote materially impacted your life through policy, you would be incentivized to carefully weigh the tradeoffs in every decision (or defer to trusted expert advisors). But if your vote is mostly about showing how compassionate you are, how committed you are to our shared values, carefully weighing tradeoffs doesn’t help you. Saying the most applause lights the fastest is what’s good for signaling.

As Bryan Caplan says “Markets do the good things that sound bad, and governments do bad things that sound good.”

The more power is divided up into tiny pieces the more it is steered by ideology instead of by self-interest. And rational self interest is much less dangerous than ideology.

As discussed, the US military could overthrow the US government and the US legal system, if it wanted to. Why doesn’t it do that? Because there’s a distributed common knowledge belief in “democracy“. Lots of people in the military sincerely believe in the democratic ideal, and even if they don’t, they believe that they believe they do, and everyone knows that everyone else would immediately oppose any attempts at an “undemocratic” military coup.

Which is to say that the thing standing between the US as it currently exists and a military dictatorship is an ideological commitment to “democracy”. This seems to have worked pretty well so far, but those scare quotes are pretty scary. If a sufficiently large faction of the military came to buy into an ideology that claimed to carry the torch of the true spirit of democracy (or Christianity, or Social Justice, or Communism, or enviornmentalism, or whatever moral ideal compels), that ideology would take over the US.

And similarly, to the extent that the US government is possessed by the spirit of Wokism, your country might suddenly become violently woke.

This isn’t a hypothetical. We’ve seen countries get possessed by Communist ideology and become violently Communist.

In contrast, consider if instead there was a single king/CEO, who has complete and total power over his domain, who controlled the military power. As long as he’s sane and competent (which has been a problem with historical monarchs but which Yarvin thinks is more-or-less solved as well as we can reasonably expect by the structure of a joint-stock corporation), this monarch would be acting from incentives that are much closer to rational self-interest, because he (and the shareholders of the joint-stock country) benefit(s) directly from the upside of actual actual policy outcomes, not just the social signaling benefits of his policies. He wants his realm to be safe and well-governed because that will increase the value of the real estate he owns, and he will make more money that way.

Especially so if he governs only one of hundreds of sovereign realms in a patchwork. In that case there’s competitive pressure to get policy right, and maintain rule of law. If he does a bad job of ruling, residents will leave to live somewhere else, taking their tax revenue with them.

This is not perfect. Any given king might be bad, just as any given CEO can be bad. There’s no guarantee that a king won’t be possessed by and ideology (it’s certainly happened before! Ferdinand II of the Holy Roman Empire and Alexander I of Russia, come to mind). But it’s better than the alternatives. Especially if the shareholders can remove a bad king from power and if there’s competition between sovereign realms, both of which introduce selection pressure for sane, self-interested kings.

It’s true that the sovereign could, by right, have any person in his realm who ticked him off quietly assassinated. But, realizing that sovereignty is conserved, that becomes less of a problem of monarchy in particular, and more of an inescapable problem of power in general, one which we obscure but don’t eliminate with limited governments of ostensive checks and balances.

Plus, assassinating people, even if you have the legal right to do it, is generally going to be bad for business—an erratic CEO doesn’t inspire the confidence that causes people to want to live in his realm. Enough shenanigans like that, and his sovereign corporation will start losing customers, and his shareholders will sell it’s stock and/or have him removed at CEO. And if the CEO is actually sovereign, that removes the strongest incentive on historical monarchs for having people assassinated: as a means of securing his power.5

But most importantly, a monarch-CEO is much much less likely than a democracy to get riled up and implement Communism. Communism is transparently bad for business, but sounds good (meaning it is a good way to signal your compassion or tribal loyalty). The incentives of CEOs leave them less vulnerable to takeover by parasitic ideologies compared to masses of people in democracies. And ideological revolutions and generic bad-but-sounds-good policy is the serious threat-model. The all-powerful CEO who has the legal and pragmatic power of life and death over you is just much less dangerous than a state controlled by competing ideologies, which might decide that doing massive harm (from burning down your cities in the name of black lives, to rounding up all the jews, sending your scientists to work camps) is morally obligatory, in a fit of runaway virtue-signaling.

And indeed, when there’s some political power in the hands of the people, a good strategy for an ambitious person seizing power is to craft or adapt an ideology that enflames the people’s passions, and empowers you personally. That’s what Hitler and Lenin did. When sovereignty is in the hands of shareholders and their CEO-delegate., ideologies are less adaptive for gaining power, and so less pervasive in the first place. But this is a separate thread of Modbugian philosophy: that democracy causes ideology, that’s less central to the point that CEO-kings operating under the constraints of the profit motive and market competition are less vulnerable to ideologies than democracies.

Given that we can’t escape power, the profit motive of a king is a much stronger guarantee of good outcomes than ideological commitment, because ideologies are crazy, or at least can turn crazy fast.

Once you have that attitude, the fact that sovereignty in our present era seems to bottom out in basically ideological institutions seems…rather concerning. Every time you read “democratically controlled” you might mentally replace it with “more or less controlled by at least one more-or-less insane egregor.”

When I think in these terms, Yarvin’s political philosophy clicks into place for me as a coherent take on the world.

I’m not sure if I buy it, overall.

I agree that we don’t have literal and complete equality before the law: there are elites who get special treatment, and there may be individuals in the system that can literally get away with murder (though my guess is that’s only true in pretty limited circumstances?). But the US social and legal system really is demonstrably more egalitarian, closer to the ideal of equality before the law, than the European aristocratic systems that proceeded it. And that seems like something to be justly proud.

I think he’s underselling separation of powers. It’s true that the government can do whatever it wants, but we’ve set it up so that the government has difficulty mustering up unified and coherent wants to act on. Government is, in practice, limited by earth’s low coordination capacity. Which gives us a kind of safety from tyranny.

If someone in the intelligence community wanted to “disappear” me, they would have to keep it secret, because they would have political opponents, and just principled detractors, who would, if they could, expose the criminal and have them arrested. Nixon was removed from office for violating the law. It might not be perfect equality before the law, but it’s a pretty impressive example of something approaching that.

Further, I’m less pessimistic than my read of Yarvin about constructing systems in which NO individual is above the law in the sense of being able to unilaterally violate it. eg systems where everyone enforces the law on everyone else. (Systems like these are vulnerable to 51% attacks, and the number of actual people required to implement a 51% attack falls as political and/or social power is consolidated. But that’s true of literally every system of law, and the question is how we can do best.)

It does seem possible that a CEO-monarch who can be removed by a vote of the stockholders is more likely to act from straightforward material rational self-interest than voters do, currently. (Actual historical monarchies have a number of critical-level problems, from crazy kings to violent succession disputes as the norm). It seems like it is likely to have other problems—namely a principle agent problem between the shareholders and their delegate.6 I’m curious to see a government that runs on that system, and see how it behaves. Maybe it would result in good policy.

However, I think there are other schemes, mostly untried, that do a better job of incentivizing good judgement from voters, while also getting the historically-validated stability benefits of democratic governance. I’m thinking of systems like futarchy (or just prominent, public, prediction markets) and quadratic voting.

The main feature that’s doing the work in Yarvin’s conception, is the multitude of micronations competing for residents. As long as you have sufficiently low transaction costs involved in moving from one country to another, and at least some countries have politically unified enough governance that they can and do adopt the explicit policy goal of optimizing tax revenue (or, for that matter, any of a number of possible social welfare functions, or baskets of indicators), you get all the benefits of the Moldbugian system. The bit about CEO-kings isn’t actually critical. Maybe that’s the best way to optimize policy for tax revenue, or maybe not. Possibly that the king has authority to kill any citizen for any reason is net-beneficial for security and stability, such that many people prefer living in a nation where the chief executive has that level of legal authority, and overall tax revenue is higher. But then again, maybe not. (The optics are pretty unnerving, at least.)

It sounds to me that the problem is not that we don’t have kings, in particular, but just that there’s so little room for governance experimentation, in general, and so new radical ideas don’t get tried.

  1. For instance, I’m unimpressed with Yarvin’s claim that his political schema would lead to world peace. He spends a few sentences asserting that his realm-CEOs, being rational, would have no issues solving collective action problems, and would have no need for a higher governmental structure above them to enforce collective action, and then moves on. 🙄 ↩︎
  2. See, for instance, here.

    > The key is that word should. When you say your government “should do X,” or “should not do Y,” you are speaking in the hieratic language of democracy. You are postulating some ethereal and benign higher sovereign, which can enforce promises made by the mere government to whose whims you would otherwise be subject. In reality, while your government can certainly promise to do X or not to do Y, there is no power that can hold it to this promise. Or if there is, it is that power which is your real government.
    ↩︎
  3. We might try to conceive of clever schemes under which this is not so: legal systems based on blockchain smart contracts where there’s no enforcement mechanism outside of the computerized legal corpus, itself. Maybe in some scenario like that, we would have effectively grounded out the root of power into the law itself, and escaped the basic dynamic that someone is always above the law (in much the same way that reconstructing life to use encrypted genomes would potentially allow us to escape the so far inexorable pull of natural selection). ↩︎
  4. > It is immediately clear that the neocameralist should, the tight rope, is far inferior to the ethereal should, the magic leash of God. (Typically these days arriving in the form of vox populi, vox Dei. Or, as a cynic might put it: vox populi, vox praeceptoris.)
    > Given the choice between financial responsibility and moral responsibility, I will take the latter every time. If it were possible to write a set of rules on paper and require one’s children and one’s children’s children to comply with this bible, all sorts of eternal principles for good government and healthy living could be set out.
    > But we cannot construct a political structure that will enforce moral responsibility. We can construct a political structure that will enforce financial responsibility. Thus neocameralism. We might say that financial responsibility is the raw material of moral responsibility. The two are not by any means identical, but they are surprisingly similar, and the gap seems bridgeable.

    From Profit Strategies for Our New Corporate Overlords, here. ↩︎
  5. Crucially the board of directors of a realm, the people who do have the power to remove the CEO-king, should not live in that realm, for precisely the reason that this represents an incentive for the king to use his complete power over you, as your sovereign, his ability to have you and your family killed or tortured, to get you to vote as he demands in board meetings. ↩︎
  6. If the CEO-king has absolute power over his realm that seems like it gives him a lot of leeway to control the information flows about how the realm is doing back to the shareholders that might hold him accountable to profit. ↩︎

Culture vs. Mental Habits

~ elityre ~ Leave a comment

[epistemic status: personal view of the rationality community.]

In this “post”, I’m going to outline two dimensions on which one could assess the rationality community and the success of the rationality project. This is hardly the only possible break-down, but it is one that underlies a lot of my thinking about rationality community building, and what I would do, if I decided rationality community building were a strong priority.

I’m going to call those two dimensions Culture and Mental Habits. As we’ll see these are not cleanly distinct categories, and they tend to bleed into each other. But they have separate enough focuses that one can meaningfully talk about the differences between them.

Culture

By “culture” I mean something like…

  • Which good things are prioritized?
  • Which actions and behaviors are socially rewarded?
  • Which concepts and ideas are in common parlance?

Culture is about groups of people, what those groups share and what they value.

My perception is that on this dimension, the Bay area rationality community has done extraordinarily well.

Truth-seeking is seen as paramount: individuals are socially rewarded for admitting ignorance and changing their minds. Good faith and curiosity about other people’s beliefs is common.

Analytical and quantitative reasoning is highly respected, and increasingly, so is embodied intuition.

People get status for doing good scholarship (e.g. Sarah Constantin), for insightful analysis of complicated situations (e.g. Scott Alexander, for instance), or for otherwise producing good or interesting intellectual content (e.g. Eliezer).

Betting (putting your money where your mouth is) is socially-encouraged. Concepts like “crux” and “rationalist taboo” are well known enough to be frequently invoked in conversation.

Compared to the backdrop of mainline American culture, where admitting that you were wrong means losing face, and trying to figure out what’s true is secondary (if not outright suspicious, since it suggests political non-allegiance), the rationalist bubble’s culture of truth seeking is an impressive accomplishment.

Mental habits

For lack of a better term, I’m going to call this second dimension “mental habits” (or perhaps to borrow Leverage’s term “IPs”).

The thing that I care about in this category is “does a given individual reliably execute some specific cognitive move, when the situation calls for it?” or “does a given individual systematically avoid a given cognitive error?”

Some examples, to gesture at what I mean

  • Never falling prey to the planning fallacy
  • Never falling prey to sunk costs
  • Systematically noticing defensiveness and deflinching or a similar move
  • Systematically noticing and responding to rationalization phenomenology
  • Implementing the “say oops” skill, when new evidence comes to light that overthrows an important position of yours
  • Systematic avoidance of the sorts of errors I outline my Cold War Cognitive Errors investigation (this is the only version that is available at this time).

The element of reliability is crucial. There’s a way that culture is about “counting up” (some people know concept X, and use it sometimes) and mental habits is about “counting down” (each person rarely fails to execute relevant mental process Y).

The reliability of mental habits (in contrast with some mental motion that you know how to do and have done once or twice), is crucial, because it puts one in a relevantly different paradigm.

For one thing, there’s a frame under which rationality is about avoiding failure modes: how to succeed in a given domain depends on the domain, but rationality is about how not to fail, generally. Under that frame, executing the correct mental motion 10% of the time is much less interesting and impressive than executing it everytime (or even 90% of the time).

If the goal is to avoid the sorts of errors in my cold war post, then it is not even remotely sufficient for individuals to be familiar with the patches: they have to reliably notice the moments of intervention and execute the patches, almost every time, in order to avoid the error in the crucial moment.

Furthermore, systematic execution of a mental TAP allows for more complicated cognitive machines. Lots of complex skills depend on all of the pieces of the skills working.

It seems to me, that along this dimension, the rationality community has done dismally.

Eliezer wrote about Mental Habits of this sort in the sequences and in his other writing, but when I consider even very advanced members of my community, I think very few of them systematically notice rationalization, or will reliably avoid sunk costs, or consistently respond to their own defensiveness.

I see very few people around me who explicitly attempt to train 5-second or smaller rationality skills. (Anna and Matt Fallshaw are exceptions who come to mind).

Anna gave a talk at the CFAR alumni reunion this year, in which she presented two low-level cognitive skills of that sort. There were about 40 people in the room watching the lecture, but I would be mildly surprised if even 2 of those people reliably execute the skills described, in the relevant-trigger situation, 6 months from that talk.

But I can imagine a nearby world, where the rationality community was more clearly a community of practice, and most of the the people in that room, would watch that talk and then train the cognitive habit to that level of reliability.

This is not to say that fast cognitive skills of this sort are what we should be focusing on. I can see arguments that culture really is the core thing. But nevertheless, it seems to me that the rationality community is not excelling on the dimension of training it’s members in mental TAPs.

[Added note: Brienne’s Tortoise skills is nearly archetypal of what I mean by “mental habits”.]

RAND needed the “say oops” skill

~ elityre ~ 1 Comment

[Epistemic status: a middling argument]

A few months ago, I wrote about how RAND, and the “Defense Intellectuals” of the cold war represent another precious datapoint of “very smart people, trying to prevent the destruction of the world, in a civilization that they acknowledge to be inadequate to dealing sanely with x-risk.”

Since then I spent some time doing additional research into what cognitive errors and mistakes  those consultants, military officials, and politicians made that endangered the world. The idea being that if we could diagnose which specific irrationalities they were subject to, that this would suggest errors that might also be relevant to contemporary x-risk mitigators, and might point out some specific areas where development of rationality training is needed.

However, this proved somewhat less fruitful than I was hoping, and I’ve put it aside for the time being. I might come back to it in the coming months.

It does seem worth sharing at least one relevant anecdote, from Daniel Ellsberg’s excellent book, the Doomsday Machine, and analysis, given that I’ve already written it up.

The missile gap

In the late nineteen-fifties it was widely understood that there was a “missile gap”: that the soviets had many more ICBM (“intercontinental ballistic missiles” armed with nuclear warheads) than the US.

Estimates varied widely on how many missiles the soviets had. The Army and the Navy gave estimates of about 40 missiles, which was about at parity with the the US’s strategic nuclear force. The Air Force and the Strategic Air Command, in contrast, gave estimates of as many as 1000 soviet missiles, 20 times more than the US’s count.

(The Air Force and SAC were incentivized to inflate their estimates of the Russian nuclear arsenal, because a large missile gap strongly necessitated the creation of more nuclear weapons, which would be under SAC control and entail increases in the Air Force budget. Similarly, the Army and Navy were incentivized to lowball their estimates, because a comparatively weaker soviet nuclear force made conventional military forces more relevant and implied allocating budget-resources to the Army and Navy.)

So there was some dispute about the size of the missile gap, including an unlikely possibility of nuclear parity with the Soviet Union. Nevertheless, the Soviet’s nuclear superiority was the basis for all planning and diplomacy at the time.

Kennedy campaigned on the basis of correcting the missile gap. Perhaps more critically, all of RAND’s planning and analysis was concerned with the possibility of the Russians launching a nearly-or-actually debilitating first or second strike.

The revelation

In 1961 it came to light, on the basis of new satellite photos, that all of these estimates were dead wrong. It turned out the the Soviets had only 4 nuclear ICBMs, one tenth as many as the US controlled.

The importance of this development should be emphasized. It meant that several of the fundamental assumptions of US nuclear planners were in error.

First of all, it meant that the Soviets were not bent on world domination (as had been assumed). Ellsberg says…

Since it seemed clear that the Soviets could have produced and deployed many, many more missiles in the three years since their first ICBM test, it put in question—it virtually demolished—the fundamental premise that the Soviets were pursuing a program of world conquest like Hitler’s.

…

That pursuit of world domination would have given them an enormous incentive to acquire at the earliest possible moment the capability to disarm their chief obstacle to this aim, the United States and its SAC. [That] assumption of Soviet aims was shared, as far as I knew, by all my RAND colleagues and with everyone I’d encountered in the Pentagon:

The Assistant Chief of Staff, Intelligence, USAF, believes that Soviet determination to achieve world domination has fostered recognition of the fact that the ultimate elimination of the US, as the chief obstacle to the achievement of their objective, cannot be accomplished without a clear preponderance of military capability.

If that was their intention, they really would have had to seek this capability before 1963. The 1959–62 period was their only opportunity to have such a disarming capability with missiles, either for blackmail purposes or an actual attack. After that, we were programmed to have increasing numbers of Atlas and Minuteman missiles in hard silos and Polaris sub-launched missiles. Even moderate confidence of disarming us so thoroughly as to escape catastrophic damage from our response would elude them indefinitely.

Four missiles in 1960–61 was strategically equivalent to zero, in terms of such an aim.

This revelation about soviet goals was not only of obvious strategic importance, it also took the wind out of the ideological motivation for this sort of nuclear planning. As Ellsberg relays early in his book, many, if not most, RAND employees were explicitly attempting to defend US and the world from what was presumed to be an aggressive communist state, bent on conquest. This just wasn’t true.

But it had even more practical consequences: this revelation meant that the Russians had no first strike (or for that matter, second strike) capability. They could launch their ICBMs at American cities or military bases, but such an attack had no chance of debilitating US second strike capacity. It would unquestionably trigger a nuclear counterattack from the US who, with their 40 missiles, would be able to utterly annihilate the Soviet Union. The only effect of a Russian nuclear attack would be to doom their own country.

[Eli’s research note: What about all the Russian planes and bombs? ICBMs aren’t the the only way of attacking the US, right?]

This means that the primary consideration in US nuclear war planning at RAND and elsewhere, was fallacious. The Soviet’s could not meaningfully destroy the US.

…the estimate contradicted and essentially invalidated the key RAND studies on SAC vulnerability since 1956. Those studies had explicitly assumed a range of uncertainty about the size of the Soviet ICBM force that might play a crucial role in combination with bomber attacks. Ever since the term “missile gap” had come into widespread use after 1957, Albert Wohlstetter had deprecated that description of his key findings. He emphasized that those were premised on the possibility of clever Soviet bomber and sub-launched attacks in combination with missiles or, earlier, even without them. He preferred the term “deterrent gap.” But there was no deterrent gap either. Never had been, never would be.

To recognize that was to face the conclusion that RAND had, in all good faith, been working obsessively and with a sense of frantic urgency on a wrong set of problems, an irrelevant pursuit in respect to national security.

This realization invalidated virtually all of RAND’s work to date. Virtually every, analysis, study, and strategy, had been useless, at best.

The reaction to the revelation

How did RAND employees respond to this reveal, that their work had been completely off base?

That is not a recognition that most humans in an institution are quick to accept. It was to take months, if not years, for RAND to accept it, if it ever did in those terms. To some degree, it’s my impression that it never recovered its former prestige or sense of mission, though both its building and its budget eventually became much larger. For some time most of my former colleagues continued their focus on the vulnerability of SAC, much the same as before, while questioning the reliability of the new estimate and its relevance to the years ahead. [Emphasis mine]

…

For years the specter of a “missile gap” had been haunting my colleagues at RAND and in the Defense Department. The revelation that this had been illusory cast a new perspective on everything. It might have occasioned a complete reassessment of our own plans for a massive buildup of strategic weapons, thus averting an otherwise inevitable and disastrous arms race. It did not; no one known to me considered that for a moment. [Emphasis mine]

According to Ellsberg, many at RAND were unable to adapt to the new reality and continued (fruitlessly) to continue with what they were doing, as if by inertia, when the thing that they needed to do (to use Eliezer’s turn of phrase) is “halt, melt, and catch fire.”

This suggests that one failure of this ecosystem, that was working in the domain of existential risk, was a failure to “say oops“: to notice a mistaken belief, concretely acknowledge that is was mistaken, and to reconstruct one’s plans and world views.

Relevance to people working on AI safety

This seems to be at least some evidence (though, only weak evidence, I think), that we should be cautious of this particular cognitive failure ourselves.

It may be worth rehearsing the motion in advance: how will you respond, when you discover that a foundational crux of your planning is actually mirage, and the world is actually different than it seems?

What if you discovered that your overall approach to making the world better was badly mistaken?

What if you received a strong argument against the orthogonality thesis?

What about a strong argument for negative utilitarianism?

I think that many of the people around me have effectively absorbed the impact of a major update at least once in their life, on a variety of issues (religion, x-risk, average vs. total utilitarianism, etc), so I’m not that worried about us. But it seems worth pointing out the importance of this error mode.

A note: Ellsberg relays later in the book that, durring the Cuban missile crisis, he perceived Kennedy as offering baffling terms to the soviets: terms that didn’t make sense in light of the actual strategic situation, but might have been sensible under the premiss of a soviet missile gap. Ellsberg wondered, at the time, if Kennedy had also failed to propagate the update regarding the actual strategic situation.

I believed it very unlikely that the Soviets would risk hitting our missiles in Turkey even if we attacked theirs in Cuba. We couldn’t understand why Kennedy thought otherwise. Why did he seem sure that the Soviets would respond to an attack on their missiles in Cuba by armed moves against Turkey or Berlin? We wondered if—after his campaigning in 1960 against a supposed “missile gap”—Kennedy had never really absorbed what the strategic balance actually was, or its implications.

I mention this because additional research suggests that this is implausible: that Kennedy and his staff were aware of the true strategic situation, and that their planning was based on that premise.

Initial thoughts about the early history of Circling

~ elityre ~ Leave a comment

I spent a couple of hours over the past week looking into the origins and early history of Circling, as part of a larger research project.

If you want to read some original sources, this was the most useful and informative post on the topic that I found.

You can also read my curated notes (only the things that were most interesting to me), including my thinking about the Rationality Community.

A surprising amount of the original work was done while people were in college. Notably, Bryan, Decker, and Sarah, all taught and developed Circling / AR in the living spaces of their colleges:

“Even before this, Bryan Bayer and Decker Cunov had independently discovered the practice as a tool to resolve conflicts in their shared college household in Missouri,”

“Sara had been a college student, had discovered Authentic Relating Games, had introduced them into her college dorm with great success”

It reminds me that a lot the existence and growth of EA was driven by student groups. I wonder if most movements are seeded by people in their early 20s, and therefore college campuses have been the background for the origins of most movements throughout the past century.

There’s in  way in which the teaching of Circling spread, the way the teaching of rationality didn’t.

It sounds like many of the people who frequently attended the early weekend programs that Guy and Jerry (and others) were putting on, had ambitions to develop and run similar programs of their own one day. And to a large degree, they did. There’s been something like 10 to 15 for pay circling-based programs, across at least 4 organizations. In contrast Rationality has one CFAR, that primarily runs a single program.

I wonder what accounts for the difference?

Hypotheses:

  • Circlers tend to be poor, where rationalist tend to be software engineers. Circlers could dream of doing Circling full time, but there’s not much appeal for rationalists to be teaching rationality full time. (That would be a pay cut, and there’s no “activity” that rationalist love and that they would get to do as their job.)
  • Rationality is too discrete and explicit. Once you’ve taught the rationality techniques you know, you’re done (or you have to be in the business of inventing new ones), whereas teaching Circling is more like a service: there’s not a distinct point when the student “has it” and doesn’t need your teaching, but a gradual apprenticeship.
  • Relatedly, maybe there’s just not enough demand for rationality training. A CFAR workshop is, for most rationalists, is a thing that you do once, whereas Circlers might attend several Circling immersion or trainings in a year. Rationality can become a culture and a way of life, but CFAR workshops are not. As a result, the demand for rationality training amounts to 1 workshop per community member, instead of something like 50 events per community member.
    • Notably, if CFAR had a slightly different model, this feature could change.
  • Rationality is less of concrete thing, separate from the CFAR or LessWrong brands.
    • Because of this, I think most people don’t feel enough ownership of “Rationality” as an independent thing. It’s Eliezer’s thing or CFAR’s thing. Not something that is separate from either of them.
    • Actually, the war between the founders might be relevant here. That Guy and Decker were both teaching Circling highlighted that is was separate from any one brand.
    • I wonder what the world would look like if Eliezer coined a new term for the thing we call rationality, instead of taking on a word that already has meaning in the wider world. I expect there would be less potential for a mass movement, but more and affordance to teach the thing, a feeling that one could be expert at it.
  • Maybe the fact the Circling was independently discovered by Guy and Jerry, and Decker and Bryan, made it obvious that no one owned it.
    • If we caused a second rationality-training organization to crop up, would that cause a profusion of rationality orgs?
  • Circling people acquired enough confidence in their own skills that they felt comfortable charging for them, rationalist don’t.
    • It is more obvious who the people who are skilled in circling is, because you can see it in a Circle.
    • Circling has an activity that is engaging to spend many an hour at and includes a feedback loop, so people became skilled at it in a way that rationalists don’t.

There aren’t people who are trying to build Rationality empires the way Jordan is trying to build a Circling empire.

I get the sense that a surprising number of the core people of circling are what I would call “jocks.” (Though my actual sample is pretty limited)

  • Guy originally worked as a personal trainer.
  • Sean Wilkinson and John Thompson ran a personal tennis academy before teaching Circling.
  • Jordan was a model.

“Many of us lived together in communal houses and/or were in relationships with other community members.”

They had group houses and called themselves “the community”. I wonder how common those threads are, in subcultures across time (or at least across the past century).

When do you need traditions? – A hypothesis

~ elityre ~ Leave a comment

[epistemic status: speculation about domains I have little contact with, and know little about]

I’m rereading Samo Burja’s draft, Great Founder Theory. In particular, I spent some time today thinking about living, dead, and lost traditions and chains of Master-Apprenticeship relationships.

It seems like these chains often form the critical backbone of a continuing tradition (and when they fail, the tradition starts to die). Half of Nobel winners are the students of other Noble winners.

But it also seems like there are domains that don’t rely, or at least don’t need to rely on the conveyance of tacit knowledge via Master-Appreticeship relationships.

For instance, many excellent programmers are self-taught. It doesn’t seem like our civilization’s collective skill in programming depends on current experts passing on their knowledge to the next generation via close in-person contact. As a thought experiment, if all current programers disappeared today, but the computers and educational materials remained, I expect we would return to our current level of collective programing skill within a few decades.

In contrast, consider math. I know almost nothing about higher mathematics, but I would guess that if all now-living mathematicians disappeared, they’ed leave a lot of math, but progress on the frontiers of mathematics would halt, and it would take many years, maybe centuries, for mathematical progress to catch up to that frontier again. I make this bold posit on the basis of the advice I’ve heard (and I’ve personally verified) that learning from tutors is way more effective than learning just from textbooks, and that mathematicians do track their lineages.

In any case, it doesn’t seem like great programers run in lineages the way that Nobel Laureates do.

This is in part because programming in particular has some features that lends itself to autodidactictry: in particular, a novice programer gets clear and immediate feedback: his/her code either compiles or it doesn’t. But I don’t think this is the full story.

Samo discusses some of the factors that determine this difference in his document: for instance, traditions in domains that provide easy affordance for “checking work” against the territory  (such as programming) tend to be more resilient.

But I want to dig into a more specific difference.

Theory:

A domain of skill entails some process that when applied, produces some output.

Gardening is the process, fruits are the output. Carpentry (or some specific construction procedure) is the process, the resulting chair is the output.  Painting is the process, the painting is the output.

To the degree that the output is or embodies the generating process, master-apprenticeship relationships are less necessary.

It’s a well trodden trope that a program is the programmer’s thinking about a problem. (Paul Graham in Holding a Program in One’s Head: “Your code is your understanding of the problem you’re exploring.“) A comparatively large portion of a programmer’s thought process is represented in his/her program (including the comments). A novice programer, looking at a program written by a master, can see not just what a well-written program looks like, but also, to a large degree, what sort of thinking produces a well-writen program. Much of the tacit knowledge is directly expressed in the final product.

Compare this to say, a revolutionary scientist. A novice scientist might read the papers of elite groundbreaking science, and the novice might learn something, but so much of the process – the intuition that the topic in question was worth investigating, the subtle thought process that led to the hypothesis, the insight of what experiment would elegantly investigate that hypothesis – are not encoded in the paper, and are not legible to the reader.

I think that this is a general feature of domains. And this feature is predictive of the degree to which skill in a given domain relies strongly on traditions of Master- Apprenticeship.

Other examples:

I have the intuition, perhaps false (are there linages of award-winning novelist the way there are linages of Nobel laureates?), that novelists mostly do not learn their craft in apprenticeships to other writers. I suggest that writing is like programing: largely self-taught, except in the sense that one ingests and internalizes large numbers of masterful works. But enough of the skill of writing great novels is contained in the finished work that new novelists can be “trained” this way.

What about Japanese wood-block printing? From the linked video, it seems as if David Bull received about an hour of instruction in wood carving once every seven years or so. But those hours were enormously productive for him. Notably, this sort of wood-carving is a step removed from the final product: one carves the printing block, and then uses the block to make a print. Looking at the finished block, it seems, does not sufficiently convey the techniques used for creating the block. But on top of that the block is not the final product, only an intermediate step. The novice outside of an apprenticeship may only ever see the prints of a master-piece, not the blocks that make the prints.

Does this hold up at all?

That’s the theory. However, I can come up with at least a few counter proposals and confounding factors:

Countertheory: The dominating factor is the age of the tradition. Computer Science is only a few decades old, so recreating it can’t take more than a few decades. Let it develop for a few more centuries (without the advent of machine intelligence or other transformative technology), and the Art of Programming will have progressed so far that it does depend on Master/Apprentice relationships, and the loss of all living programers would be as much as a hit as the loss of all living mathematicians.

This doesn’t seem like it explains novelists, but maybe “good writing” is mostly a matter of fad? (I expect some literary connoisseurs would leap down my throat at that. In any case, it doesn’t seem correct to me.)

Confounder: economic incentive: If we lost all masters of Japanese wood-carving, but there was as much economic incentive for the civilization to remaster it as there would be for remastering programming, would it take any longer? I find that dubious.

Why does this matter? 

Well for one thing, if you’re in the business of building traditions to last more than a few decades, it’s pretty important to know when you will need to institute close-contact lineages.

Separately, this seem relevant whenever one is hoping to learn from dead masters.

Darwin surely counts among the great scientific-thinkers. He successfully abstracted out a fundamental structuring principle of the natural world. As someone interested in epistemology, it seems promising to read Darwin, in order to tease out how he was thinking. I was previously planning to read the Origins of Species. Now, it seems much more fruitful to read Darwin’s notebooks, which I expect to contain more of his process than his finished works do.

 

 

 

Initial Comparison between RAND and the Rationality Cluster

~ elityre ~ 2 Comments

I’m currently reading The Doomsday Machine: Confessions of a Nuclear War Planner by Daniel Ellsberg (the man who leaked the Pentagon Papers), on the suggestion of Anna Salamon.

I’m interested in the cold war planning communities because they might be relevant to the sort of thinking that is happening, or needs to happen, around AI x-risk, today. And indeed, there are substantial resemblances between the RAND corporation and at least some of the orgs that form the core of the contemporary x-risk ecosystem.

For instance…

A narrative of “saving the world”:

[M]y colleagues were driven men. They shared a feeling—soon transmitted to me—that we were in the most literal sense working to save the world. A successful Soviet nuclear attack on the United States would be a catastrophe, and not only for America.

A perception of the inadequacy of the official people in power:

But above all, precisely in my early missile-gap years at RAND and as a consultant in Washington, there was our sense of mission, the burden of believing we knew more about the dangers ahead, and what might be done about them, than did the generals in the Pentagon or SAC, or Congress or the public, or even the president. It was an enlivening burden.

We were rescuing the world from our Soviet counterparts as well as from the possibly fatal lethargy and bureaucratic inertia of the Eisenhower administration and our sponsors in the Air Force.

Furthermore, a major theme of the book is the insanity of US Nuclear Command and Control polices.  Ellsberg points repeatedly at the failures of decision-making and morality amongst the US government.

A sense of intellectual camaraderie:

In the middle of the first session, I ventured—though I was the youngest, assigned to be taking notes, and obviously a total novice on the issues—to express an opinion. (I don’t remember what it was.) Rather than showing irritation or ignoring my comment, Herman Kahn, brilliant and enormously fat, sitting directly across the table from me, looked at me soberly and said, “You’re absolutely wrong.” A warm glow spread throughout my body. This was the way my undergraduate fellows on the editorial board of the Harvard Crimson (mostly Jewish, like Herman and me) had routinely spoken to each other; I hadn’t experienced anything like it for six years. At King’s College, Cambridge, or in the Society of Fellows, arguments didn’t remotely take this gloves-off, take-no-prisoners form. I thought, “I’ve found a home.”

Visceral awareness of existential failure:

At least some of the folks at RAND had a visceral sense of the impending end of the world. They didn’t feel like they were just playing intellectual games.

I couldn’t believe that the world would long escape nuclear holocaust. Alain Enthoven and I were the youngest members of the department. Neither of us joined the extremely generous retirement plan RAND offered. Neither of us believed, in our late twenties, we had a chance of collecting on it.

That last point seems particularly relevant. Folks in our cluster invest in the development and practice of tools like IDC in part because of the psychological pressures that accompany the huge stakes of x-risk.

At least some of the “defense intellectuals” of the Cold War were under similar pressures.[1]

For this reason, the social and intellectual climate around RAND and similar organizations during the Cold War represents an important case study, a second data point for comparison to our contemporaries working on existential risk.

How did RAND employees handle the psychological pressures? Did they spontaneously invent strategies for thinking clearly in the face of the magnitude of the stakes? If so, can we emulate those strategies? If not, does that imply that their thinking about their work was compromised? Or does it suggest that our emphasis on psychological integration methods are misplaced?

And perhaps most importantly, what mistakes did they make? Can we use their example to foresee similar mistakes of our own and avoid them?

[1] – Indeed, it seems like they were under greater pressures. There’s a sense of franticness and urgency that I feel in Ellsberg’s description that I don’t feel around MIRI. But I think that this is due to the time horizons that RAND and co. were operating under compared to the those that MIRI is operating under. I expect that as we near the arrival of AGI, there will be a sense of urgency and psychological pressure that is just as great and greater than that of the cold war planners.

End note: In addition to all these more concrete correlations, there’s also the intriguing intertwining of existential risk and decision theory in both of the data points of nuclear war planning and AI safety. I wonder if that is merely coincidence or represents some deeper connection.