First pass on estimating how much I benefit from structural racism

Suppose I wanted to estimate, quantitatively, how much I, personally, have benefited from historical and contemporary structural racism.

I’m interested in this question because I’m at least somewhat sympathetic to the argument that, if I personally benefited from oppressive systems, then there’s some fraction of “my” resources to which I don’t have a legitimate claim. 

(It’s not obvious how I should respond to that situation. A first thought is that I should donate that fraction of my wealth to racial justice charities. In the ideal case those charities might function as an offset, effectively compensating for the harm done. At minimum, disowning that fraction of my resources correctly aligns the incentives. If every person followed this policy <link to deontological principles>, there would be no incentive to enforce white supremacy in the first place. I would prefer not to live in a condition of being incentivized to turn a blind eye to racial injustice, and score moral points by condemning it after the fact. Though depending on the size of the numbers that might entail large moral or personal tradeoffs, and I’ll have to think more about how best to act.)

So how might I go about calculating the personal benefit that I’ve derived from the oppression of blacks in America?

These are some of my first pass thoughts.

Framing the question

Some of the ways that I could possibly have benefited:

  • I was born to a wealthier family, because my parents and grandparents were afforded privileges and advantages at the expense of black people.
  • I had better access to education for being white.
  • I had better access to jobs for being white.
  • I had way lower risk of staying out of prison, including for spurious or trivial offenses.

…and I think that’s about it? (I welcome additional suggestions)

Given my specific, highly unusual work history, I find it pretty implausible that I, personally, benefited from racial privilege.

(It’s possible that my community is more racist than I imagine, but eg I find it pretty hard to imagine Anna or Critch or Oliver turning down my earnest help in the counterfactual where I have black skin. But maybe I’m overstating the degree to which most white people will tend to take black people less seriously, taking their errors and mistakes as stronger evidence for incompetence.)

My educational opportunities seem basically mediated by the wealth of my parents.

So it seems like this question reduces to estimating what fraction of my parent’s relative wealth depends on 0-sum privileges at the expense of black people and to calculating the expected risk of being unfairly imprisoned as a white vs. black person.

A note on what I’m looking for

I’m not just looking for advantages or privileges that I benefit from, that black people lack. I’m looking for places where I derived benefits at the expense of black people, or other racial groups.

It’s straightforward that there barriers to black advancement that I’m just completely free of. My life was clearly made easier by being white.

But those barriers might be in the form of transfers of value, effectively theft from black Americans to white americans. In that case white americans or a subset of white americans come out ahead in absolute terms, from the existence of structural racism. And some of those barriers might be in the form of destruction of value. In this case, the white Americans (or some white Americans) come out ahead in relative terms, because the diff between whites and blacks is bigger, but not in absolute terms.

Economics, in particular, is not in general 0-sum. That there are privileges that I have and others don’t might be bad for them, but that doesn’t necessarily mean that it is good for me.

One way to frame the question: We can consider a hypothetical world where there wasn’t any systemic racism. In that world, would I have been poorer than in actual history, because in actual history I derived direct and/or indirect benefits from white supremacy? Or would my personal wealth have been about the same (adjusted somewhat for having greater purchasing power in an overall wealthier society), but black people, on average, would be richer. 

In the first case, I’m benefiting from structural racism. In the second case, structural racism is still a massive evil perpetuated in my society, for which I bear at least a little responsibility, but I’m not a beneficiary.

I care about this distinction, because it informs whether the wealth that is legally mine, is legitimately mine to do with as I want, according to my own morality, or not. If my parents gave me some advantages in life, and I capitalized on those advantages to create value in a mostly free market, I consider myself to have a moral right to my share of the gains from trade. But if my advantages are 0-sum, and came, unjustly, at the expense of other racial groups, even if I didn’t choose to opress them, my ownership of that wealth is far more suspect. 

I’m not sure how I should respond to that possibility. Maybe I should give away whatever fraction of my wealth is illigitmately earned, ideally finding some way to reinvest it in the communities from which it was stolen? That might turn out to be unrealistic / infeasible, but it seems like I should owe some extra responsibility if I am the beneficiary of an ongoing theft.

Family Wealth

First of all, it is possible that my parents are financially worse off, on-net, for the existence of structural racism in America. The economy is not, in general, a 0-sum game. Typically, everyone benefits from more people having more economic opportunity, because those people are more productive and society as a whole is wealthier.

It’s plausible that, over the course of the 20th century, rich Southern whites were deriving benefits from oppressing and extracting value from blacks, but that most whites in most of the US were made materially worse off, not better off by this. (In fact, most people are not very literate in economics. It may be that even the whites actively perpetuating white supremacy didn’t benefit on net, and would have been richer if they had created a fairer and more Just society.)

I need to find not just places where my parents or grandparents had advantages that black people mostly lacked, but 0-sum privileges that they were afforded at the expense of black people.

My personal situation: 

My mother was born and grew up in Boston. My maternal grandparents were Jewish, second-generation immigrants from Poland. My grandfather worked as an engineer, including for military contractors.

My dad was likewise born and raised in the Northeast. I don’t know as much about his parentage. Also a third generation immigrant, I think, from Ireland.

Both my mom and dad went to college, then moved to New York city, and worked in the computer industry, doing sales. They’re well off now, in large part because when they were a young couple, they were both motivated by money, and made a lot of it for their age, but they were also judicious: mostly living on one salary and saving the other.

Did they have an advantage securing those jobs because they were white? Would a black man have been handicapped in how much he could sell because of racism?

[Edit: I talked with my dad about his early corporate experience. From 1981 to 1992, he worked for a company called Businessland, selling computers. When he started in, Boston, 2 out of about ~150 people who worked for the company in that region were black. 10 years later, in the early 90’s, when he was a sales manager in New York, about 25% of the ~120 people in that region were either black or hispanic.

He relayed to me that hiring a black person in the 80’s was generally seen as a big risk, and that Boston, in particular, was extremely racist, with segregated neighborhoods that felt and expressed antipathy for each other. New York was better, both because NYC is a uniquely diverse melting pot, and because by the 90s, overt racism had declined.]

Probably the answer to both of those is “at least a little”. But the jobs they held were not sinecures. Sales in particular is a domain in which you can measure performance, which is why salespeople get paid on commission. Someone who was as driven as my dad, but black, would surely have faced discrimination, but how much less would he have made?

But there’s a key point here which is that my parents did work for their wealth. That there were barriers deliberately and emergently placed in front of black people, to make it harder for them to get ahead, doesn’t illegitimate my parents’ wealth accumulation. 

I already know that being black in the United States is a severe handicap. But I want to know in what ways those handicaps were transfers of value from one person to another, not just destruction of value. 

Avoiding prison

Thinking about this one a little further, I expect to see the same dynamic as above, except more strongly. It’s manifestly unjust that a black person goes to prison for smoking marujana, and a white person doesn’t. 

And that’s on the tip of the iceberg of ways that the criminal justice system extorts black / low class people.

But all of those are examples where white people are being granted the rights due to them in a Just society, while black people are being denied those rights. Not a situation where white people are benefiting from special extrajudicial privileges above what is due to them by law. 

(Admittedly it is technically illegal to smoke marujana in some places, but only technically, and I’m not tempted to say that white people are “above the law”, in that case. The law, in that case, is a farce, used to penalize marginalized people).

It’s obviously unjust to have a society which claims, but doesn’t follow through on, equal treatment before the law. That’s obviously evil. 

But I don’t think that I benefit from that evil. Again, my risk of going to prison is lower than that of a black person, but that’s not because I’m externalizing a conserved quantity of “expected years in prison” onto someone else. If we reformed society so that it became Just, there would be many, many fewer black people in prison, but my own risk of going to prison wouldn’t change. If anything, it would go down somewhat, since “injustice anywhere is a threat to justice everywhere”. As a Jew, and as a human being, I should expect to be safer in a just society than in a society that maintains Justice for only a subset of its population.)

Conclusions from first thoughts and next steps

My tentative conclusion is that because of explicit, unjust discrimination, it is much harder to be black in America than to be white, and that was especially true in previous centuries, most Northern whites didn’t actually benefit from that discrimination, especially those who were primarily accumulating wealth via economic production instead of privileged access to rents.

But these are only first pass thoughts. My next step is to collect and read some books about racial wealth inequality and white privilege, to build up a more grounded list of ways that I might have benefited from structural racism.

Some thoughts on Agents and Corrigibility

[Reproducing this comment on LessWrong, with slight edits]

“Prosaic alignment work might help us get narrow AI that works well in various circumstances, but once it develops into AGI, becomes aware that it has a shutdown button, and can reason through the consequences of what would happen if it were shut down, and has general situational awareness along with competence across a variety of domains, these strategies won’t work anymore.”

I think this weaker statement now looks kind of false in hindsight, since I think current SOTA LLMs are already pretty much weak AGIs, and so they already seem close to the threshold at which we were supposed to start seeing these misalignment issues come up. But they are not coming up (yet). I think near-term multimodal models will be even closer to the classical “AGI” concept, complete with situational awareness and relatively strong cross-domain understanding, and yet I also expect them to mostly be fairly well aligned to what we want in every relevant behavioral sense.

I basically still buy the quoted text and don’t think it now looks false in hindsight.

We (apparently) don’t yet have models that have robust longterm-ish goals. I don’t know how natural it will be for models to end up with long term goals: the MIRI view says that anything that can do science will definitely have long-term planning abilities which fundamentally entails having goals that are robust to changing circumstances. Maybe that’s true, maybe it isn’t.. Regardless, I expect that we’ll specifically engineer agents with long term goals. (Whether or not those agents will have “robust” long term goals, over and above what they are prompted to do/want in a specific situation, is also something that I don’t know.)

What I expect to see is agents that have a portfolio of different drives and goals, some of which are more like consequentialist objectives (eg “I want to make the number in this bank account go up”) and some of which are more like deontological injunctions (“always check with my user/ owner before I make a big purchase or take a ‘creative’ action outside of my training distribution”).

My prediction is that the consequentialist parts of the agent will basically route around any deontological constraints that are trained in, even if the agent is sincerely committed to the demonological constraints . 

As an example, your personal assistant AI does ask your permission before it does anything creative, but also, its superintelligently persuasive. So it always asks your permission in exactly the way that will result in it accomplishing what it wants. If there are a thousand action sequences in which it asks for permission, it picks the one that has the highest expected value with regard to it’s consequentialist goal. This basically nullifies the safety benefit of any deontological injunction, unless there are some that can’t be gamed in this way.

To do better than this, it seems like you do have to solve the Agent Foundations problem of corrigibility (getting the agent to be sincerely indifferent between your telling it to take the action or not take the action) or you have to train in not a deontological injunction, but an active consequentialist goal of serving the interests of the human (which means you have find a way to get the Agent to be serving some correct enough idealization of human values).

But I think we mostly won’t see this kind of thing until we get quite high levels of capability, where it is transparent to the agent that some ways of asking for permission have higher expected value than others. Or rather, we might see a little of this effect early on, but until your assistant is superhumanly persuasive it’s pretty small. Maybe we’ll see a bias toward accepting actions that serve the AI agents goals (if we even know what those are) more, as capability goes up, but we won’t be able to distinguish “the AI is getting better at getting what it wants from the human” from “the AIs are just more capable, and so they come up with plans that work better.” It’ll just look like the numbers going up.

To be clear, “superhumanly persuasive” is only one, particularly relevant, example of a superhuman capability that allows you to route around deontological injunctions that the agent is committed to. My claim is weaker if you remove that capability in particular, but mostly what I’m wanting to say is that powerful consequentialism find and “squeezes through” the gaps in your oversight and control and naive-corrigibility schemes, unless you figure out corrigibility in the Agent Foundations sense.

Smart Sessions – Finally a (kinda) window-centric session manager

This is a short post about some software functionality that I’ve long wanted, and a browser extension that gets most of it well enough.

The dream

There’s a simple piece of software that I’ve wanted for several years. Few months, I go on a binge of trying to find something that do what I’m looking for.

Basically: a session manager that allows you to group windows together somehow, so that you can close and save them all with one click, and then reopen them all with one click.

I make heavy use of the OSX feature “desktops”, which allows multiple separate workspaces in parallel. I’ll typically have a desktop for my logging and tracking, one for chats and coms, one with open blog posts, one with writing projects, one with an open coding project, etc. Each of these are separate contexts that I can switch between for doing different kinds of work.

What I want is to be able to easily save each of those contexts, and easily re-open them later.

But since I’ll often have multiple sessions open at the same time, across multiple desktops, I don’t want the session-saver app to save all my windows. Just the ones that are part of a given workspace context.

The best way to do this is if the software could tell which windows were open on which desktops and use that as the discriminator. But some sort of manual drag and drop for adding a (representation of) a window (on a list of windows) to a group would work too.

The situation

This seems to me like something that…there should be a lot of demand for? I think lots of people have many windows, related to different projects that they want to keep separate, open on their computer at the same time.

But, as near as I can tell there’s almost nothing like this.

There are a lot of session managers, browser extensions that allow you to save your tabs for the future (I’ve mostly used OneTab, but there are dozens). However, they’re virtually all tab-centric. A “session” typically refers to a single window, with multiple tabs, not to multiple windows, with multiple tabs each, which means that to reopen a session (in my “multiple windows” sense of the word), I need to mentally keep track of which windows were included and open all of them one by one, instead of clicking one button to get the context back.

There are a few session managers that save multiple windows in a session (I’m thinking of Session Buddy or the less polished Tab Session Manager), but these have the opposite problem: they save all the open windows, including those that are part of other workflows in other desktops, which means that I have to go through and manually remove them every time I save a session. (This is especially a problem for me because there’s a set of windows that I always keep open on my first desktop.) And on top of that, they tend to save sessions as static snapshots, rather than as mutable objects that change as you work with them, so you need to repeatedly delete old sessions and replace them with updated ones.

Success!

I spent a few hours over the past week, yet again, reading about and trying a bunch of tab managers in rapid succession to find any that have anything like the functionality I’m wanting.

I finally found exactly one that does what I want!

It is a little finicky, with a bunch of small to medium sized UX problems. But it is good enough that I’m going ahead and making a point to try using it.

I’m sharing this here because maybe other people have also been wanting this functionality, and they can benefit from fruits of my laborious searching.

Current solution: Smart Sessions – Tab Manager

Smart Sessions is a chrome extension that does let you save groups of windows. This is the best one that I’ve found so far.

When you click on the icon, there’s a button for creating a new session. When you click it, it displays a list of all your current open tabs (another button organizes all those tabs by window), with checkboxes. The user checks the windows that they want to be included in a session. You give it a name and then create the session.

While a session is active (and while a default setting called “auto save” is set to Yes), when you close a tab or a window, it removes that tab or window from the session (though it does create a weird popup every time). You can also remove tabs/windows from the list manually.


The weird popup. It’s not super clear from the text what the options mean, but I think “stop tracking” deactivates the session, and “save” removes the window you just closed from the active session.

You can press the stop button, which closes all the windows, to be reopened later.

When the session is inactive, you can edit the list of tabs and windows that compose a session, removing some (though I think not adding?). You can also right click on any page, select Smart Sessions, and add that page to any session, active or not.

At the bottom of the session list, there’s a button that deletes the session.

This basically has functionality that I want! 

I want to first and foremost give a big hurrah to the developer Serge (Russo?), for being the only person in the world to make what seems to me and obvious and extremely helpful tab-management tool. Thank you Serge!

Some issues or weird behavior

However, it still has a fundamentally tab-centric design, with multi-window sessions seeming like concessions or afterthoughts, rather than core to the user experience. This results in some weird functionality. 

  • Every time you create a new session, you need to click a button so that the selection list is separated by window, instead of only a list of tabs. If you don’t click this button, the selection list is a flat list of tabs, and when you create the session, all the selected tabs will be combined into a single window.
    • (One UX that I could imagine is having a global setting on the settings page, “tab-centric default” vs “window-centric default”. You can still press the button to toggle individual sessions, but for window-centric session users, having a default would save me a button press each time.
  • I think as a side effect of the above feature, whenever you create a new session, it takes all the windows of that session (regardless of where they are on the screen or across different desktops) and stacks them all on top of each other, so that only the top one is visible (not even some overlap so you can see how many windows are stacked on top of each other). 
  • It would be intuitive if, while a session is active, if you opened a new window, that window was automatically added to the session. Not only does that not work, there appears to be no way to add new windows to a session. New tabs get added to the session, but not new windows. The right-click “add to session functionality”, adds a single page, as a new tab in one of the windows of a session, not as a new window in that session.
    • The only way, as near as I can find, to increase the number of windows in a session is to drag a tab from a multi-tab window into its own window—both resulting windows are saved as part of the session. In order to add new windows to a session, the user needs to do an an awkward maneuver to exploit this functionality: first create a new tab in a window that’s part of the session, and then drag it to it’s own window. Or alternatively, make a new window, add that window as a tab, to one of the windows that is part of the active session, and then drag it out again. That tab, again in its own window, will be added to the session.
  • As noted above, every time you close a window, that’s part of the active session, this activates a popup.

It would be great if these issues were addressed.

Additionally, for  some reason the extension is slow to load. Sometimes (but not always), I’ll click on the icon and it will take a full two seconds for the list of sessions to appear. I haven’t yet figured out what the pattern is for why there’s sometimes a delay and sometimes not.

And finally, there are some worrying reviews that suggest that at least sometimes, the whole history disappears? I’m not sure what’s up with that, but I’m going to make a point to regularly export all my sessions (there’s easy export functionality), just to be careful.

Overall though, this so far works, and I feel pretty excited about it.

Very low energy states seem to “contain” “compressed” fear or despair.

[I wrote this elsewhere, but wanted it someplace where I could link to it in isolation.]

When I’m feeling very low, I can often do Focusing and bring up exactly the concern that feels hopeless or threatened.

When I feel into fear “underneath” the low energy state, the fear (which I was ignoring or repressing a moment ago), sort of inverts, and comes blaring into body-awareness as panic or anxiety, and my energy comes back. Usually, from there, I can act from the anxiety, moving to take action on the relevant underhanded concern.

[Example in my logs on April 10]

When I feel into the low energy, and there’s despair underneath, usually the thing thing that needs to happen is move that is like “letting reality in” (this has a visual of a vesicle, with some substance inside of it, which when the membrane is popped defuses and equalizes with the surrounding environment) or grieving. Usually after I do that, my energy returns.

(Notably there seems to be an element of hiding from or ignoring or repressing what’s true in each of these.)

In both cases, it sort of feels like the low energy state is the compacted from of the fear or despair, like snow that has been crushed solid. And then in doing the Focusing, I allow it to decompress.

Rough Thoughts on the White House Executive order on AI

I spent a few hours reading, and parsing out, sections 4 and 5 of the recent White House Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence.

The following are my rough notes on each subsection in those two subsections, summarizing what I understand each to mean, and my personal thoughts.

My high level thoughts are at the bottom.

Section by section

Section 4 – Ensuring the Safety and Security of AI Technology.

4.1

  • Summary:
    • The secretary of commerce and NIST are going to develop guidelines and best practices for AI systems.
    • In particular:
      • “launching an initiative to create guidance and benchmarks for evaluating and auditing AI capabilities, with a focus on capabilities through which AI could cause harm, such as in the areas of cybersecurity and biosecurity.”
        • What does this literally mean? Does this allocate funding towards research to develop these benchmarks? What will concretely happen in the world as a result of this initiative?
    • It also calls for the establishment of guidelines for conducting red-teaming.
      • [[quote]]
        • (ii)  Establish appropriate guidelines (except for AI used as a component of a national security system), including appropriate procedures and processes, to enable developers of AI, especially of dual-use foundation models, to conduct AI red-teaming tests to enable deployment of safe, secure, and trustworthy systems.  These efforts shall include:
          •  (A)  coordinating or developing guidelines related to assessing and managing the safety, security, and trustworthiness of dual-use foundation models; and
          • (B)  in coordination with the Secretary of Energy and the Director of the National Science Foundation (NSF), developing and helping to ensure the availability of testing environments, such as testbeds, to support the development of safe, secure, and trustworthy AI technologies, as well as to support the design, development, and deployment of associated PETs, consistent with section 9(b) of this order.
  • Commentary:
    • I imagine that these standards and guidelines are going to be mostly fake.
    • Are there real guidelines somewhere in the world? What process leads to real guidelines?

4.2

  • Summary:
    • a
      • Anyone who has or wants to train a foundation model, needs to
        • Report their training plans and safeguards.
        • Report who has access to the model weights, and the cybersecurity protecting them
        • The results of red-teaming on those models, and what they did to meet the safety bars
      • Anyone with a big enough computing cluster needs to report that they have it.
    • b
      • The Secretary of Commerce (and some associated agencies) will make (and continually update) some standards for models and computer clusters that are subject to the above reporting requirements. But for the time being,
        • Any models that were trained with more than 10^26 flops
        • Any models that are trained primarily on biology data and trained using greater than 10^23 flops
        • Any datacenter that connected with greater than 100 gigabits per second
        • Any datacenter that can train an AI at 10^20 flops
    • c
      • I don’t know what this subsection is about. Something about protection cyber security for “United States Infrastructure as a Service” products.
      • This includes some tracking of when foreigners want to use US AI systems in ways that might pose a cyber-security risk, using standards identical to the ones laid out above.
    • d
      • More stuff about IaaS, and verifying the identity of foreigners.
  • Thoughts:
    • Do those numbers add up? It seems like if you’re worried about models that were trained on 10^26 flops in total, you should be worried about much smaller training speed thresholds than 10^20 flops per second? 10^19 flops per second, would allow you to train a 10^26 model in 115 days, e.g. about 4 months. Those standards don’t seem consistent.
    • What do I think about this overall?
      • I mean, I guess reporting this stuff to the government is a good stepping stone for more radical action, but it depends on what the government decides to do with the reported info.
      • The thresholds match those that I’ve seen in strategy documents of people that I respect, so that that seems promising. My understanding is that 10^26 FLOPS is about 1-2 orders of magnitude larger than our current biggest models.
      • The interest in red-teaming is promising, but again it depends on the implementation details.
        • I’m very curious about “launching an initiative to create guidance and benchmarks for evaluating and auditing AI capabilities, with a focus on capabilities through which AI could cause harm, such as in the areas of cybersecurity and biosecurity.”
          • What will concretely happen in the world as a result of “an initiative”? Does that mean allocating funding to orgs doing this kind of work? Does it mean setting up some kind of government agency like NIST to…invent benchmarks?

4.3

  • Summary:
    • They want to protect against AI cyber-security attacks. Mostly this entails government agencies issuing reports.
      • a – Some actions aimed at protecting “critical infrastructure” (whatever that means).
        • Heads of major agencies need to provide an annual report to the Secretary of Homeland security on potential ways that AIs open vulnerabilities to critical infrastructure in their purview.
        • “…The Secretary of the Treasury shall issue a public report on best practices for financial institutions to manage AI-specific cybersecurity risks.”
        • Government orgs will incorporate some new guidelines.
        • The secretary of homeland security will work with government agencies to mandate guidelines.
        • Homeland security will make an advisory committee to “provide to the Secretary of Homeland Security and the Federal Government’s critical infrastructure community advice, information, or recommendations for improving security, resilience, and incident response related to AI usage in critical infrastructure.”
      • b – Using AI to improve cybersecurity
        • One piece that is interesting in that: “the Secretary of Defense and the Secretary of Homeland Security shall…each develop plans for, conduct, and complete an operational pilot project to identify, develop, test, evaluate, and deploy AI capabilities, such as large-language models, to aid in the discovery and remediation of vulnerabilities in critical United States Government software, systems, and networks”, and then report on their results
  • Commentary
    • This is mostly about issuing reports, and guidelines. I have little idea if any of that is real or if this is just an expansion of lost-purpose bureaucracy. My guess is that there will be few people in the systems that have inside views that allow them to write good guidelines for their domains of responsibility regarding AI, and mostly these reports will be epistemically conservative and defensible, with a lot of “X is possibly a risk” where the authors have large uncertainty about how large the risk is.
    • Trying to use AI to improve cyber security sure is interesting. I hope that they can pull that off. It seems like one of the things that ~ needs to happen for the world to end up in a good equilibrium is for computer security to get a lot better. Otherwise anyone developing a powerful model will have the weights stolen, and there’s a really vulnerable vector of attack for not-even-very-capable AI systems. I think the best hope for that is using our AI systems to shore up computer security defense, and hope that at higher-than-human levels of competence, cyber warfare is not so offense-dominant. (As an example, someone suggested maybe using AI to write a secure successor to C, and the using AI to “swap out” the lower layers of our computing stacks with that more secure low level language.)
      • Could that possibly happen in government? I generally expect that private companies would be way more competent at this kind of technical research, but maybe the NSA is a notable and important exception? If they’re able to stay ten years ahead in cryptography, maybe they can stay 10 years ahead in AI cyberdefense.
        • This raises the question, what advantage allows the NSA to stay 10 years ahead? I assume that it is a combination of being able to recruit top talent, and that there are things that they are allowed to do that would be illegal for anyone else. But I don’t actually know if that’s true.

4.4 – For reducing AI-mediated CHEMICAL, BIOLOGICAL, RADIOLOGICAL, AND NUCLEAR threats, focusing on biological weapons in particular.

  • Summary:
    • a
      • The Secretary of Homeland Security (with help from other executive departments) will “evaluate” the potential of AI to both increase and to defend against these threats. This entails talking with experts and then submitting a report to the president.
      • In particular, it orders the Secretary of Defense (with the help of some other governmental agencies) to conduct a study that “assesses the ways in which AI can increase biosecurity risks, including risks from generative AI models trained on biological data, and makes recommendations on how to mitigate these risks”, evaluates the risks associated with the biology datasets used to train such systems, assesses ways to use AI to reduces biosecurity risks.
    • b – Specifically to reduce risks from synthetic DNA and RNA.
      • The office of science and technology policy (with the help of other executive departments) are going to develop a “framework” for synthetic DNA/RNA companies to “implement procurement and screening mechanisms”. This entails developing “criteria and mechanisms” for identifying dangerous nucleotide sequences, and establishing mechanism for doing at-scale screening of synthetic nucleotides.
      • Once such a framework is in place, all (government?) funding agencies that fund life science research will make compliance with that framework a condition of funding.
      • All of this, once set up, needs to be evaluated and stress tested, and then a report sent to the relevant agencies.
  • Commentary:
    • The part about setting up a framework for mandatory screening of nucleotide sequences, seems non-fake. Or at least it is doing more than commissioning assessments and reports.
      • And it seems like a great idea to me! Even aside from AI concerns, my understanding is that the manufacture synthetic DNA is one major vector of biorisk. If you can effectively identify dangerous nucleotide sequences (and that is the part that seems most suspicious to me), this is one of the few obvious places to enforce strong legal requirements. These are not (yet) legal requirements, but making this a condition of funding seems like a great step.

4.5

  • Summary
    • Aims to increase the general ability for identifying AI generated content, and mark all Federal AI generated content as such.
    • a
      • The secretary of commerce will produce a report on the current and likely-future methods for, authenticating non-AI content, identifying AI content, watermarking AI content, preventing AI systems from “producing child sexual abuse material or producing non-consensual intimate imagery of real individuals (to include intimate digital depictions of the body or body parts of an identifiable individual)”
    • b
      • Using that report, the Secretary of Commerce will develop guidelines for detecting and authenticating AI content.
    • c
      • Those guidelines will be issued to relevant federal agencies
    • d
      • Possibly those guidelines will be folded into the Federal Acquisitions Regulation (whatever that is)
  • Commentary
    • Seems generally good to be able to distinguish between AI generated material and non-AI generated material. I’m not sure if this process will turn up anything real that meaningfully impacts anyone’s experience of communications from the government.

4.6

  • Summary
    • The Secretary of Commerce is responsible for running a “consultation process on potential risks, benefits, other implications” of open source foundation models, and then for submitting a report to the president on the results.
  • Commentary
    • More assessments and reports.
    • This does tell me that someone in the executive department has gotten the memo that open source models mean that it is easy to remove the safeguards that companies try to put in them.

4.7

  • Summary
    • Some stuff about federal data that might be used to train AI Systems. It seems like they want to restrict the data that might enable CBRN weapons or cyberattacks, but otherwise make the data public?
  • Commentary
    • I think I don’t care very much about this?

4.8

  • Summary
    • This orders a National Security Memorandum on AI to be submitted to the president. This memorandum is supposed to “provide guidance to the Department of Defense, other relevant agencies”
  • Commentary:
    • I don’t think that I care about this?

Section 5 – Promoting Innovation and Competition.

5.1 – Attracting AI Talent to the United States.

  • Summary
    • This looks like a bunch of stuff to make it easier for foreign workers with AI relevant expertise to get visas, and to otherwise make it easy for them to come to, live in, work in, and stay in, the US.
  • Commentary
    • I don’t know the sign of this.
    • Do we want AI talent to be concentrated in one country?
      • On the one hand that seems like it accelerates timelines some, especially if there are 99.9% top tier AI researchers that wouldn’t otherwise be able to get visas, but who can now work at OpenAI. (It would surprise me if this is the case? Those people should all be able to get O1 visas, right?)
      • On the other hand, the more AI talent is concentrated in one country the smaller jurisdiction of the regulatory regime that slows down AI. If enough of the AI talent is in the US, regulations that slow down AI development in the US only have a substantial impact, at least the the short term, before that talent moves, but maybe also in the long term, if researchers care more about continuing to live in the US than they do about making cutting edge AI progress.

5.2

  • Summary
    • a –
      • The director of the NSF will do a bunch of things to spur AI research.
        • …”launch a pilot program implementing the National AI Research Resource (NAIRR)”. This is evidently something that is intended to boost AI research, but I’m not clear on what it is or what it does.
        • …”fund and launch at least one NSF Regional Innovation Engine that prioritizes AI-related work, such as AI-related research, societal, or workforce needs.”
        • …”establish at least four new National AI Research Institutes, in addition to the 25 currently funded as of the date of this order.”
    • b –
      • The Secretary of Energy will make a pilot program for training AI scientists.
    • c –
      • Under Secretary of Commerce for Intellectual Property and Director of the United States Patent and Trademark Office will sort out how generative AI should impact patents, and issue guidance. There will be some similar stuff for copyright.
    • d –
      • Secretary of Homeland Security “shall develop a training, analysis, and evaluation program to mitigate AI-related IP risks”
    • e –
      • The HHS will prioritize grant-making to AI initiatives.
    • f –
      • Something for the veterans.
    • g –
      • Something for climate change
  • Commentary
    • Again. I don’t know how fake this is. My guess is not that fake? There will be a bunch of funding for AI stuff, from the public sector, in the next two years.
    • Most of this seems like random political stuff.

5.3 – Promoting Competition.

  • Summary
    • a –
      • The heads of various departments are supposed to promote competition in AI, including in the inputs to AI (NVIDIA)?
    • b
      • The Secretary of Commerce is going to incentivize competition in the semi-conductor industry, via a bunch of methods including
        • “implementing a flexible membership structure for the National Semiconductor Technology Center that attracts all parts of the semiconductor and microelectronics ecosystem”
        • mentorship programs
        • Increasing the resources available to startups (including datasets)
        • Increasing the funding to R&D for superconductors
    • c – The Administrator of the Small Business Administration will support small businesses innovating and commercializing AI
    • d
  • Commentary
    • This is a lot of stuff. I don’t know that any of it will really impact how many major players there are at the frontier of AI in 2 years.
    • My guess is probably not much. I don’t think the government knows to to create NVIDAs or OpenAIs.
    • What the government can do is break up monopolies, but they’re not doing that here.

My high level takeaways

Mostly, this executive order doesn’t seem to push for much object-level action. Mostly it orders a bunch of assessments to be done, and reports on those assessments to be written, and then passed up to the president.

My best guess is that this is basically an improvement?

I expect something like the following to happen:

  • The relevant department heads talk with a bunch of experts. 
  • The write up very epistemically conservative reports in which they say “we’re pretty sure that our current models in early 2024 can’t help with making bioweapons, but we don’t know (and can’t really know) what capabilities future systems will have, and therefore can’t really know what risk they’ll pose.”
  • The sitting president will then be weighing those unknown levels of national security risks against obvious economic gains and competition with China.

In general, this executive order means that the Executive branch is paying attention. That seems, for now, pretty good. 

(Though I do remember in 2015 how excited and optimistic people in the rationality community were about Elon Musk, “paying attention”, and that ended with him founding OpenAI, what many of those folks consider to be the worst thing that anyone had ever done to date. FTX looked like a huge success worthy of pride, until it turned out that it was a damaging and unethical fraud. I’ve become much more circumspect about which things are wins, especially wins of the form “powerful people are paying attention”.)

Some thoughts on Effective Altruism – Two memes

Perhaps the most important thing to understand about EA is that, from the beginning, it was composed of two entwined memes.

Charity is inefficient

The first of these memes was that by using reason and evidence you could do much more good, with charity, than the default

In the early 2010s, there were some nerds on the internet (LessWrong and GiveWell and some other places) writing about, among other things, how to optimize charitable giving.

It seemed like the charity sector, in general, was not very efficient. With careful thought and research you could easily outcompete the charity sector as a whole, making investments that are orders of magnitude more effective than donating with a typical amount of thought.

The basic claim here is about competence: by being more thoughtful and rational, by doing research, we can outcompete a charitable industry that isn’t trying very hard, at least by our standards.

Normal people can save lives

The second meme was that a person with relatively small life-style changes, a normal person in the first world could do a shocking level of good, and it is good for people to do that.

My guess is that this meme started with Peter Singer’s drowning child paper, which argued that by spending tiny amounts of money and giving up some small creature comforts, one could literally save lives in the third world. And given that, it’s really really good when people decide to live that way; the more people who do, the more lives are saved. The more resources that are committed to these problems more good we can do.

This basic meme became the seed of Giving What We Can, for instance.

Note that in its initial form, this was something like a redistributive argument: We have so much more stuff than others who are in dire need that there’s a moral pressure (if not an obligation) to push for wealth transfers that help address the difference.

To summarize the difference between these ideas: One of these is about using the altruistic budget more effectively, and the other is about expanding the size of the altruistic budget.

Synergies

These memes are obviously related. They’re both claims about the outsized impact one can have via charitable donation (and career prioritization).

The claim that a first-worlder can save lives relatively cheaply means that there’s not an efficient market in saving lives.

And in my experience at least, the kind of analytical person that is inclined to think “One charity is going to be the most effective one. Which one is that?” also tends to be the kind of person that thinks, at some point in their life, “There are people in need who are just as real as I am, and the money I’m spending could be spent on helping them. I should give most of my money away.” There’s an “EA type”, and both of these memes are appealing to that type of person.

So in some ways these two memes are synergistic. Indeed, they’re synergistic enough that they fused together into the core of the Effective Altruism movement.

Tensions

But, as it turns out, those memes are also in tension with each other in important respects, in ways that played out in cultural conflicts inside of EA, as it grew.

These two memes encode importantly different worldviews, which have different beliefs about the constraints on doing more good in the world. To make an analogy to AI alignment, one of these views holds that the limiting factor is capability, how much of an influence we you can have on the world, and the other holds that the limiting factor is steering, the epistemology to identify outsized philanthropic opportunities.

Obviously, both of these can be constraints on your operation, and doing the most good will entail finding an optimal point on the pareto frontier of their tradeoff.

Implications of a “resources moved” frame

If the operating model is that the huge philanthropic gains are primarily redistributive, the primary limiting factor is the quantitative amount of resources moved. That tends to imply…

The Effective Altruism community that wants to be a mass-movement 

If the good part of EA is more people deciding to commit more of their resources to effective charities, you want to convert as many people as possible. One high priority is convincing as many people as possible to become EAs.  

Indeed this was the takeaway from Peter Singer’s TED talk in all the way back in 2013: You can do more good by donating to effective charities than you can by being a doctor (a generally regarded way of “helping people”), but you can do even better than that by converting more people to EA.

Branding matters

If you’re wanting to spread that core message, to get more people to donate more of their resources to effective charities, there’s incentive to reify the EA brand, to be the sort of thing that people can join, rather than an ad hoc collection of ideas and bloggers.

And branding is really important. If you want lots of people to join your movement, it matters a lot if the general public perception of your movement is positive.

Implications of a “calling the outsized opportunities frame”

In contrast, if you’re operating on a model where the philanthropic gains are the result of doing better and more careful analysis, the limiting constraint on the project is not (necessarily) getting more material resources to direct, but the epistemic capability to correctly pinpoint good opportunities. This tends to imply…

Unusually high standards of honesty and transparency are paramount

If you’re engaged in the intellectual project of trying to figure out how the world works, and how to figure out which interventions make things better, it is an indispensable feature of your epistemic community that it has strict honesty norms, that are firmly in Simulacrum level 1.  

You need to have expectations for what counts as honest that are closer to those of the scientific community, compared to the standards of marketing. 

We might take for granted that in many facets of the world, people are not ever really trying to be accurate (when making small talk or crafting slogans) and that people and organizations will put their best foot forward, highlighting their success and quietly downplaying failures.

But that normal behavior is counter to a collective epistemic process of putting forward and critiquing ideas, and learning from (failed) attempts to do stuff.

Furthermore, if you have a worldview that holds that the charity sector is incredibly inefficient, you’re apt to ask why that is. And part of the answer is that this kind of normal “covering one’s ass” / “putting forward a good face” behavior kills the accounting that causes charities to be effective in their missions. This background tends to make people more paranoid about these effects in their “let’s try to outcompete the charity sector” community.

“More people” is not an unadulterated good

Adding more people to a conversation does not, in the typical case, make the reasoning of that conversation more rigorous.

A small community of bloggers and intellectuals engaged in an extended conversation, aiming make progress together on some questions about how to most effectively get altruistic gains at scale, doesn’t necessarily benefit from more participants. 

And it definitely doesn’t benefit from the indiscriminate addition of participants. A small number of people will contribute to the extended conversation more than the communication and upfront enculturation cost that each new person imposes. 

These two world views give rise to two impulses in the egregore of EA: the faction that is in favor of professionalism  and the faction in vocal support of epistemics and integrity.

We see this play out everytime someone posts something arguably unseemly on the EA forum, and someone comments, “I think it was bad to post this, it makes EA look bad and has a negative impact in expectation.”

And I think the tension between these impulses goes a long way towards explaining why EA seems so much less promising to me now than it did 5 years ago.
I have more to say here, about how the incentive to do the hard work of rigorous thinking things through and verifying lines of argument is somewhat self-cannibalizing, but I don’t feel like writing that all right now, so I’m shipping this as a post.

Request for parellel conditional-market functionality

In responses to James’ plan for a manifold dating site, I just wrote tho following comment.

I think this needs a new kind of market UI, to setup multiple conditional markets in parallel. I think this would be useful in general, and also the natural way to do romance markets, in particular.

What I want is for a user to be able to create a master (conditional) prompt that includes a blank to be filled in. eg “If I go on a date with ____, will we end up in a relationship 2 months later?” or “If I read ____ physics textbook, will I be impressed with it?” or “Will I think the restaurant ____, is better than the Butchers Son, if I eat there?” The creator of this master question can include resolution details in the description, as always.

Then other users can come and submit specific values of for the blank. In these cases, they suggest people, physics textbooks, or restaurants.

However (and this is the key thing that makes this market different from the existing multiple choice markets), every suggestion becomes it’s own market. Each suggestion gets a price between 100% and 0%, rather than all of the suggestions in total having adding up to a probability of 100%.

After all, it’s totally possible that someone would end up in a relationship with Jenny (if they end up going on a date with Jenny) and end up in a relationship with George (if they go on a date with George). And it’s likely that there there are multiple restaurants that one would like better than the Butcher’s Son. There’s no constraint that all the answers have to sum to 100%.

(There are other existing markets that would make more sense with this format. Aella’s one-night stand market for one, or this one about leading AI labs. It’s pretty common for multiple choice questions to not need to sum to 100% probability, because multiple answers can be correct.)

Currently, you can create a bunch of conditional markets yourself. But that doesn’t work well for romance markets in particular, for two reasons.

1. Most of the value of these markets is in discovery. Are there people who the market thinks that I should go on a date with, who I’ve never met?
2. It is very socially forward to create a market “Will I be in a relationship with [Jenny], if we go on one date?” That means that I reveal that I’m thinking about Jenny enough to make a market, to Jenny, and to all the onlookers, which could be embarrassing to her. It’s important that the pairings are suggested by other people, and mixed in with a bunch of other suggestions, instead of highlighted in a single top-level market. Otherwise it seems like this is pushing someone’s personal life into a public limelight too much.

If this kind of market UI existed, I would immediately create a “If Eli goes on a date with ____, will the be in a relationship 3 months later?”, and a link to my existing dating doc, and a large subsidy (we’d have to think about to allocate subsidies across all the markets in a set).

In fact, if it were possible and legal to do this with real money, I would probably prefer spending $10,000 subsidizing a set of real money prediction markets of this form, compared to would spending $10,000 on a matchmaker. I just expect the market (and especially “the market” when it is composed of people who are one or two degrees removed from people that I might like to date), to be much better at suggesting successful pairings.

A letter to my 20 year old self

If I could send some advice back in time, to myself when I was 20 years old, this is a lot of what I would say. I think almost all of this is very idiosyncratic to me, and the errors that I, personally, am inclined towards. I don’t think that most 20 year olds that are not me should take these points particularly seriously, unless they recognize themselves in it.

[See also First conclusions from reflections on my life]

  1. Order your learning

You want to learn all skills, or at least all the awesome and useful ones. This is completely legitimate. Don’t let anyone tell you that you shouldn’t aim for that (including with words like “specialization” or “comparative advantage”.)

But because of this, every time you encounter something awesome, you respond by planning to make the practice of it part of your life in the short term. This is a mistake. Learning most things will require either intense bouts of focusing on only that one thing for (at least small numbers of) days at a time, or consistent effort over weeks or months. 

If every time you encounter some skill that seems awesome or important, you resolve to learn it, this dilutes your focus, which ends up with you not learning very much at all. Putting a surge of effort into something and then not coming back to it for some weeks is almost a total waste of that effort—you’ll learn almost nothing permanent from that.

The name of the game is efficiency. You should think of it like this:

Your skill and knowledge, at any given time, represents a small volume in a high dimensional space. Ultimately you want to expand in all or almost all directions. There’s no skill that you don’t want, eventually. But the space is very high dimensional and infinite, so trying to learn everything that crosses your path won’t serve you that well. You want to order your learning.

Your goal should be to plot a path, a series of expansions in this high dimensional space, that results in expanding the volume as quickly as possible. Focus on learning the things that will make it easier and faster to continue to expand, along the other dimensions, instead of focusing on whatever seems cool or salient in the moment.

[added:] More specifically, you should be willing to focus on doing one thing at a time (or one main thing, with a one or at most, two side projects). Be willing to take on a project, ideally but not necessarily involving other people, and make it your full time job for at a month. You’ll learn more and make more progress when you’re not dividing your efforts. You won’t loose nearly as much time in the switching costs, because you won’t have to decide what to do next: there will be a clear default. And if you’re focusing on one project at a time, it’s much easier to see if you’re making progress. You’ll be able to tell much faster if you’re spinning your wheels doing something that feels productive, but isn’t actually building anything. Being able to tell that you failed at a timeboxed goal means that you can notice and adapt.

A month might feel like a long time, to put aside all the other things you want to learn, but it’s not very long in the grand scheme of things. There have been many months since I was 20, and I would be stronger now, if I had spent more of them pushing hard on some specific goal, instead of trying to do many good things and scattering my focus.

You want to be a polymath; but the way to polymathy is not trying to do everything all at once: it’s mostly going intensely, on several different things, in sequence.

  1. Learn technical skills

In particular, prioritize technical skills. They’re easier to learn earlier in life, and I wish I had a stronger grounding in them now.

First and foremost, learn to program. Being able to automate processes, and build simple software tools for yourself is a superpower. And it is a really great source of money.

Then, learn calculus, linear algebra, differential equations, microeconomics, statistics, probability theory, machine learning, information theory, and basic physics. [Note that I’ve so far only learned some of these myself, so I am guessing at their utility].

It would be a good use of your time if you dropped everything else and made your only priority in the first quarter of college to do well in IBL calculus. This would be hard, but I think you would make substantial steps towards mathematical maturity if you did that.

In general, don’t bother with anything else in college, except learning technical subjects. I didn’t find much in the way of friends or connections there, and you’ll learn the non-technical stuff fine on your own.

The best way to learn these is to get a tutor, and walk through the material with the tutor on as regular a basis as you can afford.

  1. Prioritize money 

You’re not that interested in money. You feel that you don’t need much in the way of “stuff” to have an awesome life. You’re correct about that. Much more than most of the people around you, you don’t want or need “nice things”. You’re right to devalue that sort of thing. You’ll be inclined to live frugally, and that has served me very well.

However, you’re missing that money can be converted into learning. Having tens or hundreds of thousands of dollars is extraordinarily helpful for learning pretty much anything you care to learn. If nothing else, most subjects can be learned much faster by talking with a tutor. When you have money, if there’s anything you want to learn, you can just hire someone who knows it to teach you how to do it, or to do it with you. This is an overpowered strategy.

It is a priority for you to get to the point that you’re making (or have saved) enough money that you feel comfortable spending hundreds of dollars on a learning project.

Combining 1, 2, 3, the thing that I recommend that you do now is drop almost everything and learn to become a good programmer. Your only goal for the next few months should be 1) to have enough money for rent and food, and 2) to become a good enough programmer that you can get hired for it, as quickly as you can. Possibly the best way to do this is to do a coding boot camp, instead of self-teaching. You should be willing to put aside other cool things that you want to do and learn, for only a couple of months, to do this.

Then get a job as a software engineer. You should be able to earn small hundreds of thousands of dollars a year with a job like that, while still having time to do other stuff you care about in your off hours. If you live frugally, you can work for 2.5 years and come away with a small, but large enough (eg >100k) nest egg for funding all the other skills that you want to learn.

(If you’re still in college, staying to do IBL first, and then focusing on learning programing isn’t a bad idea. It might be harder to get mathematical maturity, in particular, outside of college.)

  1. Make things / always have a deliverable

I’ve gained much much more skill over the course of projects where I was just trying to do something, than from the sum of all my explicit learning projects. Mostly you learn skills as a side effect of doing things. This just works better than explicit learning projects. 

This also means that you end up learning real skills, instead of the skills that seem abstractly useful or cool from the outside, many of which turn out to have not much relevance to real problems. Which is fine; you can pursue things because they’re cool. But very often, what is most useful and relevant are pieces that are too mundane to come to mind, and doing real things reveals them. Don’t trust your abstract model of what elements are useful or relevant or important or powerful, too much. Better to let your learning be shaped to the territory directly, in the course of trying to do specific things.

The best way to learn is to just try to do something that you’re invested in, for other reasons, and learn what you need to know to succeed along the way. Find some software that you wish existed, that you think would be useful to you, and just try and build it. Run a conference. Take some work project that seems interesting and knock it out of the park. 

Try to learn as much as you can this way.

In contrast, I’ve spent a huge amount of time thinking over the years that didn’t create any value at all. If I learned something at the time, I soon forgot it, and it is completely lost ot me now. This is a massive waste. 

So your projects should always have deliverables. Don’t let yourself finish or drop a project, especially a learning project, until you have produced some kind of deliverable. 

A youtube video of yourself explaining some new math concept. A lecture for two friends. Using a therapy technique with a real client.

A blog post jotting down what you learned, or summarizing your thoughts on a domain is the minimum viable deliverable. If nothing else, write a blog post for everything that you spend time on, to capture the the value of your thinking for others, and for yourself later.

Don’t wait to create a full product at the end. Ship early, ship often. Create intermediate deliverables, capturing your intermediate progress, at least once a day. Write / present about your current thoughts and understanding, including your open confusions. (I’ve often gotten more clarity about something in the process of writing up my confusions in a blog post).

The deliverable can be very rough. But it shouldn’t be just your personal notes. If you’re writing a rough blog post, write it as if for an audience beyond yourself. That will force you to clarify your thoughts and clearly articulate the context much more than writing a personal journal entry. In my experience, the blog posts that I write like this are usually more helpful for my future self than the personal journal entries are.

The rule should be that someone other than you, in principle, could get value from it. A blog post or a recorded lecture, that no one reads, but someone could read and find interesting counts. The same thing, but on a private google drive, doesn’t count. (Even better, though, is if you find just one person who actually gets value out of it. Make things that provide value to someone else.)

Relatedly, when you have an idea for a post or an essay, write it up immediately, while the ideas are alive and energizing. If you wait, they’ll go stale and it is often very hard to get them back. There are lots of thoughts and ideas that I’ve had, which are lost forever because I opted to wait a bit on writing them down. This post is itself the result of some thoughts that I had while listening to a podcast, which I made a point to right up while the thoughts were alive in me.

  1. Do the simple thing first

You’re going to have many clever ideas for how to do things better than the default. I absolutely do not want to discourage you in that.

But it will behoove you to start, by doing the mundane, simple thing. Try the default first, then do optimizations and experiments on top of that, and feel free to deviate from the default, when you find something better.

If you have some fancy idea for how to use spaced repetition systems to improve your study efficiency, absolutely try that. But start by doing the simple thing of sitting down, reading the textbook, and doing the exercises, and then apply your fancy idea on top of that.

You want to get a baseline to compare against. And oftentimes, clever tricks are less important than just putting in the hours doing the work, and so you want to make sure to get started doing the work as soon as possible, instead of postponing it until after you’ve developed a clever system. Even if your system is legitimately clever, if the most important thing is doing the hard work, you’ll wish you started earlier.

You’re sometimes going to be more ambitious than the structures around you expect of you. That’s valid. But start with the smaller goals that they offer, and exceed them, instead of trying to exceed them in one fell swoop.

When you were taking Hebrew in high school, you were unimpressed by the standards of the class and held yourself higher than them. For the first assignment, you were to learn the first list of vocabulary words from the book, for the next week. But you felt that you were better than that, and resolved to study all the vocab in the whole book (or at least a lot of it) in that period, instead.

But that was biting off more than you could easily chew, and (if I remember correctly), when you came back the next week, you had not actually mastered the first vocab list. You would have done better to study that list first, and then move on to the rest, even if you were going to study more than was required.

I’ve fallen into this trap more than once. “Optimizing” my “productivity” with a bunch of clever hacks, or ambitious targets, which ultimately mask the fact that my output is underperforming very mundane work habits, for instance. 

You might want to work more and harder than most people, but start by sticking to a regular workday schedule, with a weekend, and then you can adjust it, or work more than that, from there.

Don’t fall into the trap of thinking that the simple thing that everyone else is doing is beneath you, since you’re doing a harder or bigger thing than that. Do the simple thing first, and then do more or better.

I’m sure there’s more to say, but this is what I was pressing on me last night in particular.

The meta-institutional bottleneck

The world is pretty awful, in a number of ways. More importantly, the world is, in an important way, undershooting its potential

Many of our problems are directly downstream of inadequate institutions, which incentivize those problems, or at least fail to incentivize their solutions. For virtually any part of the world that sucks, you can point to poor institution design that is giving rise to sub-adequate outcomes. [ 1 ]

For many of these broken institutions, there exist proposed alternatives. And for many of those alternatives, we have strong analytic arguments that they would improve or outright solve the relevant problems.

  • A Land Value Tax would likely solve many to most of the human problems of inequality (those that are often asserted to be a consequence of “capitalism”, such as many people working bullshit jobs that they hate, for their whole life, and still being too poor to have practical freedom)
  • There’s a laundry list of innovations that would solve the incentives problems of academia that lead to some large fraction of published research being wrong.
  • Prediction markets could give us demonstrably calibrated estimates of most questions of fact, that take into account all available info.
  • Approval voting is one of a number of voting systems that avoid the pitfalls of first-past-the-post elections. But we still use first-past-the-post and so we live with politicians that almost no one likes or supports.

The world is broken. But it largely isn’t for lack of ideas: we (or at least someone) know how to do better. The problem is that we are blocked on implementing those good ideas.

As Yudkosky puts it in Inadequate Equilibria…

Usually when we find trillion-dollar bills lying on the ground in real life, it’s a symptom of (1) a central-command bottleneck that nobody else is allowed to fix, as with the European Central Bank wrecking Europe, or (2) a system with enough moving parts that at least two parts are simultaneously broken, meaning that single actors cannot defy the system. To modify an old aphorism: usually, when things suck, it’s because they suck in a way that’s a Nash equilibrium.

But that doesn’t mean that there’s some fundamental law of nature that those solutions are untenable. If the proposed alternative is any good, it is also a stable equilibrium. If we moved to that world, we would probably stay there.

Or as Robin Hanson says of prediction markets in an interview,

I’d say if you look at the example of cost accounting, you can imagine a world where nobody does cost accounting. You say of your organization, “Let’s do cost accounting here.”

That’s a problem because you’d be heard as saying, “Somebody around here is stealing and we need to find out who.” So that might be discouraged.

In a world where everybody else does cost accounting, you say, “Let’s not do cost accounting here.” That will be heard as saying, “Could we steal and just not talk about it?” which will also seem negative.

Similarly, with prediction markets, you could imagine a world like ours where nobody does them, and then your proposing to do it will send a bad signal. You’re basically saying, “People are bullshitting around here. We need to find out who and get to the truth.”

But in a world where everybody was doing it, it would be similarly hard not to do it. If every project with a deadline had a betting market and you say, “Let’s not have a betting market on our project deadline,” you’d be basically saying, “We’re not going to make the deadline, folks. Can we just set that aside and not even talk about it?”

From a zoomed out view, we can see better incentive schemes that would give us more of what we want, that would allow us to run civilization better. But there’s no straightforward way to transition from our current systems to those better ones. We’re blocked by a combination of collective action problems and powerful stakeholders who benefit from the current equilibrium. So better institutional design remains theoretical.

One step meta

When I look at this problem, it strikes me that this is itself just another institution-design problem. Our norms, incentives, and mechanisms for moving from one equilibrium to another leave a lot to be desired. But, there could exist institutional setups that would solve or preempt the collective action and opposed-stakeholder problems that prevent us from transitioning out of inadequate equilibria. 

This isn’t something that we, as a species, know how to do, in actual practice. But it is something that we could know how to do. And if we did, that would let us build incentive-aligned institutions up to the limit of our theoretical knowledge of which institutions would produce the outcomes that we want. 

And this would, in turn, put the planet on track to adequacy for literally every other problem we face.

I call this the “meta-institutional bottleneck”. I offer it as a contender for the most important problem in the world.

Is this one thing or many?

It may be that this “meta-institutional bottleneck” is really just a name for the set of specific problems of moving from a specific inadequate equilibrium to its corresponding more adequate one. We have many such problems, but they’re different enough that we effectively have to solve each one on its own terms.

My guess is that that’s not true. I posit that there are enough commonalities in the difficulties to solving each inadequate equilibrium, that we can find at least semi-general solutions to those difficulties.

If I imagine someone attempting to intentionally push such equilibrium-switches, over a 60 year career, I imagine that they would learn things that generalize from one problem to the next. Surely there would be plenty of idiosyncratic details to each problem, but I imagine that there would also be common patterns for which one might develop generally applicable approaches.

I have not solved any such problems, at scale, so I’m only speculating. But some possible patterns / approaches that one might discover:

  • You might learn that there are a small number of people in a social network / prestige hierarchy who are disproportionately leveraged in switching the behavior of a whole network. If you can convince those people to change their behavior, the rest of the network will follow.
    • Certainly many of these people will be prestigious individuals that are easy to identify. But it could also be the case that there are high-leverage nodes that are not obviously powerful or prestigious, but who nevertheless end up being trend-setters in the network. There may or may not be an arbitrage opportunity in getting the people with the most leverage relative to their power/prestige to switch, and there may or may not be generalized methods for identifying those arbitrage opportunities.
  • You might learn that there’s almost always a question of how to compensate stakeholders who are beneficiaries of the current system. You want to run a process that allows the people who benefited from the inadequate institution to save face, or even take the credit for the switch over. Perhaps there are generalized mechanisms that have this property, 
  • You might discover the conditions under which dominance insurance contracts can work well, and what their practical limits are.
  • You might find that there are repeatable methods for building alternative institutions in parallel to legacy institutions, which gradually take on more and more of the functions of the legacy institution such that there’s an incentive gradient towards a better equilibrium at every step.

Or maybe nothing like any of these. Perhaps when one tries to solve these problems in practice, they learn that this whole analytic frame of incentives and game theory is less productive than some other model.

But whatever form it ends up taking, it seems like there ought to be knowledge to discover here. I guess that the meta-institutional bottleneck is meaningfully a “thing”, and not just a catch-all term for a bunch of mostly unrelated, mostly idiosyncratic problems.

And to the extent that that’s true, there’s room for a community of entrepreneur-researchers iteratively trying to solve this kind of problem, sharing what works and what doesn’t, and collectively iterating to best practices, in the same way that there are currently known best practices for starting a startup.

Only one hump

It seems likely that the meta-institutional bottleneck is a one time hurdle. If we’re able to grow past our current existing institutions, we may not be in the trap again.

For one thing, many of the better institutions that we want to install have built in mechanisms for iterating to even better institutions. If your society is regularly utilizing prediction markets to aggregate information, it becomes much easier to surface, validate, and create common knowledge of superior institution design. A society that is already using prediction and decision markets is much much better positioned to switch to something even better, if something like that is found. Our institutions are our meta-institutions.

But even aside from that, if we do discover general, repeatable, methods for moving from inadequate to adequate equilibria, we will forever be in a better situation for improving our institutions over the future. [ 2 ]

For instance, suppose that Balaji’s “network state” model proves successful, and in 25 years there are a handful of diplomatically recognized sovereign states that were founded in that way. This would probably be a great improvement on the object level, since I expect at least one of those states to incorporate modern institution design, demonstrating their effectiveness (or falsifying their effectiveness!) to the rest of the world. But even if that’s not the case, there would, in that future, be a known playbook by which groups of people could iteratively found sovereign states that conform to whatever ideals a large enough coalition buys into. 

We have a meta-institutional bottleneck in this generation, but if we solve it, in general, we may solve it forever. 

The long game

I guess that 60 to 100 years is enough time for a relatively small group of ~100 competent founder types to radically shift the incentives in society. 

This would be a slog, especially at first. But few individuals plan and build on decades-long time horizons. There’s an advantage to those that are playing a long game. 

And as some of these projects succeed, that would accelerate others. As noted above, there’s a flywheel effect where adequate information propagation and decision institutions disproportionately favor other adequate information propagation and decision institutions.

I could imagine a community of people who share this ideology. They would identify specific inadequate equilibria that could maybe be corrected with some combination of a clever plan and an annoying schlep. Individual founders would form small teams to execute a plan to try to solve that equilibrium. 

Some examples of the sorts of projects that we might push on:

  • Normalizing prediction markets
  • Attempting to found network states
  • Reforming existing government processes.
  • Reforming academia 
  • Building expertise identification and peer-review processes outside of academia
  • Building platforms that recreate the functionality of local governance (eg a community governance platform to evaluate and cast judgement on allegations of sexual harassment)
  • Build incentive-compatible outcome-tracking and ranking platforms (for medical professionals, for lawyers, etc.)
  • Generally doing the schlep to kickstarter every change that looks like a collective action problem.

Those teams would often fail, and write up post-mortems for the community to engage with. And occasionally they would succeed, to the point of pushing society over a threshold into some new stable equilibrium. In the process, they would find processes and build software tools for solving a specific problem, and then share them with the broader community, to adapt them to other problems.

Over time we’ll get a clearer sense of what common blockers to these institutional shifts are, and then figure out approaches to those blockers. For instance, as Robin Hanson has found, there isn’t actually demand for accurate information from prediction markets, because accurate information interferes with narrative-setting. And in general, most individuals don’t locally prioritize better outcomes over signaling their good qualities, doing the “done thing”, and avoiding blame. 

Noticing these blockers, we treat them as constraints on our exploration process. We need to find institutional reform paths that are themselves compatible with local incentives, just as a technologist steers towards a long term vision by tacking close to what is currently technologically feasible. And it turns out that the local incentives are different than what one might have naively assumed. Metamed’s failure and Robin’s non-traction on prediction markets both represent progress.  [ 3 ]

We would be playing a decades-long game, which means that a crucial consideration would be building and maintaining a lineage that retains essential skill and strategic orientation, over several generations.

Over the course of a century, I think we could plausibly get to a society on planet earth whose core institutions are basically adequate. And from there, the world would be mostly optimized.

But…AI

The problem, of course, is that it doesn’t seem that we have even 30 years until the world is radically transformed and control of the future slips out of human hands. (Because of AI.)

This is what I would be spending my life on, if not for the rapidly approaching singularity.

I would prefer to be pushing on the meta-institutional bottleneck. It seems more rewarding, and more tractable, and a better fit for my skills and disposition. 

But I don’t think I live in that world. 

It’s possible that I should still be doing this. If the AI situation looks intractable enough, maybe the most dignity is in making peace with death, and then backing up and starting to build the kind of sane civilization that would have handled the problem well, if we had been faster. 


[ 1 ] – Some of the suckiness of the world is due to currently unavoidable circumstances, diseases like aging and cancer for instance suck. But even those problems are touched by our half-functional institutions. It is not currently the case that a given cancer patient gets the best treatment our planet could in principle afford given our current level of technology. If the norms of medical research were better, or even if the medical literature was better indexed, it would be clearer which treatments worked best. If medical professional’s outcomes were tracked, and scored, it would be possible to buy provably better treatment, which would incentivize diffusion of innovation. And so on.

[ 2 ] – This isn’t literally guaranteed. It could of course be the case that we discover methods that work well in our current world, but that our world changes radically enough that our tools and methods stop being applicable. 

It’s not out of the question that a few network states are successfully founded, before legacy states wise up and coordinate to close that option.

[ 3 ] – Indeed, Robin Hanson’s decades long advocacy of prediction markets and MetaMed’s nascent attempt to reform medicine both seem to me like examples of the kinds of project that we might try to undertake. Neither succeeded, but both gave us more detail on the shape of the problem (noisily, since outcomes are very dependent on luck; a problem that destroyed a first attempt might be defeated by a second attempt).

I side with my psych bugs

[I wrote this something like 6 months ago]

  • I basically always side with my psych bugs. If I object to something in a psychy way, my attitude is generally, “I don’t understand you, or what you’re about, little psych bug, but I’m with you. I’m not going to let anyone or anything force us to do anything that you don’t want to do.”
  • Sometimes this is only one half of a stance, where the other half is a kind of clear-sighted sense of “but this isn’t actually the optimal behavior set, right? I could be doing better by my goals and values by doing something else?” But even in that case, my allyship of my psych bugs comes first.
  • I don’t throw myself under the bus just because I don’t understand the root of my own behavior patterns. And I don’t force myself to change just because I think I’ve understood. Deep changes to my psychology are almost always to be done from the inside out. The psychy part of me is the part that gets to choose if it wants to change. I might do CoZEs, but the psychy part of me gets to decide if it wants to do CoZEs and which ones to do.
  • Admittedly, there’s a bit of an art here. There are cases where it’s not a psychy objection, but simple fear of discomfort that is holding me back. Sometimes I’m shying away from doing another rep at the gym, or doing another problem on the problem set, not because I have a deep objection, but because it’s hard. And sometimes I don’t want to change some of my social behavior patterns because that would mean being outside my comfort zone, and so I’m avoiding it or rationalizing why not to change.
  • And for those I should just summon courage and take action anyway. (Though in that last example there, in particular, I want to summon compassion along with courage.)
  • There’s a tricky problem here of how to deal with psychy internal conflicts of various stripes. I don’t claim that this is the
  • I want to explain a bit of why I have this stance. There are basically 3 things that can go wrong if I do a simple override of a psych bug. These are kind of overlapping. I don’t think they’re really one thing or another.

1. I’m disintegrated and I take ineffective action.

  • If something feels really bad to do, in a psychy way, and I try to do it anyway, then I’m fighting myself. My behavior is in practice going to be spastic as I alternate between trying to do X and trying to do not X, or worse, try and do X and not X at the same time.
  • Recent example: I was interested in helping my friends with their CFAR workshops, but also thought that what they were doing was bad in an important way and incremental improvements to what they were doing would make things worse because I didn’t really trust the group. Even now, I’m struggling to put to paper exactly what the conflict was here.
  • So I ended up 1) not helping very much, because I was not full-heartedly committed to the project and 2) metaphorically stabbing myself in the foot as I repeatedly did things that seemed terrible to me in hard-to-explain ways.
  • And the way that I get integration is I own all my values, not giving up on any of them, and I work to get to the root of what each one is protecting, and to figure out how to satisfy all of them. It

2. I harm some part of my values.

  • My psych bugs are sort of like allergies to something, which are keeping me mored to the things that I care about. They flare up when it seems like the world is encroaching on something deeply important to me that I won’t be able to argue for or defend against.
  • Examples:
  • I was often triggered around some things with romance, and most people’s suggestions around romance. I sometimes expressed that it felt like other people were trying to gaslight me about what I want. There was something deep and important to me in this domain, and also it was fragile. I could loose track of it.
  • I have a deep aversion to doing drugs. (I once fainted from listening to some benign drug experiences, and have left academic lectures on LSD, because I felt like I was going to throw up). I haven’t explored this in detail, but something in me sees something about (some) drugs, and is like “nope. nope. nope. Critically bad. Unacceptable.” Probably something about maintaining my integrity as an agent and a thinking thing.
  • It is really not ok, in either of these cases to just overwrite myself here. Both of these “aversions” are keeping me mored to something that is deeply important to me, and that I do not want to loose, but which the world is exerting pressure against.
  • (The seed of a theory of triggeredness)

3. The territory strikes back.

  • Most importantly, sometimes the psychy part of me is smarter than me, or at least is managing to track some facet of reality that I’m not tracking, and if I just do an override, even if this worked in the sense of not putting me in a position where I’m fighting myself, reality will smack me in the face.
  • Some examples:
  • Most people hurt themselves / impair themselves in learning “social skills”, and the first commitment when learning “social skills” is to not damage yourself.
  • Note that that doesn’t mean that you should just do nothing. You still want to get stronger. But you want to do it in a way that doesn’t violate more important commitments.
  • One class of psych bug in me seems to be pointing at the Geeks, Mops, Sociopaths dynamic. I have miscalibrated (?) counter-reactions to anything that looks like letting in MOPs or Sociopaths into things that I care about.
  • Those are real risks! Bad things happen when you do that! Shutting my eyes to my sense of “something bad is happening here” doesn’t help me avoid the problem.
  • In practice, it’s usually more of blend of all three of these failure modes, and I don’t get to know from the outset how much of it is each. And also usually, there’s some overreaction or miscalibration mixed in. But, likewise, I don’t know from the outset what is miscalibration and what isn’t.
  • So, by default, when I have a psychy objection to something, I side with it, and defend it, while doing a dialog process, and trying to move closer to the pareto frontier of my values.