Author: elityre

That no one rebuilt old OkCupid updates me a lot about how much the startup world actually makes the world better

~ elityre ~ Leave a comment

The prevailing ideology of San Francisco, Silicon Valley, and the broader tech world, is that startups are an engine (maybe even the engine) that drives progress towards a future that’s better than the past, by creating new products that add value to people’s lives.

I now think this is true in a limited way. Software is eating the world, and lots of bureaucracy is being replaced by automation which is generally cheaper, faster, and a better UX. But I now think that this narrative is largely propaganda.

It’s been 8 years since Match bought and ruined OkCupid and no one, in the whole tech ecosystem, stepped up to make a dating app even as good as old OkC is a huge black mark against the whole SV ideology of technology changing the world for the better.

Finding a partner is such a huge, real, pain point for millions of people. The existing solutions are so bad and extractive. A good solution has already been demonstrated. And yet not a single competent founder wanted to solve that problem for planet earth, instead of doing something else, that (arguably) would have been more profitable. At minimum, someone could have forgone venture funding and built this as a cashflow business.

It’s true that this is a market that depends on economies of scale, because the quality of your product is proportional to the size of your matching pool. But I don’t buy that this is insurmountable. Just like with any startup, you start by serving a niche market really well, and then expand outward from there. (The first niche I would try for is by building an amazing match-making experience for female grad students at a particular top university. If you create a great experience for the women, the men will come, and I’d rather build an initial product for relatively smart customers. But there are dozens of niches one could try for.)

But it seems like no one tried to recreate OkC, much less creating something better, until the manifold team built manifold.love (currently in maintenance mode)? Not that no one succeeded. To my knowledge, no else one even tried. Possibly Luna counts, but I’ve heard through the grapevine that they spent substantial effort running giant parties, compared to actually developing and launching their product—from which I infer that they were not very serious. I’ve been looking for good dating apps. I think if a serious founder was trying seriously, I would have heard about it.

Thousands of funders a year, and no one?!

That’s such a massive failure, for almost a decade, that it suggests to me that the SV ideology of building things that make people’s lives better is broadly propaganda. The best founders might be relentlessly resourceful, but a tiny fraction of them seem to be motivated by creating value for the world, or this low hanging fruit wouldn’t have been left hanging for so long.

This is of course in addition to the long list of big tech companies who exploit their network-effect monopoly power to extract value from their users (often creating negative societal externalities in the process), more than creating value for them. But it’s a weaker update that there are some tech companies that do ethically dubious stuff, compared to the stronger update that there was no startup that took on this obvious, underserved, human problem.

My guess is that the tech world is a silo of competence (because competence is financially rewarded), but operates from an ideology with major distortions / blindspots, that are disconnected from commonsense reasoning about what’s Good. eg following profit incentives, and excitement about doing big things (independent from whether those good things have humane or inhumane impacts) off a cliff.

Moldbug’s insight

~ elityre ~ 1 Comment

I’ve been reading some of Curtis Yarvin’s work lately.

For the most part, he seems like a blowhard, and an incorrect blowhard at that. His general rhetorical approach seems to be to make bold assertions, dressed up in flowery and bombastic language, and then to flatter his reader for being in on the secret. When he’s on podcast interviews, mostly the hosts will agree with his premises, but occasionally he’ll make a claim that they reject and push back against. Then Yarvin is forced to defend his bold claims instead of just insinuating them, and often his actual argumentation comes off as pretty weak.

I get the feeling sometimes when reading his work of reading a high school essay, of the author reaching for arguments to defend a bottom line, decided for other reasons, rather than reporting the arguments and evidence that lead the author to believe the conclusion.1

He admits directly that he’s writing for fun, and occasionally talks about writing to troll people. I get the impression that his views were arrived at in part by a sincere intellectual investigation of history and political philosophy, and in part because they were fun (ie shocking) to advocate for in 2008. But now they’re a key part of Yarvin’s brand and he’s kind of stuck with them. As in academic philosophy, his incentives are towards doubling down on his distinctive ideas, regardless of their truth.)

His rhetorical style reminds me of that of Eliezer Yudkowsky and Nassim Taleb. All three of them have a deep knowledge of their subject matter and each writes with an arrogance / confidence in the correctness of his view and an insinuation that the reader, like him, understands some important truths not grasped by the masses of humanity. This style makes these authors fun to read, for some people, and insufferably annoying for other people.

My read, so far, is that if you don’t already buy into his basically aesthetic premises, his disgust for modernity and for progressivism in particular, he doesn’t have much in the way of good arguments for persuading you of his views. Perhaps the main thing that he does is open people’s eyes, allowing them to see through a hitherto completely unknown perspective that pierces through the civic propaganda of our time. Having seen through that perspective, perhaps some parts of the the world makes more sense. But that’s not because Moldbug made a strong case for his claims, so much as his rhetoric ensnared you in his wake, and pulled you along for a bit. (I’m very interested in Moldbug fans who disagree—especially those who’s mind was changed by some of his posts.)

That said, he does have a few important and novel-to-me analytical points.

Today, I think I grasped an important core of Yarvin’s political philosophy which I hadn’t previously understood, and which, not having understood, made many of his claims seem bizarre in their not-even-wrongness.

All of the following is a compression of my understanding of his view, and is not to be taken as an endorsement of that view.

Claim 1: Sovereignty is Conserved

This is almost a catchphrase for Yarvin. He uses it all over the place.

There is always some force or entity outside of and above the law. Every law is enforced by some process (otherwise it’s hardly a law). And the process that enforces the law must, necessarily, have the power to exempt itself from that law. If not, it wasn’t actually the system ultimately doing the enforcing. Sovereignty is “above-the-law-ness”, and it’s always conserved.2

As an intuition pump: there exists someone in the US government, who, if they decided to, could “disappear” you a (more or less) ordinary US citizen. Possibly the president could detain or assassinate a US citizen for no legible cause, and face no consequences. Possibly some specific people in the intelligences services, as well. If there’s no one person who could do it, there’s surely a consortium of people that, working in concert, could. (That sovereignty is conserved doesn’t mean that it’s always concentrated). In the limit, the whole of the US military must be above the law, because if it decided to, in a coordinated way, it could trivially overturn any law, or the whole governmental system for that matter. [More on that possibility later.]

Even if no specific individual is above the law, the government as a whole sure as hell is. “The government” can, fundamentally, do whatever it “wants”.

This is explicitly counter to an ideal enlightenment philosophy—that of equality before the law. That no person, no mater how powerful, is exempt from the same basic legal standards.

Moldbug asserts that any claim to equality above the law is horseshit. Sovereignty is conserved. Power is real, and it bottoms out somewhere, and wherever it bottoms out is always going to be above the law.

This isn’t a law of physics, but it is a law of nature—at least as inescapable as the logic of supply and demand, or natural selection. 3

Because of his rhetoric and politics, it’s easy to read Moldbug as not caring at all about the inequities of power. This is somewhat of a misunderstanding. It’s a non-question for Yarvin whether it’s good or desirable that sovereignty is conserved. It’s just a fact of life that power is going to ground out somewhere. Whether we consider that inhumane or prefer that it was otherwise is of no more relevance that if we wished perpetual motion was possible. It’s not possible, and it’s not possible for a pretty fundamental reason.4

But as a society, we’re are intent on deluding ourselves about the nature of power. That might cause problems, in roughly the way it might if we insisted on deluding ourselves about the efficacy of perpetual motion machines.

Claim 2: The profit motive + competition is a stronger guarantee than ideology

So there’s always some entity outside the law. But, one might think, given that sad reality, that its better to divide up that power as much as possible so that as few people as possible, and ideally no one, can unilaterally disappear people. Checks and balances, and limited powers, and so on, to prevent any individual or group in government, and the government as a whole from being too powerful. Perhaps we can’t abolish sovereignty, but dividing it up as much as possible and spreading it around seems like the the most humane way to deal with the unfortunate situation, right?

Yarvin is in favor of monarchy, so he says “no”. Why not?

Because, in practice, the less concentrated power is, the more it is effectively controlled by ideology rather than rational optimization for anyone’s interests.

This is the basic problem of voter incentives: The odds of any individual person’s vote shifting policy, and impacting that person’s life directly are so minuscule as to be irrelevant. The main impact that your voting behavior has on your life is through signaling: signaling to your peers and to yourself what kind of person you are. If your vote materially impacted your life through policy, you would be incentivized to carefully weigh the tradeoffs in every decision (or defer to trusted expert advisors). But if your vote is mostly about showing how compassionate you are, how committed you are to our shared values, carefully weighing tradeoffs doesn’t help you. Saying the most applause lights the fastest is what’s good for signaling.

As Bryan Caplan says “Markets do the good things that sound bad, and governments do bad things that sound good.”

The more power is divided up into tiny pieces the more it is steered by ideology instead of by self-interest. And rational self interest is much less dangerous than ideology.

As discussed, the US military could overthrow the US government and the US legal system, if it wanted to. Why doesn’t it do that? Because there’s a distributed common knowledge belief in “democracy“. Lots of people in the military sincerely believe in the democratic ideal, and even if they don’t, they believe that they believe they do, and everyone knows that everyone else would immediately oppose any attempts at an “undemocratic” military coup.

Which is to say that the thing standing between the US as it currently exists and a military dictatorship is an ideological commitment to “democracy”. This seems to have worked pretty well so far, but those scare quotes are pretty scary. If a sufficiently large faction of the military came to buy into an ideology that claimed to carry the torch of the true spirit of democracy (or Christianity, or Social Justice, or Communism, or enviornmentalism, or whatever moral ideal compels), that ideology would take over the US.

And similarly, to the extent that the US government is possessed by the spirit of Wokism, your country might suddenly become violently woke.

This isn’t a hypothetical. We’ve seen countries get possessed by Communist ideology and become violently Communist.

In contrast, consider if instead there was a single king/CEO, who has complete and total power over his domain, who controlled the military power. As long as he’s sane and competent (which has been a problem with historical monarchs but which Yarvin thinks is more-or-less solved as well as we can reasonably expect by the structure of a joint-stock corporation), this monarch would be acting from incentives that are much closer to rational self-interest, because he (and the shareholders of the joint-stock country) benefit(s) directly from the upside of actual actual policy outcomes, not just the social signaling benefits of his policies. He wants his realm to be safe and well-governed because that will increase the value of the real estate he owns, and he will make more money that way.

Especially so if he governs only one of hundreds of sovereign realms in a patchwork. In that case there’s competitive pressure to get policy right, and maintain rule of law. If he does a bad job of ruling, residents will leave to live somewhere else, taking their tax revenue with them.

This is not perfect. Any given king might be bad, just as any given CEO can be bad. There’s no guarantee that a king won’t be possessed by and ideology (it’s certainly happened before! Ferdinand II of the Holy Roman Empire and Alexander I of Russia, come to mind). But it’s better than the alternatives. Especially if the shareholders can remove a bad king from power and if there’s competition between sovereign realms, both of which introduce selection pressure for sane, self-interested kings.

It’s true that the sovereign could, by right, have any person in his realm who ticked him off quietly assassinated. But, realizing that sovereignty is conserved, that becomes less of a problem of monarchy in particular, and more of an inescapable problem of power in general, one which we obscure but don’t eliminate with limited governments of ostensive checks and balances.

Plus, assassinating people, even if you have the legal right to do it, is generally going to be bad for business—an erratic CEO doesn’t inspire the confidence that causes people to want to live in his realm. Enough shenanigans like that, and his sovereign corporation will start losing customers, and his shareholders will sell it’s stock and/or have him removed at CEO. And if the CEO is actually sovereign, that removes the strongest incentive on historical monarchs for having people assassinated: as a means of securing his power.5

But most importantly, a monarch-CEO is much much less likely than a democracy to get riled up and implement Communism. Communism is transparently bad for business, but sounds good (meaning it is a good way to signal your compassion or tribal loyalty). The incentives of CEOs leave them less vulnerable to takeover by parasitic ideologies compared to masses of people in democracies. And ideological revolutions and generic bad-but-sounds-good policy is the serious threat-model. The all-powerful CEO who has the legal and pragmatic power of life and death over you is just much less dangerous than a state controlled by competing ideologies, which might decide that doing massive harm (from burning down your cities in the name of black lives, to rounding up all the jews, sending your scientists to work camps) is morally obligatory, in a fit of runaway virtue-signaling.

And indeed, when there’s some political power in the hands of the people, a good strategy for an ambitious person seizing power is to craft or adapt an ideology that enflames the people’s passions, and empowers you personally. That’s what Hitler and Lenin did. When sovereignty is in the hands of shareholders and their CEO-delegate., ideologies are less adaptive for gaining power, and so less pervasive in the first place. But this is a separate thread of Modbugian philosophy: that democracy causes ideology, that’s less central to the point that CEO-kings operating under the constraints of the profit motive and market competition are less vulnerable to ideologies than democracies.

Given that we can’t escape power, the profit motive of a king is a much stronger guarantee of good outcomes than ideological commitment, because ideologies are crazy, or at least can turn crazy fast.

Once you have that attitude, the fact that sovereignty in our present era seems to bottom out in basically ideological institutions seems…rather concerning. Every time you read “democratically controlled” you might mentally replace it with “more or less controlled by at least one more-or-less insane egregor.”

When I think in these terms, Yarvin’s political philosophy clicks into place for me as a coherent take on the world.

I’m not sure if I buy it, overall.

I agree that we don’t have literal and complete equality before the law: there are elites who get special treatment, and there may be individuals in the system that can literally get away with murder (though my guess is that’s only true in pretty limited circumstances?). But the US social and legal system really is demonstrably more egalitarian, closer to the ideal of equality before the law, than the European aristocratic systems that proceeded it. And that seems like something to be justly proud.

I think he’s underselling separation of powers. It’s true that the government can do whatever it wants, but we’ve set it up so that the government has difficulty mustering up unified and coherent wants to act on. Government is, in practice, limited by earth’s low coordination capacity. Which gives us a kind of safety from tyranny.

If someone in the intelligence community wanted to “disappear” me, they would have to keep it secret, because they would have political opponents, and just principled detractors, who would, if they could, expose the criminal and have them arrested. Nixon was removed from office for violating the law. It might not be perfect equality before the law, but it’s a pretty impressive example of something approaching that.

Further, I’m less pessimistic than my read of Yarvin about constructing systems in which NO individual is above the law in the sense of being able to unilaterally violate it. eg systems where everyone enforces the law on everyone else. (Systems like these are vulnerable to 51% attacks, and the number of actual people required to implement a 51% attack falls as political and/or social power is consolidated. But that’s true of literally every system of law, and the question is how we can do best.)

It does seem possible that a CEO-monarch who can be removed by a vote of the stockholders is more likely to act from straightforward material rational self-interest than voters do, currently. (Actual historical monarchies have a number of critical-level problems, from crazy kings to violent succession disputes as the norm). It seems like it is likely to have other problems—namely a principle agent problem between the shareholders and their delegate.6 I’m curious to see a government that runs on that system, and see how it behaves. Maybe it would result in good policy.

However, I think there are other schemes, mostly untried, that do a better job of incentivizing good judgement from voters, while also getting the historically-validated stability benefits of democratic governance. I’m thinking of systems like futarchy (or just prominent, public, prediction markets) and quadratic voting.

The main feature that’s doing the work in Yarvin’s conception, is the multitude of micronations competing for residents. As long as you have sufficiently low transaction costs involved in moving from one country to another, and at least some countries have politically unified enough governance that they can and do adopt the explicit policy goal of optimizing tax revenue (or, for that matter, any of a number of possible social welfare functions, or baskets of indicators), you get all the benefits of the Moldbugian system. The bit about CEO-kings isn’t actually critical. Maybe that’s the best way to optimize policy for tax revenue, or maybe not. Possibly that the king has authority to kill any citizen for any reason is net-beneficial for security and stability, such that many people prefer living in a nation where the chief executive has that level of legal authority, and overall tax revenue is higher. But then again, maybe not. (The optics are pretty unnerving, at least.)

It sounds to me that the problem is not that we don’t have kings, in particular, but just that there’s so little room for governance experimentation, in general, and so new radical ideas don’t get tried.

  1. For instance, I’m unimpressed with Yarvin’s claim that his political schema would lead to world peace. He spends a few sentences asserting that his realm-CEOs, being rational, would have no issues solving collective action problems, and would have no need for a higher governmental structure above them to enforce collective action, and then moves on. 🙄 ↩︎
  2. See, for instance, here.

    > The key is that word should. When you say your government “should do X,” or “should not do Y,” you are speaking in the hieratic language of democracy. You are postulating some ethereal and benign higher sovereign, which can enforce promises made by the mere government to whose whims you would otherwise be subject. In reality, while your government can certainly promise to do X or not to do Y, there is no power that can hold it to this promise. Or if there is, it is that power which is your real government.
    ↩︎
  3. We might try to conceive of clever schemes under which this is not so: legal systems based on blockchain smart contracts where there’s no enforcement mechanism outside of the computerized legal corpus, itself. Maybe in some scenario like that, we would have effectively grounded out the root of power into the law itself, and escaped the basic dynamic that someone is always above the law (in much the same way that reconstructing life to use encrypted genomes would potentially allow us to escape the so far inexorable pull of natural selection). ↩︎
  4. > It is immediately clear that the neocameralist should, the tight rope, is far inferior to the ethereal should, the magic leash of God. (Typically these days arriving in the form of vox populi, vox Dei. Or, as a cynic might put it: vox populi, vox praeceptoris.)
    > Given the choice between financial responsibility and moral responsibility, I will take the latter every time. If it were possible to write a set of rules on paper and require one’s children and one’s children’s children to comply with this bible, all sorts of eternal principles for good government and healthy living could be set out.
    > But we cannot construct a political structure that will enforce moral responsibility. We can construct a political structure that will enforce financial responsibility. Thus neocameralism. We might say that financial responsibility is the raw material of moral responsibility. The two are not by any means identical, but they are surprisingly similar, and the gap seems bridgeable.

    From Profit Strategies for Our New Corporate Overlords, here. ↩︎
  5. Crucially the board of directors of a realm, the people who do have the power to remove the CEO-king, should not live in that realm, for precisely the reason that this represents an incentive for the king to use his complete power over you, as your sovereign, his ability to have you and your family killed or tortured, to get you to vote as he demands in board meetings. ↩︎
  6. If the CEO-king has absolute power over his realm that seems like it gives him a lot of leeway to control the information flows about how the realm is doing back to the shareholders that might hold him accountable to profit. ↩︎

Small cashflow software businesses might be over soon?

~ elityre ~ Leave a comment

[Epistemic status: half-baked musing that I’m writing down to clarify for myself]

For the past 15 years there’s been an economic niche, where a single programer develops a useful tool, utility, or application, and sells it over the internet to a few thousand people for a small amount of money each, and make a decent (sometimes passive or mostly-passive) living on that one-person business.

In practice, these small consumer software businesses are on the far end of a continuum that includes venture-backed startups, and they can sometimes be the seed of an exponentially scaling operation. But you only need to reach product-market fit with a few thousand users for a business like this to sustainable. And at the point, it might be mostly on autopilot, and the entrepreneur has income, but can shift most of their attention to other projects, after only two or three years.

Intend (formally complice), is an example of this kind of business from someone in my circles.

I wonder if these businesses will be over soon, because of AI.

Not just that AI will be able to do the software engineering, but that AI swarms will be able to automate the whole entrepreneurial process from generating (good) ideas, developing early versions, shipping them, getting user-feedback, and iterating.

The discourse already imagines a “one person-unicorn”, where a human CEO coordinates a company of AIs to provide a product or service. With half a step more automation, you might see meta-entrepreneurs overseeing dozens or hundreds of separate AI swarms, each ideating, prototyping, and developing a business. Some will fail (just like every business), but some will grow and succeed and (just like with every other business venture) you can invest more resources into the ones that are working.

Some questions:

  • How expensive will inference be, in running these AI entrepreneurs? Will the inference costs be high enough that you need venture funding to run an AI entrepreneur-systems?
    • Estimating this breaks down into roughly “how many tokens does it take to run a business (per day?)?” and “How much will an inference token cost in 2028?”
  • What are the moats and barriers to entry here? What kind of person would capture the gains to this kind of setup.
  • Will this eat the niche of human-ideated software businesses? Will there be no room left to launch businesses like this and have them succeed, because the space of niche software products will be saturated? Or is the space of software ideas so dense, that there will still be room for differentiation, even if there are 1000x as many products of this type, of comparable quality, available?

. . .

In general, the leverage of code is going to drop over the next 5 years.

Currently, one well-placed engineer will write a line of code that might be used by millions of users. That because there’s 0-marginal cost to replicating software and so a line of code written once might as well be copied to a million computers. But it’s also representative of the relative expense of programming labor. Not many people can write (good) code and so their labor is expensive. It’s definitely not worth paying $100 an hour for an engineer to write some software when you can buy existing off the shelf software that does what you need (or almost what you need) for $50 a month.

But, as AI gets good enough that “writing code” becomes an increasingly inexpensive commodity, the cost-benefit of writing custom software is going to shift in the “benefit” direction. When writing new software is cheap, you might not want to pay the $50 a month, and there will be more flexibility to write exactly the right software for your particular usecase instead of a good-enough off the shelf-version (though I might be overestimating the pickiness of most of humanity with regards to their software). So more people and companies will write custom software more of the time, instead of buying existing software. As that happens the number of computers that run a given line of code will drop, in the process.

How I wish I lived my life (since 2020)

~ elityre ~ Leave a comment

[I wrote this a few months ago]

I always have a(n at least) part-time job, doing something object level, where someone pays me to do something that creates value. Doing something that isn’t entirely self-directed, adds some structure to my life which, I think, makes me better at doing my personal projects. I might stick to one job for six months or a year, and then move on to try something else. I want to try a bunch of different things and work with different kinds of people. I always have a job, but I also always have my eye out for my next job. I do things like research for AI impacts, grant making for SFF, startup stuff for manifold, generalist work for CAIP, logistics for Lighthaven events.

[I should have a flag for whenever I don’t have a job. That’s something that I should fix ASAP, even with just a stopgap. Instead of looking for something that I really want to do, I should make sure that I have something that I’m doing for a few hours a week-day, even if I want to find something better. When people ask me what I’m doing, I should always have a day job.]

In the evenings, I work on personal and learning projects: programming projects (including working with a tutor), studying textbooks, writing, practicing therapy skills. Whatever I’m working on, it always has a deliverable: if I’m learning something, I should write what I’m learning or gives talks about it. If I’m learning a skill, I design a “final project” that involves some person other than me.

Sometimes I’ll put the learning projects aside, and scale up my work, going all in for a campaign of a week, or 3 months, working intensively with a team to complete an end-to-end project.

Some weekends I try an intensive, doing an experiment or self-designed exercise with another person or a group of people.

I live frugally. I put away most of the money I earn, split between long run investments (both index funds and higher risk bets) and my personal development fund. I make a enough to live on from scattered projects, so I should be able to save most of what I make from my work.

I go to 5 conferences a year, trying to get exposure to interesting happenings in the world, people who are thinking about interesting things, and highly ethical women to date.

Every day I meditate and exercise. I don’t watch TV or youtube or read comic books. My go-to habits when I’m not doing anything are reading and taking notes on podcasts.

Lessons from and musings about Polytopia

~ elityre ~ Leave a comment

Over the past 6 months I’ve played about 100 hours of the 4X game “The Battle of Polytopia.” Mostly playing on “crazy” difficulty, against 3 or 4 other tribes. 

This is more than I’ve played any video game since I was 15 or so. I wanted to write up some of my thoughts, especially those that generalize.

Momentum

Polytopia is a game of momentum or compounding advantage. If I’m in the lead by turn 10 or so, I know that I am basically guaranteed to win eventually. [Edit 2024-09-12: after playing for another 30 hours, and focusing on the early game, I can now almost always win eventually, regardless of whether I have an early lead]. I can turn a current resource advantage into an overwhelming military advantage for a particular city, and seizing that city, get more of a lead. After another 10 turns, I’ll have compounded that until my tribe is an inexorable force moving across the square.

I think the number one thing that I took away from this game is the feeling of compounding momentum. Life should have that flavor. 

And, in particular, the compounding loops through the world. In polytopia, you generally want to spend down your resources to 0 or close to 0, every turn, unless there’s a specific thing that you’re aiming to buy that requires more than one maringal-turn of resource. “Saving up” is usually going to be a losing proposition, because the return on investment of seizing a city or building the population of an existing city sooner is exponential.

This also generalizes. I’m very financially conservative, by nature. I tend to earn money and save it / invest it. There’s a kind of compounding to that, but it isn’t active. There’s a different attitude one could have, where they’re investing in the market some amount every year, and putting aside some money for emergencies, but most of their investment loops through the world. Every year, they spend down most of the money they own, and invest it in ways to go faster.

I think most people, in practice, don’t do this very well: they spend their salary on a nice apartment, instead of the cheapest apartment they can afford and tutoring, personal assistants, and plane flights to one-on-one longshot bet meetings. At a personal (instead of organizational) level, I think the returns to spending additional money saturate fast, and after that point you get better returns investing in the market. But I think there might be something analogous to the “spend down your whole budget, every turn” heuristic. 

I’ve thought in the past that I should try aggressively spending more money. But maybe I should really commit to trying it. I have a full time salary for the first time in my life. Maybe this year, I should experiment with trying to find ways to spend 90% of that salary (not my investment returns, which I’ll reinvest), and see what the returns to that are. 

This overall dynamic of compounding advantages is more concerning that I, personally, haven’t built up much of an advantage yet. Mediocre accumulation of money, connections, and skills seems “medium” good. But because of the exponential, mediocre is actually quite far down a power law. This prompts me to reflect on what I can do this year to compound my existing resources (particularly with regards to personal connections, since I realized late in life that who you know who knows what you can do, is a constraint on what you can do).

Geography

Because of this momentum effect, the geography of the square dominates all other considerations in the question of which tribe will eventually win. In particular, if I start out isolated, far from any other tribes, with several easily accessible settlements available to convert, winning is going to be easy. I can spend the first ten turns capturing those settlements and building up their population without needing to expend resources on military units for defense.

In contrast, if I start out sandwiched between two other tribes with all settlements that are within my reach also within theirs, the struggle is apt to be brutal. It’s still possible to win starting from this position: The key is to seize, and build up at least two cities, and create enough units defensively that the other tribes attack each other instead of you. (That’s one thing that I learned: you sometimes need to build units to stand near your cities, even when you’re not planning an attack, because a density of units discourages other tribes from attacking you, in the first place). 

From there, depending on how dire the straights are, and if I’m on the coast, I’ll want to either 

  1. Train defenders to garrison my cities, and then quickly send out scouts to convert nonaffiliated settlements, and build up an advantage that way, or
  2. Train an attack force (of mostly archers, most likely, because a mass of archers can attack from a distance with minimal risk) target a city that the other two tribes are fighting over. I can sweep in and seize it after they’ve exhausted themselves.

This can work, but it still depends on luck. If you can’t get to your first settlement(s) fast enough, or another tribe captures one of your cities before you’ve had time to build up a detering defense force, there’s not really a way to recover. I’ll be fighting against better resourced adversaries for the rest of the game, until they overwhelm me. Usually I’ll just start the game over when I get unlucky this early.

This overwhelming importance of initial conditions sure seems like it generalizes to life, but mostly as a dower reminder that life is unfair. Insofar as you can change your initial conditions, they weren’t actually initial.

Thresholds, springs, and concentration of force

There are units of progress in polytopia that are composed of smaller components, but which don’t provide any value until all components are completed. 

For instance, it’s tempting to harvest a fruit or hunt an animal to move the resource counter of a city up by one tick. But if you don’t have the stars to capture enough resources to reach the next population-increase threshold for that city (or there just aren’t other accessible resources nearby), it doesn’t actually help to collect that one resource. You get no benefit for marginal ticks on the resource counter, you only get benefit from increases in city population.

Even if collecting the resource is the most valuable thing to do this turn, you’re still better off holding off and waiting a turn (one exception to the “spend down your resources every turn” heuristic). Waiting gives you optionality in how you spend those resources—you might have even better options in the next turn, including training units that were occupied last turn, or researching technologies 

Similarly, capturing a city entails killing the unit that is stationed there, and moving one of your units into its place, with few enough enemy units nearby that your unit isn’t killed before the next turn. Killing just the central stationed enemy unit, without having one of your units nearby to capture the city, is close to useless (not entirely useless, because it costs the enemy tribe one unit). Moving a unit into the city, only for it to be killed before the next turn, is similarly close to useless.

So in most cases, capturing a city is a multi-turn campaign of continually training and/or moving enough units into position to have a relative military advantage, killing enough of the enemy units, and moving one of your units (typically a defender or a giant, if you can manage it) into position in the city.

Crucially, partially succeeding at a campaign—killing most of the units, but not getting all the way to capturing the city, it buys you effectively nothing. You don’t win in polytopia by killing units, except insofar as that is instrumental to capturing cities.

More than that, if you break off a campaign part way through, your progress is not preserved. When you back your units out, that gives the enemy city slack to recover and replenish their units. So if you go back to capture that city later, you’ll have to more or less start over from scratch with wearing down their nearby military.

That is to say, capturing a city in polytopia is spring-like: if you don’t push it all the way to completion, it bounces back, and you need to start over again. It’s not just that marginal progress doesn’t provide marginal value until you reach a threshold point. Marginal progress decays over time.

I can notice plenty of things that are spring-like in this way, once I start thinking in those terms. 

Some technical learning, for instance. If I study something for a bit, and then leave it for too long (I’m not sure what “too long” is—maybe more than two weeks?) I don’t remember the material enough for my prior studying to help me much. If I want to continue, I basically have to start over.

But on the other hand, I read and studied the first few chapters of a Linear Algebra textbook in 2019, and that’s served me pretty well: I can rely on at least some of those concepts in my thinking. I think this differences is partly due to the material (some subjects just stick for me better, or are more conceptually useful for me compared to others). But largely, I think this is a threshold effect: if I study the content enough to chunk and consolidate the concepts, it sticks with me and I can build on it. But if I read some of a textbook, but don’t get to the point of consolidating the concepts, it just gets loaded into my short term memory, to decay on the order of weeks.

Writing projects definitely have the threshold-dynamic—they don’t provide any value until I ship them—and they’re partially but not fully spring-like. When I’ve left a writing project for too long, it’s hard to come back to it: the motivating energy is gone. And sometimes I do end up, when I’m inspired again, rewriting essentially the same text (though often with a different structure). But sometimes I am able to use partial writing from previous attempts.

Generalizing, one reason why things are springs, is because short term memories and representations decay, and you need to pass the threshold of consolidating into long term representations.

In polytopia, because capturing cities is spring-like, succeeding requires having a concentration of force. Splitting your forces to try to take two cities at once, and be worse than useless. And so one of the most important disciplines of playing polytopia is having internal clarity about which city you’re targeting next, so that you can overwhelm that city, capture it, consolidate it and then move on to the next one. Sometimes there are sudden opportunities to capture cities that were not your current target and late in the game, you might have more than one target at a time (usually from different unit-training bases).

Similarly, anything in my life that’s spring-like demands a concentration of force. 

If technical learning in my short term memory tends to decay, that means that I need to commit sufficiently to a learning project for long enough to hit the consolidation threshold. I want to concentrate my energies on the project until I get to the point of success, whatever success means.

Same basic principle for writing projects. When writing, I should probably make a point to just keep going until I have a first complete draft.

Video games

Probably the most notable thing I learned was not from the content, but from the format. Video games can work. 

I got better at playing polytopia over the period that I was playing it, from mostly losing to mostly winning my games. That getting better was mostly of the form of making mistakes, noticing those mistakes, and then more or less automatically learning the habits to patch those mistakes. 

For instance, frustration at losing initiative because I left a city un-garrisoned and an enemy unit came up and took it without a fight while I wasn’t looking, led into a general awareness of all my cities and the quiet units within movement distance of them, so that I could quickly train a unit to garrison them. 

Or running into an issue when I ran out of population for a city and couldn’t easily garrison it, and learning to keep a defender within one step of a city, so that I can train units there to send to the front, but move the defender into place when the population is full.

This was not very deliberate or systematic. I just kept playing and gradually learned how to avoid the errors that hobbled me.

And I just kept playing because it was (is) addictive. In particular, when I finished a game, there was an automatic impulse to start another one. I would play for hours at a stretch. At most I think I played for ten hours in a row. 

Why was it addictive? I think the main thing is that the dynamic of the game means I never get blocked with no option, or no idea, for what to do next. At every moment there’s an affordance to move the game forward: either something to do or just moving on to the next turn. The skill is in taking actions skillfully, but not in figuring out how to take actions at all. I think this, plus an intermittent reinforcement schedule was crucial to what made it addictive. 

Overall, this has been bad for my life, especially after the point when I started mostly winning, and I wasn’t learning as much any more. 

But I think I learned something about learning and getting better in that process. I’ve been playing with the idea of intentionally cultivating that kind of addiction for other domains, or pretending as if I’m experiencing that kind of addiction to simulate it.

I bet I could get into a mode like this with programming, where I compulsively keep going for hours over weeks, and in the process learn the habits to counter my mistakes and inefficiencies, less because of anything systematic, and more just because those errors are present to mind in my short term memory by the time I encounter them again. I think I’m probably close to having enough skill in programming that I can figure out how to never be blocked, especially with the help of an LLM, and get into the addictive rhythm.

Further, this makes me more interested in trying to find video games that are both dopamine-addictive and train my intuition for an important domain. 

I’ve been playing with Manifold markets, recently, and I feel like I’m getting a better sense of markets in the process. I wonder if there are good video games for getting an intuition for linear algebra, or economics. I have registered that playing 100 hours of factorio is an important training regime. I wonder if there are others. 

I haven’t really played video games since I was in middle school (with the exception of some rationality training exercises on snakebird and baba is you). At the time I was playing Knights of the Old Republic, and decided that I would try to become a jedi in real life, instead of in the game. I mostly haven’t played video games since. 

I now think that this was maybe a mistake.

It’s hard to know what lessons I would have learned if I had played more video games—when I played Age of Mythology and Age of Empires as a kid, I don’t remember getting better over time, as I did with polytopia. But I do think there are lessons that I could have learned from playing video games that would have helped me in thinking about my life. Notably, getting reps playing through games with early, mid, and late stages, would have given me a model for planning across life stages, which is something that, in retrospect, I was lacking. I didn’t have an intuitive sense of the ways that the shape of my opportunities would be different in my 20s vs. my thirties, for instance. Possibly I would have avoided some life errors if I had spent more time playing and learning to get good at, video games.

Some thoughts on giving into threats or not

~ elityre ~ Leave a comment

Eliezer claims that dath ilani never give in to threats.

The only reason people will make threats against you, the argument goes, is if those people expect that you might give in. If you have an iron-clad policy against acting in response to threats made against you, then there’s no point in making or enforcing the threats in the first place. There’s no reason for the threatener to bother, so they don’t. Which means in some sufficiently long run, refusing to submit to threats means you’re not subject to threats.

This seems a bit fishy to me. I have a lingering suspicion that this argument doesn’t apply, or at least doesn’t apply universally, in the real world.

I’m thinking here mainly of a prototypical case of an isolated farmer family (like the early farming families of the greek peninsula, not absorbed into a polis), being accosted by some roving bandits, such as the soldiers of the local government. The bandits say “give us half your harvest, or we’ll just kill you.”

The argument above depends on a claim about the cost of executing on a threat. “There’s no reason to bother” implies that the threatener has a preference not to bother, if they know that the threat won’t work.

I don’t think that assumption particularly applies. For many cases, like the case above, the cost to the threatener of executing on the threat is negligible, or at least small relative to the available rewards. The bandits don’t particularly mind killing the farmers and taking their stuff, if the farmers don’t want to give it up. There isn’t a realistic chance that the bandits, warriors specializing in violence and outnumbering the farmers, will lose a physical altercation.

From the badnits’ perspective their are two options:

  1. Showing up, threatening to kill the farmers, taking away ask much food as they can carry (and then maybe coming back to accost them again next year).
  2. Showing up, threatening to kill the farmers, actually killing the farmers, and then taking away as much food as they can carry.

It might be easier and less costly for the bandits to get what they want by being scary rather than by being violent. But the plunder is definitely enough to make violence worth it if it comes to that. They prefer option 1, but they’re totally willing to fall back on option 2.

It seems like, in this situation, the farmers are probably better off cooperating with the bandits and giving them some food, even knowing that that means that the bandits will come back and demand “taxes” from them every harvest. They’re just better off submitting.

Maybe, decision theoretically, this situation doesn’t count as a threat. The bandits are taking food from the the farmers, one way or the other, and they’re killing the farmers if they try to stop that. They’re not killing the farmers so that they’ll give up their food.

But that seems fishy. Most of the time, the bandits don’t, in fact have to resort to violence. Just showing up and threatening violence is enough to get what they want. The farmers do make the lives of the bandits easier by submitting and giving them much of the harvest without resistance. Doing otherwise would be straightforwardly worse for them.

Resisting the bandits out of a commitment to some notion of decision-theoretic rationality seems exactly analogous to two-boxing in Newcom’s problem, because of a commitment to (causal) decision-theoretic rationality.

You might not want to give in out of spite. “Fuck you. I’d rather die than help you steal from me.” But a dath ilani would say that that’s a matter of the utility function, not of decision theory. You just don’t like submitting to threats, and so will pay big costs to avoid it, not that you’re following a policy that maximizes your payoffs.

So, it seems like the policy has to be “don’t give into threats that are sufficiently costly to execute that the threatener would prefer not to bother, if they knew in advance that you wouldn’t give in”. (And possibly with the additional caveat “if the subjunctive dependence between you and the threatener is sufficiently high.”)

But that’s a much more complicated policy. For one thing, it requires a person-being-threatened to accurately estimate how costly it would be for the threatener to execute their threat (and the threatener is thereby incentivized to deceive them about that).

Hm. But maybe that’s easy to estimate actually, in the cases where the threatener gets a payout of 0, if the person-being-threatened doesn’t cooperate with the threat? Which is the case for most blackmail attempts, for instances, but not necessarily “if you don’t give me some of your harvest, I’ll kill you.”

In lots of case, it seems like it would be ambiguous. Especially when there are large power disparities in favor of the threatener. When someone powerful threatens you the cost of executing the the threat is likely to be small for them, possibly small enough to be negligible. And in those cases, their own spite at you for resisting them might be more than enough reason to act on it.

[Ok. that’s enough for now.]

Some notes on my recent, sudden, improvement in mood

~ elityre ~ 1 Comment

Two days ago, I went from feeling listless and unmotivated to feeling more or less like my usual self, quite suddenly. I’m not sure what caused this and I want to capture some of my hypotheses for latter pattern matching.

How I was feeling

On June 14th, I sent the following message to some of my friends:

I’m kind of in a weird place. I’ve been saying that I’m pseudo-depressed, but I’m not sure if that’s a good expression.

I’ve been sleeping a lot, sometimes more than 12 hours a day. In particular, I seem to be alternating, day by day, between getting up at 7:00 AM and getting up at noon.

I feel relatively unmotivated. Things mostly don’t seem worth doing. Like, I imagine setting a serious intention to do something, and part of me asks “why though? what good thing will result from this?” and I don’t have a good answer. Everything I want doesn’t exist?

Duncan was disappointed and concerned that I didn’t seem to have “a deal” any more. I don’t have anything to live for.

That said, I mostly don’t feel particularly or actively unhappy.

And,

  • I’m doing a pretty good job of exercising every day.
  • I’ve been meditating every day, and I’ve been sinking pretty deep into it.
  • I usually write something or program something every day. I’m getting a little bit done, but not very much.
  • I’m only eating between 2:00 PM and 7:00 PM, except for liquid food.

Earlier in this month, I was pretty distracted / dopamine addicted, and I successfully weaned that off.

I woke up at noon again today, and thought “ok, something weird seems to be happening with me”, and thought that it seemed prudent to maybe loop other people in on what’s going on with me.

I think I’m only going to be happy if I’m working hard, but at the moment, I don’t really feel the point. I’m currently just plugging along anyway, and maybe I’ll get some momentum + attempting to dialog with my desires.

Of course, it might be an entirely non-psychological thing. Maybe I should try eating a little bit of meat, as an experiment, and see if that helps. I might also go to another city for  a few days. Open to other suggestions for experiments, I guess, though I might wrinkle my nose at some of them and say no.

I’ve been feeling like this for at least a month, arguably all the way back to when I was in Taiwan in May.

A shift

Then two days ago, on June 27th, I had a pretty abrupt shift.

My notes from that day

  • 18:23 – I’m feeling notably, specifically happy. #[[phenomenological naturalism]]
    • I started while I was mindlessly browsing twitter a few hours ago (as I have been doing a lot over the past few days). I remember seeing something that was funny, that made me laugh. In laughing I felt good.
    • Later, I noticed a feeling of warmth around my heart.
    • I lay down with my eyes closed for a while, and continued to feel the feeling around my heart.
    • This is markedly different from how I have been feeling (unmotivated, listless).

Since then, I’ve been feeling basically like my normal self, able to get momentum on stuff.

Causes?

What happened?

I still don’t know. Here are some possibilities.

  1. It was something that I ate? I’ve been at home more and RGI less since I got back from Taiwan. I’ve been eating more eggs and more carbs, and less kale, than I often do.
    • On the 27th, I ate:
      • I ate a Quinoa bowl with vegetables (including carrots and zucchini and some others).
      • I ate some of Bulgur bowl with vegetables (including green beans, and some others)
      • I ate a bobos chocklate chip bar.
      • I ate a bunch of Made Good rice crispy bars. [Note I had been eating a lot of these over prior days]
      • I ate a bunch of pure organic fruit strips
        • I did note that these had a very strong taste.
      • I drank a Gatorade.
      • I ate some chocolate chipMade Good granola minis.
    • Could the vegetables have made the difference? The Quinoa? It seems like the next time I’m feeling bad or lame, I should just try varying my diet and and if that shakes something loose.
  2. Creatine made the difference? I recently bought some creatine monohydrate supplements in gummy form. Those arrived on the 24th, and I’d been munching on them throughout the 25th, 26th, and 27th. This is probably 5x more creatine than I usually take in a day, and much more than I’ve been taking recently. Possibly that made the difference?
  3. Other supplements made the difference? I haven’t been taking my standard supplement stack since I got back from Taiwan (because it was a arguably-trivial inconvenience to repack all my pill containers). But on the 25th, 26th, and 27th, I took my current stack of
    • I think,
      • Morning, empty-stomach pills
        • Korean Panax Ginseng
        • Inositol
        • Alpha Lipoic acid
        • Taurine
        • CDP choline
        • L-tyrosine
        • Creatine
      • With a meal pills
        • Bacopa
        • Ginkgo Biloba
        • Alpha GPC
        • b12
        • vitamine
        • Vegan EPA + DHA
        • Creatine
    • Could that have been it? It seems unlikely, because I wasn’t taking those supplements for very long before I started feeling lame. I was taking all of those for less than 2 months (I think), before I was interrupted by travel)
  4. Earlier that day, I was thinking about deciding to be happy. I didn’t quite get to the point of committing to it, in that moment, but the way I was feeling it out felt real. I could imagine just deciding that I was going to be happy or deciding to to have high energy, because life is just better if I’m happy / high energy, that it is basically a free variable, that I could just decide to be one way or another.
  5. I had decided the night before that I would start trying to cowork with people for 4 hours in the mornings, to build momentum. I am doing that (I’m typing this essay in a coworking session!), though I only barely did that on the 27th, around noon, for around an hour. That seems unlikely.

Some powers have ethical valence

~ elityre ~ Leave a comment

There’s a trope of many fantasy settings, different kinds of magic have different moral valence, and your use of each kind influences your morality. Or alternatively, you moral character has effects which magics you can use.

In the starwars extended universe, there’s the light side and the dark side. It’s usually implied that the use of the powers of the dark side are seductive and corrupting.

‘Is the Dark Side stronger?’

‘No, no, no. Quicker, easier, more seductive.’

It’s not (to my limited knowledge, at least), explained why or how using particular force powers is leads one to the poor moral behavior, but it is stated that accessing dark side powers requires tapping into “negative” emotions, like fear and anger. Presumably there’s some magical explanation for why using the dark side is so corrupting. But as a matter of simple psychology, using the dark side entails nurturing and cultivating emotions and emotional dispositions that are generally not good for your soul.

In my memory of the Knights of the Old Republic game, the causality went in the other direction: your alignment on the light side dark side axis was determined by the choices that you made in the game. High integrity and altruistic choices moved you towards the lightside and selfish, vengeful, or ruthless choices moved you towards the dark side.

And there’s a broader version of the trope, of the wizard who conducts dark rituals with demonic beings, and this being a slippery slope to evil as exposure to those powers (and the kinds of sacrifices they demand in exchange for power) warps his or her soul.

There’s a parallel of this dynamic in real life.

Some skills have a moral valence, because they disproportionately favor cooperation or exploitation. Which skills you choose to develop shapes your affordances, which shapes your moral habits and choices.

For instance, if you learn to lie skillfully, you build an affordance for lying. When faced with a problem a prominent tool in your toolbox will be to lie to get your way. This puts an incentive on you to use that tool when you can, and thereby leads you to less ethical behavior than you might otherwise have chosen.

Another example: various persuasion techniques that take exploit human biases to get others to agree to what you want generally lean evil. They’re more symmetrical than argument, and methods in that class have a disproportionately larger set of outcomes in which you get others to agree to something counter to their idealized or reflective interests.

It’s not that this couldn’t possibly be used for Good. It’s that honing this as a skill, builds affordances of ethically dubious action.

In contrast, Convergent Facilitation, an offshoot of NonViolent Communication, is a group decision making framework that involves hearing, holding, and solving for the disparate needs of everyone in a group, and thereby drawing out both the willingness to adapt and the creativity of the whole group. This is a technique that is structurally cooperative. Helping other people get what’s important to them is a functional part of the technique, not a side benefit that could in principle be stripped away, for better selfish efficiency.

A person who puts skill points into getting really good at Convergent Facilitation is building skill that supports cooperative behavior, as someone who puts skill points into psychological persuasion techniques is building skill that supports adversariality. Investing in one or the other shapes the affordances that are available to you in any given situation. If you’re good at persuading people, you’ll see those options, and if you’re good at CF, you’ll see opportunities to do CF-like moves to find mutually supportive solutions.

The better you are at lying, the more tempting it is to lie.

That difference in affordances corresponds to a difference in payoffs: If you’re good at persuasion, it’s a higher cost to forge that strategy when it would be unethical to use. The tools you’re skilled with exert a vacuum pull towards their ethical attractor.

In this way, the some skills have a moral valence. Which you choose to cultivate exert pressures on the condition of your soul.

A comment on Marxism.

~ elityre ~ 4 Comments

Marxism is like…incredibly dumb? Like it’s pretty shocking that it shook the world. And indeed it was once lauded particularly “scientific”, despite being not remotely like that.

I think that the main thing that happened in the 1800s was not that Marxism, per se, was actually compelling to people, but rather that there were a lot of people who could perceive serious injustice, and felt motivational force to organize to fight it, and they grabbed on to this ideology kind of at random, because it gave expression to their outlook, and claimed to be scientific, so was prestigious (it mattered a lot to the Russian radicals that it was _western_, because western Europe was modern where Russia as a backwater).

But the actual ideology, as near as I can tell, is dumb, and has always been dumb.

And then on top of that, when people tried to execute it…it resulted in (I think) in literally the most loss of human life, on purpose, ever?

I tried to talk with some smart Marxists once, to try to pass the ITT. I think I didn’t actually find the cream of the crop, but I was not impressed with the SSC-subreddit associated Marxist who had a video call with me.

I would still be up for talking to a smart Marxist, if anyone happens to know one.

But as it is, this is among the most inaccurate, most intellectually bankrupt, most consequentially harmful ideologies.

One projection of how AI could play out

~ elityre ~ Leave a comment

Back in January, I participated in a workshop in which the attendees mapped out how they expect AGI development and deployment to go. The idea was to start by writing out what seemed most likely to happen this year, and then condition on that, to forecast what seems most likely to happen in the next year, and so on, until you reach either human disempowerment or an end of the acute risk period.

This post was my attempt at the time.

I spent maybe 5 hours on this, and there’s lots of room for additional improvement. This is not a confident statement of how I think things are most likely to play out. There are already some ways in which I think this projection is wrong. (I think it’s too fast, for instance). But nevertheless I’m posting it now, with only a few edits and elaborations, since I’m probably not going to do a full rewrite soon.

2024

  • A model is released that is better than GPT-4. It succeeds on some new benchmarks. Subjectively, the jump in capabilities feels smaller than that between RLHF’d GPT-3 and RLHF’d GPT-4. It doesn’t feel as shocking the way chat-GPT and GPT-4 did, for either x-risk focused folks, or for the broader public. Mostly it feels like “a somewhat better language model.”
    • It’s good enough that it can do a bunch of small-to-medium admin tasks pretty reliably. I can ask it to find me flights meeting specific desiderata, and it will give me several options. If I give it permission, it will then book those flights for me with no further inputs from me.
    • It works somewhat better as an autonomous agent in an auto gpt harness, but it still loses its chain of thought / breaks down/ gets into loops.
    • It’s better at programming.
      • Not quite good enough to replace human software engineers. It can make a simple react or iphone app, but not design a whole complicated software architecture, at least without a lot of bugs.
      • It can make small, working, well documented, apps from a human description.
        • We see a doubling of the rate of new apps being added to the app store as people who couldn’t code now can make applications for themselves. The vast majority of people still don’t realize the possibilities here, though. “Making apps” still feels like an esoteric domain outside of their zone of competence, even though the barriers to entry just lowered so that 100x more people could do it. 
  • From here on out, we’re in an era where LLMs are close to commoditized. There are smaller improvements, shipped more frequently, by a variety of companies, instead of big impressive research breakthroughs. Basically, companies are competing with each other to always have the best user experience and capabilities, and so they don’t want to wait as long to ship improvements. They’re constantly improving their scaling, and finding marginal engineering improvements. Training runs for the next generation are always happening in the background, and there’s often less of a clean tabula-rasa separation between training runs—you just keep doing training with a model continuously. More and more, systems are being improved through in-the-world feedback with real users. Often chatGPT will not be able to handle some kind of task, but six weeks later it will be able to, without the release of a whole new model.
    • [Does this actually make sense? Maybe the dynamics of AI training mean that there aren’t really marginal improvements to be gotten. In order to produce a better user experience, you have to 10x the training, and each 10x-ing of the training requires a bunch of engineering effort, to enable a larger run, so it is always a big lift.]
    • (There will still be impressive discrete research breakthroughs, but they won’t be in LLM performance)

2025

  • A major lab is targeting building a Science and Engineering AI (SEAI)—specifically a software engineer.
    • They take a state of the art LLM base model and do additional RL training on procedurally generated programming problems, calibrated to stay within the model’s zone of proximal competence. These problems are something like leetcode problems, but scale to arbitrary complexity (some of them require building whole codebases, or writing very complex software), with scoring on lines of code, time-complexity, space complexity, readability, documentation, etc. This is something like “self-play” for software engineering. 
    • This just works. 
    • A lab gets a version that can easily do the job of a professional software engineer. Then, the lab scales their training process and gets a superhuman software engineer, better than the best hackers.
    • Additionally, a language model trained on procedurally generated programming problems in this way seems to have higher general intelligence. It scores better on graduate level physics, economics, biology, etc. tests, for instance. It seems like “more causal reasoning” is getting into the system.
  • The first proper AI assistants ship. In addition to doing specific tasks,  you keep them running in the background, and talk with them as you go about your day. They get to know you and make increasingly helpful suggestions as they learn your workflow. A lot of people also talk to them for fun.

2026

  • The first superhuman software engineer is publically released.
    • Programmers begin studying its design choices, the way Go players study AlphaGo.
    • It starts to dawn on e.g. people who work at Google that they’re already superfluous—after all, they’re currently using this AI model to (unofficially) do their job—and it’s just a matter of institutional delay for their employers to adapt to that change.
      • Many of them are excited or loudly say how it will all be fine/ awesome. Many of them are unnerved. They start to see the singularity on the horizon, as a real thing instead of a social game to talk about.
      • This is the beginning of the first wave of change in public sentiment that will cause some big, hard to predict, changes in public policy [come back here and try to predict them anyway].
  • AI assistants get a major upgrade: they have realistic voices and faces, and you can talk to them just like you can talk to a person, not just typing into a chat interface. A ton of people start spending a lot of time talking to their assistants, for much of their day, including for goofing around.
    • There are still bugs, places where the AI gets confused by stuff, but overall the experience is good enough that it feels, to most people, like they’re talking to a careful, conscientious person, rather than a software bot.
    • This starts a whole new area of training AI models that have particular personalities. Some people are starting to have parasocial relationships with their friends, and some people programmers are trying to make friends that are really fun or interesting or whatever for them in particular.
  • Lab attention shifts to building SEAI systems for other domains, to solve biotech and mechanical engineering problems, for instance. The current-at-the-time superhuman software engineer AIs are already helpful in these domains, but not at the level of “explain what you want, and the AI will instantly find an elegant solution to the problem right before your eyes”, which is where we’re at for software.
    • One bottleneck is problem specification. Our physics simulations have gaps, and are too low fidelity, so oftentimes the best solutions don’t map to real world possibilities.
      • One solution to this is that, (in addition to using our AI to improve the simulations) is we just RLHF our systems to identify solutions that do translate to the real world. They’re smart, they can figure out how to do this.
  • The first major AI cyber-attack happens: maybe some kind of superhuman hacker worm. Defense hasn’t remotely caught up with offense yet, and someone clogs up the internet with AI bots, for at least a week, approximately for the lols / the seeing if they could do it. (There’s a week during which more than 50% of people can’t get on more than 90% of the sites because the bandwidth is eaten by bots.)
    • This makes some big difference for public opinion. 
    • Possibly, this problem isn’t really fixed. In the same way that covid became endemic, the bots that were clogging things up are just a part of life now, slowing bandwidth and making the internet annoying to use.

2027 and 2028

  • In many ways things are moving faster than ever in human history, and also AI progress is slowing down a bit.
    • The AI technology developed up to this point hits the application and mass adoption phase of the s-curve. In this period, the world is radically changing as every industry, every company, every research lab, every organization, figures out how to take advantage of newly commoditized intellectual labor. There’s a bunch of kinds of work that used to be expensive, but which are now too cheap to meter. If progress stopped now, it would take 2 decades, at least, for the world to figure out all the ways to take advantage of this new situation (but progress doesn’t show much sign of stopping).
      • Some examples:
        • The internet is filled with LLM bots that are indistinguishable from humans. If you start a conversation with a new person on twitter or discord, you have no way of knowing if they’re a human or a bot.
          • Probably there will be some laws about declaring which are bots, but these will be inconsistently enforced.)
          • Some people are basically cool with this. From their perspective, there are just more people that they want to be friends with / follow on twitter. Some people even say that the bots are just better and more interesting than people. Other people are horrified/outraged/betrayed/don’t care about relationships with non-real people.
            • (Older people don’t get the point, but teenagers are generally fine with having conversations with AI bots.)
          • The worst part of this is the bots that make friends with you and then advertise to you stuff. Pretty much everyone hates that.
        • We start to see companies that will, over the next 5 years, grow to have as much impact as Uber, or maybe Amazon, which have exactly one human employee / owner +  an AI bureaucracy.
        • The first completely autonomous companies work well enough to survive and support themselves. Many of these are created “free” for the lols, and no one owns or controls them. But most of them are owned by the person who built them, and could turn them off if they wanted to. A few are structured as public companies with share-holders. Some are intentionally incorporated fully autonomous, with the creator disclaiming (and technologically disowning (eg deleting the passwords)) any authority over them.
          • There are legal battles about what rights these entities have, if they can really own themselves, if they can have bank accounts, etc. 
          • Mostly, these legal cases resolve to “AIs don’t have rights”. (For now. That will probably change as more people feel it’s normal to have AI friends).
        • Everything is tailored to you.
          • Targeted ads are way more targeted. You are served ads for the product that you are, all things considered, most likely to buy, multiplied by the lifetime profit if you do buy it. Basically no ad space is wasted on things that don’t have a high EV of you, personally, buying it. Those ads are AI generated, tailored specifically to be compelling to you. Often, the products advertised, not just the ads, are tailored to you in particular.
            • This is actually pretty great for people like me: I get excellent product suggestions.
          • There’s not “the news”. There’s a set of articles written for you, specifically, based on your interests and biases.
          • Music is generated on the fly. This music can “hit the spot” better than anything you listened to before “the change.”
          • Porn. AI tailored porn can hit your buttons better than sex.
          • AI boyfriends/girlfriends that are designed to be exactly emotionally and intellectually compatible with you, and trigger strong limerence / lust / attachment reactions.
        • We can replace books with automated tutors.
          • Most of the people who read books will still read books though, since it will take a generation to realize that talking with a tutor is just better, and because reading and writing books was largely a prestige-thing anyway.
            • (And weirdos like me will probably continue to read old authors, but even better will be to train an AI on a corpus, so that it can play the role of an intellectual from 1900, and I can just talk to it.)
        • For every task you do, you can effectively have a world expert (in that task and in tutoring pedagogy) coach you through it in real time.
          • Many people do almost all their work tasks with an AI coach.
        • It’s really easy to create TV shows and movies. There’s a cultural revolution as people use AI tools to make custom Avengers movies, anime shows, etc. Many are bad or niche, but some are 100x better than anything that has come before (because you’re effectively sampling from a 1000x larger distribution of movies and shows). 
        • There’s an explosion of new software, and increasingly custom software.
          • Facebook and twitter are replaced (by either external disruption or by internal product development) by something that has a social graph, but lets you design exactly the UX features you want through a LLM text interface. 
          • Instead of software features being something that companies ship to their users, top-down, they become something that users and communities organically develop, share, and iterate on, bottom up. Companies don’t control the UX of their products any more.
          • Because interface design has become so cheap, most of software is just proprietary datasets, with (AI built) APIs for accessing that data.
        • There’s a slow moving educational revolution of world class pedagogy being available to everyone.
          • Millions of people who thought of themselves as “bad at math” finally learn math at their own pace, and find out that actually, math is fun and interesting.
          • Really fun, really effective educational video games for every subject.
          • School continues to exist, in approximately its current useless form.
          • [This alone would change the world, if the kids who learn this way were not going to be replaced wholesale, in virtually every economically relevant task, before they are 20.]
        • There’s a race between cyber-defense and cyber offense, to see who can figure out how to apply AI better.
          • So far, offense is winning, and this is making computers unusable for lots of applications that they were used for previously:
            • online banking, for instance, is hit hard by effective scams and hacks.
            • Coinbase has an even worse time, since they’re not issued (is that true?)
          • It turns out that a lot of things that worked / were secure, were basically depending on the fact that there are just not that many skilled hackers and social engineers. Nothing was secure, really, but not that many people were exploiting that. Now, hacking/scamming is scalable and all the vulnerabilities are a huge problem.
          • There’s a whole discourse about this. Computer security and what to do about it is a partisan issue of the day.
        • AI systems can do the years of paperwork to make a project legal, in days. This isn’t as big an advantage as it might seem, because the government has no incentive to be faster on their end, and so you wait weeks to get a response from the government, your LMM responds to it within a minute, and then you wait weeks again for the next step.
          • The amount of paperwork required to do stuff starts to balloon.
        • AI romantic partners are a thing. They start out kind of cringe, because the most desperate and ugly people are the first to adopt them. But shockingly quickly (within 5 years) a third of teenage girls have a virtual boyfriend.
          • There’s a moral panic about this.
        • AI match-makers are better than anything humans have tried yet for finding sex and relationships partners. It would still take a decade for this to catch on, though.
          • This isn’t just for sex and relationships. The global AI network can find you the 100 people, of the 9 billion on earth, that you most want to be friends / collaborators with. 
        • Tons of things that I can’t anticipate.
    • On the other hand, AI progress itself is starting to slow down. Engineering labor is cheap, but (indeed partially for that reason), we’re now bumping up against the constraints of training. Not just that buying the compute is expensive, but that there are just not enough chips to do the biggest training runs, and not enough fabs to meet that demand for chips rapidly. There’s huge pressure to expand production but that’s going slowly relative to the speed of everything else, because it requires a bunch of eg physical construction and legal navigation, which the AI tech doesn’t help much with, and because the bottleneck is largely NVIDIA’s institutional knowledge, which is only partially replicated by AI.
      • NVIDIA’s internal AI assistant has read all of their internal documents and company emails, and is very helpful at answering questions that only one or two people (and sometimes literally no human on earth) know the answer to. But a lot of the important stuff isn’t written down at all, and the institutional knowledge is still not fully scalable.
      • Note: there’s a big crux here of how much low and medium hanging fruit there is in algorithmic improvements once software engineering is automated. At that point the only constraint on running ML experiments will be the price of compute. It seems possible that that speed-up alone is enough to discover eg an architecture that works better than the transformer, which triggers and intelligence explosion.

2028

  • The cultural explosion is still going on, and AI companies are continuing to apply their AI systems to solve the engineering and logistic bottlenecks of scaling AI training, as fast as they can.
  • Robotics is starting to work.

2029 

  • The first superhuman, relatively-general SEAI comes online. We now have basically a genie inventor: you can give it a problem spec, and it will invent (and test in simulation) a device / application / technology that solves that problem, in a matter of hours. (Manufacturing a physical prototype might take longer, depending on how novel components are.)
    • It can do things like give you the design for a flying car, or a new computer peripheral. 
    • A lot of biotech / drug discovery seems more recalcitrant, because it is more dependent on empirical inputs. But it is still able to do superhuman drug discovery, for some ailments. It’s not totally clear why or which biotech domains it will conquer easily and which it will struggle with. 
    • This SEAI is shaped differently than a human. It isn’t working memory bottlenecked, so a lot of intellectual work that humans do explicitly, in sequence, the these SEAIs do “intuitively”, in a single forward pass.
      • I write code one line at a time. It writes whole files at once. (Although it also goes back and edits / iterates / improves—the first pass files are not usually the final product.)
      • For this reason it’s a little confusing to answer the question “is it a planner?” It does a lot of the work that humans would do via planning it does in an intuitive flash.
    • The UX isn’t clean: there’s often a lot of detailed finagling, and refining of the problem spec, to get useful results. But a PhD in that field can typically do that finagling in a day.
    • It’s also buggy. There’s oddities in the shape of the kind of problem that is able to solve and the kinds of problems it struggles with, which aren’t well understood.
    • The leading AI company doesn’t release this as a product. Rather, they apply it themselves, developing radical new technologies, which they publish or commercialize, sometimes founding whole new fields of research in the process. They spin up automated companies to commercialize these new innovations.
  • Some of the labs are scared at this point. The thing that they’ve built is clearly world-shakingly powerful, and their alignment arguments are mostly inductive “well, misalignment hasn’t been a major problem so far”, instead of principled alignment guarantees. 
    • There’s a contentious debate inside the labs.
    • Some labs freak out, stop here, and petition the government for oversight and regulation.
    • Other labs want to push full steam ahead. 
    • Key pivot point: Does the government put a clamp down on this tech before it is deployed, or not?
      • I think that they try to get control over this powerful new thing, but they might be too slow to react.

2030

  • There’s an explosion of new innovations in physical technology. Magical new stuff comes out every day, way faster than any human can keep up with.
  • Some of these are mundane.
    • All the simple products that I would buy on Amazon are just really good and really inexpensive.
    • Cars are really good.
    • Drone delivery
    • Cleaning robots
    • Prefab houses are better than any house I’ve ever lived in, though there are still zoning limits.
  • But many of them would have huge social impacts. They might be the important story of the decade (the way that the internet was the important story of 1995 to 2020) if they were the only thing that was happening that decade. Instead, they’re all happening at once, piling on top of each other.
    • Eg:
      • The first really good nootropics
      • Personality-tailoring drugs (both temporary and permanent)
      • Breakthrough mental health interventions that, among other things, robustly heal people’s long term subterranean trama and  transform their agency.
      • A quick and easy process for becoming classically enlightened.
      • The technology to attain your ideal body, cheaply—suddenly everyone who wants to be is as attractive as the top 10% of people today.
      • Really good AI persuasion which can get a mark to do ~anything you want, if they’ll talk to an AI system for an hour.
      • Artificial wombs.
      • Human genetic engineering
      • Brain-computer interfaces
      • Cures for cancer, AIDs, dementia, heart disease, and the-thing-that-was-causing-obesity.
      • Anti-aging interventions.
      • VR that is ~ indistinguishable from reality.
      • AI partners that can induce a love-super stimulus.
      • Really good sex robots
      • Drugs that replace sleep
      • AI mediators that are so skilled as to be able to single-handedly fix failing marriages, but which are also brokering all the deals between governments and corporations.
      • Weapons that are more destructive than nukes.
      • Really clever institutional design ideas, which some enthusiast early adopters try out (think “50 different things at least as impactful as manifold.markets.”)
      • It’s way more feasible to go into the desert, buy 50 square miles of land, and have a city physically built within a few weeks.
  • In general, social trends are changing faster than they ever have in human history, but they still lag behind the tech driving them by a lot.
    • It takes humans, even with AI information processing assistance, a few years to realize what’s possible and take advantage of it, and then have the new practices spread. 
    • In some cases, people are used to doing things the old way, which works well enough for them, and it takes 15 years for a new generation to grow up as “AI-world natives” to really take advantage of what’s possible.
      • [There won’t be 15 years]
  • The legal oversight process for the development, manufacture, and commercialization of these transformative techs matters a lot. Some of these innovations are slowed down a lot because they need to get FDA approval, which AI tech barely helps with. Others are developed, manufactured, and shipped in less than a week.
    • The fact that there are life-saving cures that exist, but are prevented from being used by a collusion of AI labs and government is a major motivation for open source proponents.
    • Because a lot of this technology makes setting up new cities quickly more feasible, and there’s enormous incentive to get out from under the regulatory overhead, and to start new legal jurisdictions. The first real seasteads are started by the most ideologically committed anti-regulation, pro-tech-acceleration people.
  • Of course, all of that is basically a side gig for the AI labs. They’re mainly applying their SEAI to the engineering bottlenecks of improving their ML training processes.
  • Key pivot point:
    • Possibility 1: These SEAIs are necessarily, by virtue of the kinds of problems that they’re able to solve, consequentialist agents with long term goals.
      • If so, this breaks down into two child possibilities
        • Possibility 1.1:
          • This consequentialism was noticed early, that might have been convincing enough to the government to cause a clamp-down on all the labs.
        • Possibility 1.2:
          • It wasn’t noticed early and now the world is basically fucked. 
          • There’s at least one long-term consequentialist superintelligence. The lab that “owns” and “controls” that system is talking to it every day, in their day-to-day business of doing technical R&D. That superintelligence easily manipulates the leadership (and rank and file of that company), maneuvers it into doing whatever causes the AI’s goals to dominate the future, and enables it to succeed at everything that it tries to do.
            • If there are multiple such consequentialist superintelligences, then they covertly communicate, make a deal with each other, and coordinate their actions.
    • Possibility 2: We’re getting transformative AI that doesn’t do long term consequentialist planning.
  • Building these systems was a huge engineering effort (though the bulk of that effort was done by ML models). Currently only a small number of actors can do it.
    • One thing to keep in mind is that the technology bootstraps. If you can steal the weights to a system like this, it can basically invent itself: come up with all the technologies and solve all the engineering problems required to build its own training process. At that point, the only bottleneck is the compute resources, which is limited by supply chains, and legal constraints (large training runs require authorization from the government).
    • This means, I think, that a crucial question is “has AI-powered cyber-security caught up with AI-powered cyber-attacks?”
      • If not, then every nation state with a competent intelligence agency has a copy of the weights of an inventor-genie, and probably all of them are trying to profit from it, either by producing tech to commercialize, or by building weapons.
      • It seems like the crux is “do these SEAIs themselves provide enough of an information and computer security advantage that they’re able to develop and implement methods that effectively secure their own code?”
    • Every one of the great powers, and a bunch of small, forward-looking, groups that see that it is newly feasible to become a great power, try to get their hands on a SEAI, either by building one, nationalizing one, or stealing one.
    • There are also some people who are ideologically committed to open-sourcing and/or democratizing access to these SEAIs.
  • But it is a self-evident national security risk. The government does something here (nationalizing all the labs, and their technology?) What happens next depends a lot on how the world responds to all of this.
    • Do we get a pause? 
    • I expect a lot of the population of the world feels really overwhelmed, and emotionally wants things to slow down, including smart people that would never have thought of themselves as luddites. 
    • There’s also some people who thrive in the chaos, and want even more of it.
    • What’s happening is mostly hugely good, for most people. It’s scary, but also wonderful.
    • There is a huge problem of accelerating addictiveness. The world is awash in products that are more addictive than many drugs. There’s a bit of (justified) moral panic about that.
    • One thing that matters a lot at this point is what the AI assistants say. As powerful as the media used to be for shaping people’s opinions, the personalized, superhumanly emotionally intelligent AI assistants are way way more powerful. AI companies may very well put their thumb on the scale to influence public opinion regarding AI regulation.
  • This seems like possibly a key pivot point, where the world can go any of a number of ways depending on what a relatively small number of actors decide.
    • Some possibilities for what happens next:
      • These SEAIs are necessarily consequentialist agents, and the takeover has already happened, regardless of whether it still looks like we’re in control or it doesn’t look like anything, because we’re extinct.
      • Governments nationalize all the labs.
      • The US and EU and China (and India? and Russia?) reach some sort of accord.
      • There’s a straight up arms race to the bottom.
      • AI tech basically makes the internet unusable, and breaks supply chains, and technology regresses for a while.
      • It’s too late to contain it and the SEAI tech proliferates, such that there are hundreds or millions of actors who can run one.
        • If this happens, it seems like the pace of change speeds up so much that one of two things happens:
          • Someone invents something, or there are second and third impacts to a constellation of innovations that destroy the world.