"it's so crazy how men think about the roman empire, or the civil war, or history so often! i don't do that because i'm a woman. i think about poetry and literature instead, which men don't do" please stop this new girl dinner but for intellectual pursuits

is the problem here that women are not thinking about the roman empire often enough or that they are saying out loud how seldom they do

the problem here is that we're siloing interests by gender and then reinforcing them with things like these trends

"don't notice things! you might draw the most uncharitable evil implications from it!"

"actual gender differences in interests are an infohazard" is just… annoying. "girl dinner" is something different (actual siloing)

women get 40 to 50 percent of history degrees

I don't know that people have in fact successfully noticed a feature of reality in this case

and among the much larger group of people who don't get degrees in history, and using the more narrow subject of the roman empire rather than history in general?

i don't know and i don't particularly care to take a meme's word for it

people love to take a handful of observations and overattribute random noise to differences in salient demographic characteristics. they have the burden of proof on establishing there is actually a real difference they're noticing.

This is such a 2010s way of having this argument, here’s a prediction market, place your bets.

Mostly I like the hypothetical scenario dialogue here. (Some blanks filled in by me.)

Customer: “I’ve seen the lovely work you’ve done with white lilies, could you please make a similar display for me?”
Vendor: “Yes, of course, here’s our price sheet.”
Customer: “Thanks – we plan on putting this display up in our church on Easter morning!”
Vendor: “Neat! I’ll get back to you with a design for you to review.”
—
Vendor: “Here’s the design.”
Customer: “I dunno … it’s just not capturing the majesty of Christ’s resurrection, you know?”
Vendor: “So … it needs to be bigger?”

I’m a secular Jew and wouldn’t care. But Orthodox Jews are supposed to refuse to set foot in churches, or refuse to even say the word “Christ” lest they be seen as implicitly supporting Christianity. There’s a rule in the Talmud that Jews shouldn’t do any business with those of foreign religions (I think including Christians) on or within three days of the pagans’ religious holidays, lest they seem to be supporting or participating in them.

I think the sort of Orthodox Jew who takes all of these restrictions seriously would be very unhappy at the government forcing them to prepare an Easter bouquet, and I would consider it deeply evil if the government tried to punish them for refusing.

You really can't overstate how little anti-worker rhetoric has changed over a hundred fifty years.

(source)

I will never understand the human urge to leap from “lots of people have thought this was true over a long period” to “and therefore it must be false”.

(one could argue this graph shows high workforce participation in 1948, making it hard for participation to have been declining since 1894. But wages were rising very quickly from 1894-1948, suggesting it was taking more and more money to keep the same number of people in the workforce)

okay but 1) that chart only covers about half the period of the original set of quotes and 2) the repeated claim wasn’t “there has been an 8 percentage point reduction in the prime-age male labor force participation rate over a 66 year period.”

i mean, “nobody wants to work anymore” is not a specific claim like “there has been an 8 percentage point reduction in the prime-age male labor force participation rate over a 66 year period” so we can always quibble about what the former means and its relation to the statistics, but many--not all, but many--reasonable interpretations of the sentence “nobody wants to work anymore” are not “the prime age male labor force participation rate has been falling by about 0.12 percentage points per year for as long as I can remember.” OP’s complaint might be poorly defined, but it’s not self-evidently invalid, and this statisic isn’t exactly a clanging refutation of the original (admittedly ambiguous) claim

another important dimension is that this statistic does not indicate why the prime age male labor force participation rate has been declining so steadily, while a key component of the quotes OP is talking about is attributing an apparently widespread phenomenon specifically to laziness/not wanting to work hard/social welfare programs/etc., instead of specifically to (as OP implicitly contends) bad working conditions and/or pay. your chart also doesn’t refute that, and ignores a key component of the original post! that strikes me as somewhat dishonest. insofar as the original post contains a provably true or provably false claim (debatable), the claim is that people don’t want to work due to some set of social factors, not people don’t want to work, full stop.

this isn’t really a very good refutation of anything, and depending on how you interpret the data might even be taken as reinforcing, not undermining, OP’s point.

I mean one thing that list of quotes does indicate is that some complaints are perennial regardless of general cultural conditions or the state of the social safety net (1894-2022 covers a wide variety in both), which means that appeals to those factors as a cause of listlessness in the working population may, in fact, not be entirely in good faith! Cf. perennial complaints by Romans of Romans’ moral and spiritual decline leading to a weakening of the state, which started, oh, about 200 years before the Roman Empire reached its maximum extent of wealth and power.

I will say I do believe, at almost every time, that "no one wants to work". That's why we call it work and give people money for it.

Sure, there are people with jobs that are so intrinsically rewarding that they actually want to do them—I'm often one of them. (Although there's the 10-20% of the job that's "actual work" and I don't want to do that!)

And sure, people have a psychological need to feel like they're accomplishing something and creating something, but that doesn't necessarily line up with what people will pay you for.

Fundamentally, work is something that people need a motivation to do, and that's why they get paid for it. So it's basically always true to say that people "don't want to work", and the question is whether you want to pay them enough to change their minds.

Re: 1 - please see the sentence in parentheses I wrote above.

Re: 2 - It sounds like you think this is only valid if you can see into the souls of men and find something marked “laziness” or “being a bad person who hates work for no reason”. But what all of these headlines are saying is just “nobody wants to work anymore”.

Clearly that’s not literally true, since 80-something percent of people are still employed. But less hyperbolically, it sounds like they’re saying “people are less willing to work a given amount for a given price than they used to be”.

This is totally true! Over the past 150 years, number of hours worked has continuously gone down!

Even as wages continuously went up (until ~1970, see here for more)

And at the same time, people were (as per the original graph) dropping out of the labor force.

I can’t really see any reason to interpret those statements as completely accurate observations of the fact that, over the past 150 years, people have been gradually less willing to work, even as hours go down and pay goes up.

This doesn’t seem mysterious - I think it’s a natural result of the country getting richer and people not having to work as much. Some of that is welfare, but some of it is just being able to support yourself comfortably on fewer hours of work. And some of it is probably cultural change downstream of the first two.

One easy way to see this is that everyone has a stereotype of illegal Mexican immigrants as really hard-working. I think this makes sense - they don’t have access to a lot of the same opportunities to do well with medium amounts of work, opportunities that America’s rising wealth has given us over the past century or so - and so they work as hard as native-born Americans did in 1870 or whenever.

I agree that optimization power is not *necessarily* correlated with specific goals. But why wouldn't mesa-optimizers, contingently, have a specific goal. Presumably we're running gradient descent on some specific loss function, like "number of paperclips produced", and then mesa-optimizer inherits some proxy for that.

I agree humans aren't like that, and that this is surprising.

Maybe this is because humans aren't real consequentialists, they're perceptual control theory agents trying to satisfy finite drives? EG when we're hungry, our goal becomes to find food, but we don't want to tile the universe with food, we just want to eat 3000ish calories and then we're done. We have a couple of other goals like that, and when we've accomplished all of them, most people are content to just hang out on the beach until something else happens.

Might gradient descent produce a PCT agent instead of a mesa-optimizer? I don’t know. My guess is maybe, but that optimizers would be more, well, optimal, and we would get one eventually (either later in the gradient descent process, or in a different lab later). My guess is evolution didn’t make us optimizers because it hasn't had enough time to work with us while we've been intelligent. If we got locked at 20th century technology forever, I think it might, after a few million years, produce humans who genuinely wanted to tile the universe with kids.

"Even if the base objective is fixed, there's no reason to think an AGI's inner objectives won't evolve over time, or adapt in response to new experience."

Wouldn't the first thing a superintelligence with a goal did be to make sure its goal didn't drift?

If its world model is malleable during this "childhood" phase, why not its values, too?  It has no reason to single out a region of itself labeled $MESA_OBJECTIVE and make it unusually averse to updates after the end of training.

I think this is where the deception comes in. If the mesa-optimizer is smart and doesn't want people (or other parts of itself) changing its values, it will take steps to stop that, either by lying about its values or fighting back.

Maybe this is because humans aren't real consequentialists, they're perceptual control theory agents [...]

Might gradient descent produce a PCT agent instead of a mesa-optimizer? I don’t know. My guess is maybe, but that optimizers would be more, well, optimal, and we would get one eventually

I think this idea that "real consequentialists are more optimal" is (sort of) the crux of our disagreement.

But it will be easiest to explain why if I spend some time fleshing out how I think about the situation.

What are these things we're talking about, these "agents" or "intelligences"?

First, they're physical systems. (That far is pretty obvious.) And they are probably pretty complicated ones, to support intelligence. They are structured in a purposeful way, with different parts working together.

And this structure is probably hierarchical, with higher-level parts that are made up of lower-level parts. Like how brains are made of neuroanatomical regions, which are made of cells, etc. Or the nested layers of abstraction in any non-trivial (human-written) computer program.

At some level(s) of the hierarchy, there may be parts that "run optimization algorithms."

But these could live at any level of the hierarchy. They could be very low-level and simple. There may be optimization algorithms at low levels controlled by non-optimization algorithms at higher levels. And those might be controlled by optimization algorithms at even higher levels, which in turn might be controlled by non-optimization ... etc.

Consider my computer. Sometimes, it runs optimization algorithms. But they're not optimizing the same function every time. They don't "have" targets of their own, they're just algorithms.

They blindly optimize whatever function they're given by the next level up, which is part of a long stack of higher levels (such as the programming language and the operating system). Few, if any, of the higher-level routines are optimization algorithms in themselves. They just control lower-level optimization algorithms.

If I use my computer to, say, make an amusing tumblr bot, I am wielding a lot of optimization power. But most of my computer is not doing optimization.

Python isn't asking itself, "what's the best code to run next if we want to make amusing tumblr bots?" The OS isn't asking itself, "how can I make all the different programs I'm running into the best versions of themselves for making amusing tumblr bots?"

And this is probably a good thing. It's hard to imagine these bizarre behaviors being helpful, giving me a more amusing tumblr bot at the end.

Which is to say, "doing optimization well" (in the sense of hitting the target, sitting on a giant heap of utility) can happen without doing optimization at high abstraction levels.

And indeed, I'd go further, and say that it's generically better (for hitting your target) to put all the optimization at low levels, and control it with non-optimizing wrappers.

Why? The reasons include:

Goodhart's Law

• ...especially its "extremal" variant, where optimization preferentially chooses regions of solution space where the assumptions behind your proxy target break down.
• This is no less a problem when the thing choosing the target is part of a larger program, rather than a human.
• Keeping optimization at low levels decreases the blast radius of this effect.
• If the things you're optimizing are low-level intermediate results in the process of choosing the next action at the agent level, the impacts of Goodharting each one may cancel out. The agent-level actions won't look Goodharted, just slightly noisy/worse.

Speed

• Optimization tends to be slow. In a generic sense, it's the "slow, hard, expensive way" to do any given task, and you avoid it if you can. (Think of System 2 vs System 1, satisficing vs maximizing, etc)
• To press the point: why is there a distinction between "training" and "inference"? Why aren't neural networks always training at all times? Because training is high-level optimization, and takes lots of compute, much more than inference.
• Optimization gets vastly slower at higher levels of abstraction, because the state space gets so much larger (consider optimizing a single number vs. optimizing the entire world model).
• You still want to get optimal results at the highest level, but searching for improvements at high level is very expensive in terms of time/etc. In the time it takes to ask "what if the entire way I think were different, like what if it were [X]?", for one single [X] , you could instead have run thousands of low-level optimization routines.
• Optimization tends to take super-linear time, which means that nesting optimization inside of optimization is ultra-slow. So, you have to make tradeoffs and put the optimization at some levels instead of others. You can't just do optimization at every level at once. (Or you can, but it's extremely suboptimal.)

------

When is the agent an "optimizer" / "true consequentialist"?

This question asks whether the very highest level of the hierarchy, the outermost wrapper, is an optimization algorithm.

As discussed above, this is not a promising agent design! There is an argument to be had about whether it still could emerge, for some weird reason.

But I want to push back against the intuition that it's a typical result of applying optimization to the design, or that agents sitting on giant heaps of utility will typically have this kind of design.

The two questions

1. "Can my computer make amusing tumblr bots?"
2. "Is my computer as a whole, hardware and software, one giant optimizer for amusing tumblr bots?"

have very little to do with one another.

In the LessWrong-adjacent type of AI safety discussion, there's a tendency to overload the word "optimizer" in a misleading way. In casual use, "optimizer" conflates

• "thing that runs an optimization algorithm"
• "thing that has a utility function defined over states of the real world"
• "thing that's good at maximizing a utility function defined over states of the real world"
• "smart thing" (because you have to be smart to do the previous one)

But doing optimization all the way at the top, involving your whole world model and your highest-level objectives, is very slow, and tends to extremal-Goodhart itself into strange and terrible choices of action.

It's also not the only way of applying optimization power to your highest-level objectives.

If I want to make an amusing tumblr bot, the way to do this is not to ponder the world as a whole and ask how to optimize literally everything in it for maximal amusing bot production. Even optimizing just my computer for maximal amusing bot production is way too high-level. (Should I change the hue of my screen? the logic of the background process that builds a search index of my files??? It wastes time to even pose the questions.)

What I actually did was optimize just a few very simple parts of the world, a few collections of bits on my computer or other computers. And even that was very time-intensive and forced me to make tradeoffs about where to spend my GPU/TPU hours. And then of course I had to watch it carefully, applying lots of heuristics to make sure it wasn't Goodharting me (overfitting, etc).

To get back to the original topic, the kind of "mesa-optimizer" we're worried about is an optimizer at a very high level.

It's not dangerous (in the same way) for a machine to run tiny low-level optimizers at a very fast rate. I don't care how many times you run Newton's method to find the roots of a one-variable function -- it's never going to "wake up" and start trying to ensure its goal doesn't change, or engaging in deception, or whatever.

And I am doubtful that mesa-optimizers like this will arise, for the same reasons I am doubtful that the agent will do optimization at its highest level.

Once we are pointing at the agent, or a part of it, and saying "that's a superintelligence, and wouldn't a superintelligence do . . . ", we're probably not talking about something that runs optimization.

You don't spend your optimization budget at the level of abstraction where intelligence happens. You spend it at lower levels, and that's what intelligence is made out of.

I guess I’m pretty confused by this response.

Your computer isn’t optimizing for Tumblr bots. But your computer can’t create Tumblr bots; if you left it on the desk for a million years, no Tumblr bots would be made. The you-yourcomputer system *is* optimizing for Tumblr bots, with the optimization layer being in your brain. You’re not technically a Tumblrbot-maximizer, but presumably this is why you don’t create that many Tumblr bots and you spend most of your time doing other things.

A version of you who (let’s say because of weird childhood conditioning experiments) wanted with all your heart to produce as many Tumblrbots as possible probably would be very good at creating lots of Tumblr bots (as part of a you-yourcomputer system).

Even if you were a Tumblrbot maximizer, you wouldn’t spend much time fiddling with your screen brightness settings or the lowest-layer code of your operating system. But that’s because, in your effort to build as many Tumblr bots as possible, you rationally decided this wasn’t a good use of your time. If it was a good use of your time (say your screen was so dim that you couldn’t program on it), you would probably fix that.

One thing I find confusing is that it sounds like you expect the lowest levels of an AI to be optimizers, but the highest levels not to be. I would expect the opposite. My cerebellum neurons, when deciding when to fire, aren’t thinking “what firing pattern would cause me to get the most sex/food/status right now?”, they’re just running dumb firing rate algorithms (which on evolutionary timescales did correlate with getting sex/food/status, I guess, and which probably coincidentally but not intelligently optimize some information theory parameter). Meanwhile, on the highest and most abstract level, I do often make deliberate plans that result in getting sex/food/status as efficiently as possible. So I would describe myself as a top-level optimizer built out of dumber non-optimizers.

It sounds like you have a different intuition, or else I’m misunderstanding you, so I’m curious where this is breaking down.

The vast majority of post-op trans QoL studies use sample sizes of less than 30 people, unless it’s a meta-study that’s compiling the work of multiple surveys. I don’t think you want to challenge the findings of almost every single study of trans healthcare.

1. Deciding whether or not to trust a study’s results for reasons of ideological convenience rather than scientific validity is not science - it’s stupid and shortsighted.
2. I don’t need to rely on studies to determine if trans people should have the right to modify their bodies as they see fit - everyone should have that right.
3. Here are some larger-scale cohort studies (tho the largest is a survey, not a study) that examine trans healthcare outcomes with sample sizes in the hundreds or even thousands.

This is an interesting point.

I don’t have the exact link, but I was reading about the mathematics of polls, and even small-ish polls have low margins of error. If a poll of 50 people finds that 70% support Democrats, then the real number that you’d get with 500 or 5000 people is probably very close. I’ve found this myself when I run online surveys - after the first 20 people, you usually have a really good idea how they’ll turn out.

In theory, you should be able to apply this intuition to studies. A study is just a poll taken after you’ve done some randomized intervention (or measured some other independent variable). But I agree with some of the people responding here that I don’t trust any study done with fewer than 50-100 people, and I feel most confident when there are 10000+

I think there are two things going on here:

- It’s easier to do statistical trickery with fewer people. For example, if you only have twenty people, then taking out two outliers can change the entire result. You can always find a reasonable-seeming rule which changes the results to what you want, like “we only counted people who gave full answers to all questions” vs. “we counted any answer as long as it was intelligible”. Most scientists try to do sketchy things like that, and so you need a study big enough to make this hard.

- Large studies are good not because the sample size is bigger, but because anything that big is going to have a really professional operation with statisticians who know what they’re doing, independent auditors, etc. And since they’re going to be important, they’ll be critiqued by lots of people who will find any flaws before they get to you.

I’m not sure if either of these is true, or how to weight these factors, but I feel like something like this must be going on.

A poll is structurally different from what most studies are doing. (But not all!)

In a poll, you want to know “on average, what fraction of respondents match category X?” And the statistics of that are very easy and very well-understood and scale very well. It takes relatively few measurements to get a good estimate of a single population mean. (Though even then, a poll of thirty people gives you a margin of error of like 18%. You want like a hundred people to get your margin of error below 10%. Fifty isn’t enough even for this; that’s why polls survey 300 to 1000 people.)

But most studies aren’t just surveys of the prevalence of some single behavior. They’re trying to find relationships among different measurements, and that dramatically increases the amount of error you can have.

A way to see this is to imagine you’re computing, say, the correlation of income and height. If you survey a hundred people, then your measurement of average income is going to have some error, and your measurement of average height is going to have some error. But when you’re looking at the correlation, you can’t just compare those averages. You have to chop your data up into pieces.

You need to have a good idea of the income of the top quarter of your distribution, and of the second quarter, and the third, and the fourth—now your groups are a quarter of the size, so the error in each of them doubles. And you have to do the same thing for your heights. (You’re actually chopping the data up even more than this, but also you’re making some assumptions that improve it; this is an intuition pump, not an exact description of the statistics.)

And then you have your messy measurements of high incomes, and of high heights, and you have to compare those. And I wanna say that multiplies the earlier errors? Changing just a couple of data points can dramatically change the correlation even if it doesn’t change the group averages that much. (This is why removing a couple outliers can make such a huge difference.) You need a much bigger sample to get a confident correlation than to get a confident point estimate.

Now it’s true that a lot of studies do, in fact, run with thirty WEIRD undergraduates. This is one reason why a lot of studies are crap!

And the more that (1) you’re trying to detect small effects and (2) you’re studying a domain where there are lots of complicated factors, meaning that there’s a lot of variation that doesn’t depend on the thing you’re studying, and (3) the more you want to statistically control for some of those factors, the bigger this problem is and the more data you’ll need.

I think none of this applies to RCTs, where you really are just doing polls - in the active drug group, poll, how many of you got better - and in the placebo group, poll, how many of you got better? But I don’t trust 30 person RCTs any more than 30 person correlational studies.