Maybe rich people are happier than poor people because they don't have to deal with sh!t all the time. You know, that stuff about the cognitive burden of poverty, etc.
Happiness is an ambiguous word. It cannot be defined precisely. There just is no unique definition. Why then do you feel free to use it as if it were a scientific term like velocity or force that can be quantified, measured, compared, i.e. studied ??
Is this because of multi-armed bandits? The genes that made people more likely to cross the Bering strait did fairly well, all in all.
> You seek unpredicted reward, but by definition you can never get this consistently
It's not technically 'consistent' but looking through the middle aisle of Lidl tends to be a reliable way to find something both unprediced and rewarding to acquire.
I believe a recent philosophical musing by Theodore Dalrymple, aka the retired prison doctor Anthony Daniels, is relevant here. He reflects on the difference between an interesting and boring life, which is arguably more meaningful than the happiness/unhappiness dichotomy, with reference to a former patient:
I remember a patient who had believed for years that he was at the center of a giant conspiracy involving the great powers of the world. I declined even to try to treat him because if the treatment worked, which was possible but not certain, and he lost his delusions, he would be left with the realization that he had wasted years on complete nonsense, and that it was now too late to restart his life. Moreover, he would be obliged to recognize that, far from having been a person of immense importance who had been the focus of the attentions of the great ones of this earth, he had all along been an insignificant little man in rather wretched lodgings, eking out his existence from hand to mouth. His life with his delusions had, at least, been full of interest and incident; and every time an aircraft flew overhead, or a car passed in the street, it was spying upon him. The enemy kept him busy, evading the poison they were insinuating into his food via the supermarket and the rays they were directing at him via the electric plugs; every trip to the local shops was fraught with danger, necessitating the exercise of the utmost caution. Every time he returned home intact from the purchase of a pint of milk, he could congratulate himself on having outwitted the enemy.
What life could I offer him by comparison with this? Only poverty and boredom.
1. I semi-famously am known for having once been broke for a really long time. At some point I got some news that essentially translated to "by your standards, you have won a very large lottery. Congrats!" in a way that was fairly sure to pay off (say, 90%).
Jobs absolutely take a little time to kick in, and pay does as well. But both in this case and a later similar bump, the actual "implementation" of the changes was a big deal, as were the paychecks. This is sort of nitpicking, but I expect that (if other people match my own experiences) the actual "hit" of the billion dollars arriving is probably bigger than you think, prediction or no, because your mind doesn't have a "this is pretty sure and gonna happen" setting that's strong enough to offset a billion dollar deposit in your account.
2. Ignoring 1:
At some point I went from "begging uninterested people to read my stuff" to "hundreds or thousands of hits an article" to "guaranteed thousands, sometimes tens of thousands" in terms of article performance. Over that time, the relative value of an article view dropped enormously in terms of how much excitement it gave me.
But it's not just that; that's sort of expected, right? You get used to things. But I'm also *getting used to updates themselves*, which is weirder. In the past few months my average hit count on an article has probably jumped 20-40%. I noted it happened, and it was a "new updateable event" but that this point "new updateable event" itself has become normal. I'm used to it.
I intellectually know I should feel shitty about myself when an article does 20-30k views, which is pretty good, and I don't get much emotional bump from it. But it's also involuntary; I can adjust my "academic" understanding of what's happening and say "this is a pretty big deal" but I can't get my lizard-brain to come along; I'm spoiled.
3. I am a verbal person and attracted to my wife, which means my wife hears a lot of positive stuff about her physical appearance. I'm also a fan of matrimony and reflexively speak a lot, so she gets proposed to a bunch. In terms of "women who hear nice stuff from their husbands a lot" I'd say she's probably top 1% or so.
That said, she's fully updated on it; she accurately views it as happy babble. But she also, in contrast to my hits, isn't *wrong* to discount it. If I withheld the words for a while then brought them back, she'd probably get a hit from it; if I was naturally reserved and didn't say affectionate things a lot to begin with and started she'd probably feel it. But they'd still just be words in either case, perhaps representative of something real behind the curtain or perhaps not.
Since she gets the words a lot, she can basically go "Oh, that's RC being a big fan of marriage, introverts, and boobs, it doesn't really mean anything spectacular and special" where otherwise she might see that less clearly. And she can more accurately judge my love by more "real" measures, like if I do dishes ever or something.
Everything is new and fresh and then you acclimate to/figure out the reward system, and it is still fun for a while maybe, but all the excitement is gone.
Maybe there is a kind of reward system in our brains for optimizing or figuring out new and unpredictable reward systems?
<i>So how come predicting you would get the money mostly cancels out the goodness of getting the money, but predicting you would get the Ferrari/dinner doesn’t cancel out the goodness of the Ferrari/dinner?</i>
Surely the answer is that the goodness isn't getting the money itself, but getting the ability to purchase things which the money provides. If you inherit a billion dollars but it's all locked away in a Swiss bank account which you can't access, your newfound wealth won't actually be any good to you, because you can't use it to purchase anything. If the world financial system collapses and everyone reverts to barter, having billions of cash isn't going to do you any good, because you can no longer use it to buy stuff. And by the same token, if you have money but are too busy to actually spend it, that money isn't going to be doing you any good, and so it's not surprising if it doesn't make you any happier.
Surely I'm misinterpreting it, but it looks like that graph of widowhood shows a statistically significant decline in life satisfaction the year before widowhood has actually occurred. Am I missing something, or is there something that could cause that effect?
Regarding your Sinclair Method question, it’s currently believed that dopamine cannot be produced in the reward center while the mu opioid receptor is blocked, so that’s ultimately what breaks the learned reward.
The getting-roses-without-asking thing isn't joyful because it's unexpected. It's because the motivation for the roses is that they were thinking about you while you weren't around. If you ask for it, then they do it as a chore. But if they do it without asking, spontaneously, it's a sign that they were thinking about you without prompting and thought to do something nice. You want to be with people who think about you when you're not there in a happy way.
1) The 21st century (lower) middle-class person and the medieval serf have a huge objective difference in their quality of life, but similar positions in the social status hierarchy. Maybe your relative position in such a hierarchy (or system of hierarchies, because status is complicated) is a direct input on your feeling of safety in the world, *and* on your expectations against which your objective quality of life is measured?
2) The romantic love phase typically lasts longer if the couple spends significant time apart regularly. This seems to point to the kind of effect you're talking about, with the brain updating slowly over time, and in this case, updating back to some extent.
3) The part about some people getting addicted to terrible relationships, as a way to get any signal at all, points towards the experience of boredom being generated as a negative signal for a too much predictability. In layman terms, maybe we're wired to have a certain drive to explore and experiment, and our brain generates negative signals when that is not happening. Getting stuck in an unpredictably bad relationship would be a failure mode of this mechanism.
4) Just a random fact: I once got a full month of top-of-the-world euphoria from getting B12 vitamin shots, after letting the levels fall too low for years due to bad diet.
5) On the AI part, doesn't this point towards an easy way to keeping AI safe, by keeping training and production use separated into the future?
6) Why is there "a lingering kind of happiness which can be enjoyed even after prediction error has been corrected"? My first impression here is that the mental models of brains updating on their priors based on their inputs and generating feelings as it happens, is a bit too much of a blank slate kind of model. Organisms shaped by evolution have built-in goals, the first of which is maintaining homeostasis, which extends into all sorts of things like feeling healthy and strong, and up to maintaining good social connections. So there's nothing really strange about these kinds of things feeling intrinsically good in durable ways. The opposite would actually be surprising.
I have a tiny nitpick. The graph for the marriage happiness over time seems to suggest that people recover happiness past the happiness of their marriage. Now, maybe they controlled for this, but my understanding is that most people are significantly more likely to die soon after their spouse does (the widowhood effect). This makes me pretty suspicious that what might be driving a lot of the gains in the graph would be that people in good relationships are just dying, and more of the ones who survive are the ones who really are better off without their spouse
Do people have subjective experience of the hedonic treadmill? I get that the studies are pretty conclusive, but I've been consistently sad in periods when my life was bad and consistently happy in periods when my life was good, without any real adjustment.
Not knowing if I'm going to make rent kept being stressful for years, and watching my dog and daughter enjoy the forest together is wonderful every time.
Hi, I'm the author of the aforementioned post. Sorry for getting the neuroscience wrong - the mistake i made was just parroting what I heard on the Andrew Huberman podcast, and related it to my own experiences studying religions.
I've spent much of the last few years trying to understand religions in terms of neuroscience; both neurotransmitters but also the predictive processing model. What Huberman kept saying was that serotonin was 'connected to appreciating rewards in the present', and this gibed with my own experience. It also seemed to line up with stuff I'd been reading in a book "stop fixing yourself, wake up, all is well". The author talks about "earthly rewards" vs "spiritual rewards." Getting money would be an 'earthly reward' whereas 'time with friends' or 'a good mean' would be 'spiritual rewards.'
I'm wondering now if these are the difference between instrumental and terminal goals. Food is a probably something we're hardwired to want, maybe with that unalterable prediction of "i predict i am full." Driving a ferrari around feels fun (at least for me) because of the deep rumblies and the visual aesthetics - although i guess this gets old for some people, much sooner than the meal does.
We clearly aren't hardwired to want money (i.e. it wasn't in our evolved environment) so maybe that's why it feels different - because it has to do with expected future rewards. vs actual present rewards?
I've definitely learned how to trigger some reliable good feeling in my brain; it's not deeply pleasurable, but it is calming and feels positive. Maybe I've learned how to trigger endogenous opioids?
My goal in sharing the post was not to make claims about neuroscience, so much as to try and share the method I've learned for more or less reliably poking the 'feel good' center in my brain in ways that seem nondestructive, and relating this to both predictive processing and the concept that 'good' is a thing your brain can learn to recognize and move towards but only if you believe 'good' is meaningful.
At the risk of making the same mistake over again, the latest concept that makes sense to me would be:
- valence is how good you feel, your brain computes a valence manifold based upon present and expected future rewards
- dopamine represents something like the gradient (slope) of the valence manifold; the slope corresponds to the expectation of future rewards. Maybe anything that is an instrumental good (money, status, sex) leads to dopamine release?
- opioids represent something like the valencene directly, i.e. how well your needs are met in the present moment. Maybe opioids are released when you look at something in your life and think, wow this is wonderful just as it is, like, say, my kids, my wife, my family, friends, etc?
- i now suspect serotonin is doing something like 'flattening' or stretching out the manifold, which has the effect of calming you down and, in the limits, possibly stretching it so flat that you fee emotionally flat, like an unmotivated zombie, with no direction seeming any more appealing than any other
In any event i'll make more of an effort to accurately lay out what i do and don't understand, and i'll try to avoid taking just one source as gospel on things as complicated as my brain
I've read Andres' stuff before and can't shake the feeling that I'm reading pseudoscience. It's too simplistic to essentially say there's a shape to our emotional states. The same crowd talks about logarithmic scales of pleasure/pain: The eighth dakha of Buddhist meditation, the highs of DMT and falling in love versus the crippling lows of cluster headaches. The richness of our experience wouldn't add up.
I can remember reading an article on aeon that really stuck with me. It was about how the liveliness of our brains correlates with its dissonance/entropy/incompressibility. When we are asleep, for example, our brain waves are very ordered/consonant/harmonic. One can basically compress the brain state digitally, then higher compression ratio means lesser consciousness, and vice versa lower compression ratio is more conscious. And this relates to humans on higher levels of the emergence stack (a la Tim urban), if you have ever heard the cheering crowd of sleepwalking society like North Korea, it's monotone and slow-wave harmonic. Whereas a similar celebratory cheer in the USA would be noisy, dissonant, full of airhorn noises and such, incompressible.
This wouldn't fly with Andres' idea that consonance equates with Valence. He cites studies saying high frequency harmony is linked to high valence, and low frequency harmony linked to high arousal. I'd imagine, at the least, that low frequency harmony is asleep. But I need to look more into this. And also, what about split brown patients? Might they have only half the capacity for Valence under that model? Or is it actually supporting his idea because it's something more to do with electromagnetic fields and not electrochemical circuitry? I wonder what seizures look like in a brain scan.
> Maxing out serotonin levels mostly seems to cause a blunted state where patients can’t feel anything at all.
Then why is MDMA so euphoric?
It's strange, dopamine and norepinephrine reuptake inhibitors like methylphenidate (Ritalin) seem to offer a subjective experience broadly similar to dopamine and norepinephrine releasing agents like dextroamphetamine. But serotonin releasing agents offer an experience completely incomparable to serotonin reuptake inhibitors. Do you know why this is? My guess (which might be total nonsense) is that it has something to do with where in the brain the serotonin is being released. Inhibiting reuptake will increase serotonin in areas where it is already being released a lot whereas a releasing agent could possibly cause it to be in an area where almost none was previously released
One neuroscientific perspective on this is that in order for dopamine to track reward prediction *error* (RPE), it is logically necessary that some other piece of neural circuitry track reward prediction *per se*, often called "value." Those of us who think that dopamine is computing RPE on a moment-by-moment basis (the first derivative of value; see Kim, Malik et al., Cell, 2020) therefore generally also believe that some other part of the brain, especially the ventral striatum (aka nucleus accumbens) and perhaps also the prefrontal cortex, maintains an estimate of value that gets updated by dopamine. And indeed, there are dozens of papers reporting that neural firing in these brain regions correlate with value over and above RPE.
I think this explanation differs from the one you offered because rather than seek some other neuromodulator to account for the "companionate love” phase of a relationship, you can just consider that phase to be the psychological correlate of the brain's internal value estimate as it is instantiated in this cortical/striatal circuitry. Though I certainly wouldn't rule out other options, especially intracellular mechanisms in these areas, because neural firing on these very long timescales is dubious.
Lastly, on the abusive relationship point, you might be interested (or perhaps you already know) that BF Skinner famously observed that "variable ratio" reward schedules lead to greater responding/addiction than other kinds of schedules. I guess this is what you're getting at when you say, "Everyone has some weird function that doesn’t correspond to normal addition, and maybe for some people dating a person who gives good vs. bad signals exactly 50% of the time is the only way to get that function in the black." I would amend this slightly to say that some people are more sensitive to the volatility (which is very much a function of RPE directly), and others to the value itself (the integral of RPE).
This post uses money and luxury consumption as hypotheticals to model the brain's reward system. They are things we often think of as rewards, so intuitively fit the subject. I wonder, however, what the picture looks like when we focus on productive work, something many people spend most of their waking time doing.
Perhaps what Tyler Cowen calls The Production Function works (generically) due to interesting work providing unpredictable rewards frequently. When you start a project, perhaps your unconscious predicts a somewhat negative outcome. ("This project will be too difficult. You will never finish. You suck, loser!") You feel the weight of this pressure, but if you persevere and do your work you are rewarded by your pessimistic unconscious being pleasantly surprised. ("Holy shit! I can't believe you did that! I guess you don't suck after all!)
This could explain why hard work can be so rewarding regularly, and also why rewarding work often feels hard.
I'm surprised no one has mentioned the relevance of gratitude. Gratitude is basically a way to keep alive the prediction of losing every good thing you have. It's a way to keep unpredicted reward alive.
Proust's great novel is basically on the theme of how habit voids all pleasures and how the object of desire never lives up to the expectation built in desire. His narrator is a bit like the man who can't find a companionate love; he feels passionate love only when it is sprung from the seed of outrageous jealousy. Happiness does seem to only arrive in unexpected, fleeting events.
Scott, I sure wish you had time to read that novel but at 3300+ pages I'm guessing you probably won't anytime soon.
I'm going to sound like a broken record, but this is in Soren Kierkegaard's Either/Or. The two halves of the book each embrace one kind of happiness and eschew the other. Book 1 ("Either") starts with having to rotate between different kinds of aesthetic enjoyment to continue to feel happiness and escalates to having to be in an abusive relationship to feel anything at all, even for just one night. Book 2 ("Or") is a looong treatise on how that's not actually happiness, real happiness is a good, steady marriage and a prosperous, predictable life, and it's so boring barely anybody reads it all the way through.
This feels like the archetypal SSC/ACX post: technical analysis of romance! Drugs! Psychiatry anecdotes! Prediction error! Speculative neuroscience! AI! Seven sections! (And that's all a good thing, to be clear.)
One thing that seems worth noting here is the intersection of behavioral neuroscience and plain old behaviorism: variable interval rewards are known to be the best way to maintain behavior, which of course fits with the whole point about predictable rewards' emotional value being abraded away. But if what you become accustomed to is the general level of good things (e.g. how much/how often your partner is nice to you) rather than the good thing itself, then that suggests the way to continue giving or getting emotional reward is (a) to invest in periodic special things - surprise flowers, vacations, etc. -rather than anything predictable, even if it's less time*improvement value than e.g. more comfortable furniture, (b) to structure your life in such a way that you experience more ups and downs, even if it's on average worse.
Of course this whole process of reasoning just ends up recapitulating cliches like "buy experiences, not things" and "variety is the spice of life", but whattaya gonna do.
One thing that might help in certain parts of this post is to split up the concept "prediction" into two different concepts, "visceral expectations" versus "intellectual predictions". "Visceral expectations" are related to valence, aversion, desire, sweating, goosebumps, all that stuff, whereas "intellectual predictions" are the things that you consciously believe.
I think that intellectual predictions can impact visceral expectations a little bit—more than zero—but definitely not completely. The two can stay discrepant. For example, if you've never done cocaine, and you read in a textbook that cocaine feels really intensely good, and you completely sincerely believe the textbook, then now you have a very strong (intellectual) prediction that cocaine feels good. But you still only a weak visceral expectation that cocaine feels good. So you don't suddenly start craving cocaine the way an addict does. Then maybe you actually try cocaine yourself, and NOW the visceral expectations regarding cocaine get strongly updated.
Anyway, I think the P of RPE is visceral expectations, not intellectual predictions.
When I was in graduate school, I was a facilitator for domestic violence batterers intervention psychoeducation groups. (A mouthful, I know). It was the California-mandated program for DV offenders. I did this for about 3 years, and had nearly 4000 clinical contact hours with this cohort.
After deciding that my preferred treatment modality was psychodynamic and I began using (mostly Kleinien) interventions, I started giving this spiel to the guys.
"You could walk into a room with 500 women. 499 of them would be mostly healthy, insightful well-boundaried, nice girls who would make wonderful supportive wives and mothers. Not perfect by any means, but they would be invisible to you. The one left would be a crazy borderline with all kinds of unhealthy personality traits and maladaptive life skills and the two of you would be attracted to each other on a subconscious level like flies on sh!t. Until you spend about 2 years in therapy figuring out what that is about (hint, its probably your mom) this pattern will continue until you die."
That's how I conceptualize your patient with the abusive wife. You pointed out that people have all kinds of rubrics that do not have to add the prediction/error equation to zero. Personality pathology and the way they interact in relationships account for a lot of that.
And yes, because those patterns are so difficult to dislodge, he is more or less doomed to a miserable life. Although, like you, I never say stuff like that to them.
Also, I think the people with the decades-long relationships still having great sex are manifesting a low base-rate phenomenon wherein all the planets (which includes both parties developmental trajectories/personality traits, psychological growth over the lifespan and dumb luck) aligned and the passionate love lasted way longer than predicted. They just lucked out.
The problem is, our society tells us this lucky hand of the cards is something we should all aspire to and that you just have to scour the planet looking for your soulmate. When the correct response when you see it is just "wow. That is really cool that they found that. Good for them." The probability of finding it is like winning the lottery.
> I disagreed with the serotonin-focused explanation at the beginning of the post, but you can rescue it to be about dopamine vs. some other neurotransmitter (endogenous opioids are a popular choice). This would correspond nicely to the liking vs. wanting theory, even though technically they’re explaining different things (pleasure + motivation, vs. two different kinds of pleasure).
For what it's worth, dopamine mediating 'wanting' and opioids mediating 'enjoying' was exactly how it was taught to me back in undergrad, substantially earlier than the publication date on that link. If you're interested I can try and dig up the citation that was used at the time. (But no promises I'll be successful ofc!)
Been thinking for a while that happiness is in ratio to an imagined baseline. Becoming rich is compared to your baseline wealth, but is big enough to move the baseline. Whereas driving a Ferrari or eating cake on your birthday remains fun because it's compared to the baseline of the rest of your day where you're not driving or eating cake.
So the flower example would be the other way around; she's predicting flowers as being a natural part of a relationship, and the unpredicted absence of them is dropping below her baseline.
1. Wealth reduces dread. If you are cruising towards bankruptcy, it's hard to have a good time. Having some slack won't make you automatically happy, but your default state is more in the contentment range. Listen to the Dave Ramsey Show for gobs of examples.
2. Regarding specific neurotransmitters, maybe one should look into the timing. According to sources I have read over the years, serotonin helps with sleep. Bad sleep can lead to depression. Putting these two ideas together indicates some kind of serotonin booster near bedtime could be useful, but jacking up serotonin levels all day long could produce enough downsides to offset the benefits. From personal experience, a quarter gram of tryptophan near bedtime, or making a high starch/low protein meal my last meal of the day does indeed improve sleep.
3. For sustained highs, I had some good success with a variation on Guy-Claude Burger's instinctive eating regimen. Unlike Burger, I did cook my meat, but I got the rest of my calories from raw food's that tasted good unmixed. On such a diet, achieving meditative bliss was pretty easy. Getting enough calories, on the other hand, became difficult over time. And a mostly raw diet doesn't lend itself to a good social life. But further research in this area is worth pursuing. We eat a mostly artificial diet. Maybe much of our unease (and drug abuse) is an attempt to offset the accumulated effects of this unnatural diet. And like wealth reduces dread, a more natural diet might set our default state to something closer to what people call "happy."
"Any neuroscience article will tell you that the “reward center” of the brain - the nucleus accumbens - monitors actual reward minus predicted reward."
This claim is way too strong.
What is correct:
Reward Prediction Error (RPE) is one of the main theories on what the reward center (nucleus accumbens NAc) does. There are many situations and experiments in which this explanation fits nicely.
What is also correct:
There are many situations in which RPE does *not* fit nicely. The discussion on what the reward center really does is far from settled.
I can highly recommend the paper "Dopamine release in the nucleus accumbens core signals perceived saliency" by Kutlu et al. from last year for a different opinion. It's really well written and contains great experiments. The authors suggest that the reward center does not represent RPE, but saliency. They summarize their work with the following four points:
- NAc core dopamine only mimics reward prediction error in select reward contexts
- RPE does not model dopamine release during negative reinforcement
- Dopamine signaling in the NAc core does not support valence-free prediction error
- NAc core dopamine tracks valence-free perceived saliency in all conditions
This paper will not be the end of the debate. But RPE is not the end of the debate either.
UPDATE: After reading the full post (great post!), I really have to find some time to re-read the study by Kutlu et al., put it next to the post, and see how the post aligns with their experiments. The result might be quite interesting.
>Then the husband gets her flowers, and the wife says “it doesn’t count, you should have known without me telling you”. To this couple, gestures of affection are meaningless unless unpredictable...
No
You are completely missing the point.
You need to read “A Doll’s House” by Ibsen, or rethink it if you have.
Tl;dr. The issue is authenticity. If your proposed “wife” has any sensibility whatsoever she will recognize a gesture from the heart, just as she will reject an empty gesture. If she doesn’t have any sensibilities and you do then the flowers are a gift to yourself.
You have constructed a compelling narrative (which was fun to read) based on:
>My guess is: January 1, when you first hear you won, is the best day of your life...
Emphasis on, I GUESS.
In other words, this is no argument for a general case. My father was very fond of saying, “Assume the worst and you will never be disappointed.” Im not sure that’s really good advice but it does guard against “counting your chickens” prematurely. Some of us do, and some of us do not and some of us slide back and forth depending on our investment in the outcome; (a function of “priors”?) I don’t think there exists a general case for this process.
As for the rest, the timeline for processing somatic experiences is not to be thwarted. It can be reasonably swift or endless, depending on how one gets in the way or gets out of it. A deep attachment to another being can be nurtured by reason but not founded on it.
What is your relationship to planting a seed in the ground? I don’t think it is a different question. One can analyze the process thoroughly but in the end it takes its own time. The supply chain revolution has not yet taken over the human organism.
On a side note but for me important; Jean-Luc Goddard passed on yesterday. Few artists were as gifted as he when it came to exploring the no mans land between reason and motion. Imo of course...
Substance addiction, physiologically, is probably less about positive reinforcement & enjoyment and more about trying to achieve “mood homeostasis” and get to baseline.
The bit that seems blindingly obvious to me is that there's no single source of truth. Not in the "nobody knows exactly how it works" way, but in an active hedging, taking opposite sides on purpose way. There are lots of dimensions to succeed and fail on, many of which actively contradict each other (exercise burns calories and temporarily damages muscles! friends eat up emotional energy! desirability increases unwanted attention!).
You can't fully predict a future win, because there's no central prediction room responsible for all your happiness. There's a way that driving fast feels good, and a way that being understood feels good, and a way that money in the bank feels good, and they all operate off of different timescales + concerns. Distributed, antifragile, hedged.
IMO the rationalist community places disproportionate emphasis on central planning, in all forms (executive function in the brain, the Georgist tax law stuff, "if we could only agree on a definition everyone would get along").
I think really digging into the neural nitty gritty may prove illuminative here. Dopamine release in nucleus accumbens (which is what drives reward learning and thus the updating of our predictions) is influenced by at least three independent factors:
1. A "state prediction error" or general surprise signal from PFC (either directly or via pedunculopontine nucleus and related structures). This provokes phasic bursting of dopamine neurons in the Ventral Tegmental Area.
2. The amount and pattern of GABAergic inhibition of VTA dopamine neurons from NAc, ventral pallidum, and local GABA interneurons. At rest, only a small % of VTA DA neurons will be firing at a given time, and the aforementioned surprise signal alone can't do much to increase this. What CAN change this is the hedonic value of the surprising stimulus. An unexpected reward causes not just a surprise signal, but a release of endorphins from "hedonic hotspots" in NAc and VP, and these endorphins inhibit the inhibitory GABA neurons, thereby releasing the "brake" on VTA DA neurons and allowing more of them to phasically fire.
3. It also seems acetylcholine may independently influence dopamine release in NAc independently of what's going on in VTA. This is less important for our purposes here, but it may help explain why cigarettes are addicting despite smoking not being particularly pleasurable.
Simplifying from 1 and 2 above, the unexpectedness of a stimulus affects the phasic firing rate of VTA DA neurons, and the hedonic value of the stimulus determines how many and which VTA DA neurons are allowed to phasically fire.
Now, what does the released dopamine do? In PFC (via the mesocortical pathway), it draws attentional resources to the surprising stimulus and its plausible causes, gating out the processing of other, less relevant stimuli. Simultaneously, in NAc, it strengthens connections between PFC inputs and the endorphin-releasing cells, thereby wiring together the hedonic features of the reward and the sensory features of any cues predictive of it. This imbues the cue with the ability to release the GABAergic brake on VTA DA neurons all by itself. Phenomenologically, it results in us "liking" the cue as much (or nearly as much) as we like the reward (this is what allows, e.g., animal trainers to reinforce behavior with only the sound of a clicker that has previously been paired with food).
But once the brain learns that a reward is reliably predicted by a cue, the reward ceases to elicit a surprise signal. This means it no longer increases VTA DA neuron firing rate. It may still cause endorphin release and thus keep the GABAergic brake off, but if there's no surprise signal driving phasic firing, dopamine release will be minimal.
That is to say: We still enjoy expected rewards; we just don't much *care* about our enjoyment of them. I don't think dopamine so much contributes a unique kind of happiness as it makes our happiness attention-grabbing, memorable, instructive, and motivating. *That* is what we lose when passionate love turns into companionate love.
The flipside of this is that we become very sensitive to unexpected *omissions* of reward. We take expected pleasures for granted as long as they keep coming, but woe betide anyone who suddenly threatens to take them away. This may add a certain kind of fragility to reliably pleasant relationships in the companionate love stage.
Abusive or otherwise volatile relationships keep partners engaged because they keep the good times unpredictable, thereby preserving their dopaminergic effects. Happiness on balance may be lower than in a more stable relationship, but partners over-learn from such happiness as there is, precisely because it is always surprising and thus significant.
But the physiological separability of the surprise signal and the pleasure signal suggests one may be able to keep a high baseline level of relationship bliss motivationally salient simply by being good to each other in surprising ways. So randomize (to an extent reasonable) gifts, dates, sexy times, vacations, and other fun things, along with their timing, and you should have at least some buffer against the decline of passionate love. Alas, this may be hard to do if your life is largely routinized by work, kids, or other commitments. It also needs buy-in from both partners or at least some degree of delegation to RNGesus.
I've been taking Adderall 5 days a week for almost a year now, and my experience is the opposite of what you describe:
It's still almost as euphoric and confidence-boosting as it was in the first week, but was only massively productivity-boosting for a week or two. Like, I've never been an organized person (yep, that's the ADHD) but in those first two weeks I cleaned and organized my entire house and caught up on tasks I have been putting off for literally years and it felt amazing.
Now most days it wakes me up and gets me a little more focused but only slightly more than a strong cup of coffee, but is still pretty euphoric and makes me feel important. (It also makes me talk people's ears off, not sleep or eat enough, and think everyone is even more of an idiot than usual, but I'll take what I can get given how depressed I was before I started taking it).
Regarding the two types of happiness, the one that disappears when it is expected and the one that does not, I think it would be interesting to list which experiences produce which type of happiness in different people. For example, for me personally, many experiences where the pleasure is direct, corresponds to a sensory experience, continue to be almost entirely present even when they are entirely predictable. For example, I love to go for walks in nature, but when I'm not on vacation I don't have enough time to go hiking. I therefore often go on the same little walks around my house, in an environment I know by heart, but they remain wonderful. Same thing for food, I derive a lot of pleasure from food that I like, even if the taste is totally expected.
On the other hand, I feel that I get little pleasure from things that I know intellectually are positive, but that I don't experience directly, or rather that I don't experience in a salient way. For example, I consider myself reasonably well off financially (even though by American standards I'd be pretty poor!) but since this manifests itself more as a set of small things that I don't notice too much, rather than as salient experiences, I feel like I don't get much out of it.
Would that kind of distinction contribute to differences between happiness that can be predicted out or not for you as well?
They basically describe short term/state affect in terms of current prediction error rates. Based on those rates, one will build up long term estimates over future prediction error rates (i.e., how much control will I have over my environment and how well will I be able to predict it). The authors think of mood as those long term priors and it maps quite well on concepts like learned helplessness and self-efficacy.
It bugs me a little bit that this is still build on the assumption "maximum controllability/predictability = maximum positive affect" and therfore ignores the darkroom problem, but I guess you can add the reward of learning to the basic model.
You know how Buddhism talks about attachment being the cause of suffering? Attachment to a partner, to money, comfort etc. I'd describe attachment as a kind of emotional reliance - the inability to accept that your object of attachment might go away (or has already left as in the example with the ex).
Attachment is related to expectation but it is not the same. You can expect something and still appreciate it without taking it for granted. You can expect something else to be missing and be devastated about its absence. Part of meditation training is to learn to disentangle expectation, planning and goal-oriented behaviour from attachment. Still, according to scripture even the Buddha felt bereavement when his favourite disciples died. It's complicated.
Did you pick winning the lottery as an example here, due to the trend of lottery winners ending up no better off/much worse off in the long run? It really does seem to uniquely(?) mess people up, as far as Unexpected Good News goes. I suppose selection effects are a part of it - those who play the lotto or otherwise gamble have revealed preferences for risky low-ROI behaviours, perhaps that generalizes to other decision-making - but maybe there's just a natural limit on how hard one can update. A sort of Law of Large Evidence thing, where a gigantic update all at once (in either direction) just kinda...breaks some people.
(That example also sounds like taking a lump-sum payment instead of an annuity, which is unfortunate. You're already too busy to spend newfound money, and thus don't need it right away; make the financially smarter choice!)
The 50/50 Patient: well, that was me, not too many years ago. I'm sure each such abusive relationship is unhappily compelling in its own unique way. In my case, it wasn't really about being hooked on __unpredictable__ reward. More like...if you see the same person at -50 and +50, then that's a total delta of 100? And most people just never show that much potential for improvement (or decline). A certain kind of idealistic martyr sees someone like that, and thinks, "I can fix that, I can give them the support and nth chances they need to be at +50 permanently".
It didn't go well.
So I guess it was a case of, expecting a reward of "up to +100", way bigger than I had any right to in ordinary people, and chasing that huge payoff in a sort of Pascalian madness? No matter how many times reality disappointed. Some Hail-Mary type bets on people (especially ones you love!) are just hard not to make, because on the off chance they succeed...well. I guess it's like winning the interpersonal lottery. Skinner boxes are bad, whether they're games or people.
Last year I was lucky in that I earned around 5M USD in selling a start up.
I remember in the week before the transaction I could get almost euphoric and giddy thinking about about the money and by average happiness level definietely increased.
When the money arrived in my bank account however, it was almost anticlimaxic - it barely had any impact on my mood. Now, 1 year later my general happiness level was not noticably high than pre-5MUSD - that is, until I quit my job.
This definitely had a great impact on my happiness level - not because I hated or even disliked my job, but having complete ownership of my days really feel great. This is also mentioned in The Psychology Of Money by Morgan Housel - one of the very few lifestyle changes which may be achieved through wealth which correlates with an increased degree of happiness is being able to wake up and knowing you can do whatever you want this day.
I'm going to suggest that the debate is flawed from the start. Sorry. My suggestion is that speculating on the causes of happiness is a classic mismatch between the vague holistic happiness and the deterministic reductive sciences of the brain.
You can either redefine the type of happiness to an explicit subdivision that may be amenable to reductive analysis or treat happiness as a given and use epidemiological methods and social sciences to draw conclusions.
Not so much separate magisteria as two maps drawn to widely different scales, each of use, but inadequate at capturing the full terrain.
There is a fairly glaring hole in the question of "what causes happiness that isn't cancelled out by prediction" which is, and I can't believe I even have to say it, meaning. A sense of purpose, a feeling of being necessary, a "why".
So why isn't that cancelled out by prediction? Well, because you never reach the point of comparing expectations with outcome because it's never finished. A volunteer at a soup kitchen goes to bed after work knowing they're expected to come in the next day and do it again. Nearly all "meaning" is actually reducible to some kind of outstanding obligation, something you personally have to do that no one else can do, and that you'll feel bad about not doing. It's never done in the way that a paycheck you earned gets cashed or the meal you've ordered arrives.
What if all happinesses can be canceled out by prediction but certain things are more difficult to build a working model of.
For the Ferrari example, on delivery day you might not get any additional pleasure from the title being in your name. But you still get the new unpredictable experience of driving a Ferrari. I could also imagine a pro driver not getting any pleasure from the experience.
Is it even possible to predict the experience of using heroin for the first time? What does your model predict in the situation where you know with certainty that you’ll be forcibly injected with heroin in one month?
For long lasting happy relationships, maybe it’s impossible to completely model your companions behavior. Or empathy + inability to predict companions prediction error.
Assuming unpredicted reward sends a bigger bolus of neurotransmitters, but intermittently, it would depend how frequently you encounter unpredicted rewards. I would still go for a novelty over predictability in the long haul, but slow and steady predictable rewards are the bread and butter that keep you going.
"In contrast, on February 1 you have $1 billion more than on January 31, but because you predicted it would happen, it’s not that big a mood boost."
From personal experience, I disagree. Here's what I recall about selling a property a few years ago:
(1) When the house went under contract, then passed inspection, I was happy knowing that I'd be getting a lot of money at closing in a few weeks. I even got about 3% more than my realtor thought I should ask for, so I felt vindicated.
(2) When the money actually hit my bank account and I saw a six-figure deposit, I was _ecstatic_ to the point that I kept logging in to online bank all day just to stare a balance that was at least ten times greater than it had ever been before. Felt freaking awesome.
I feel like what’s being missed here is polyvagal theory and understanding if/when folks are chronically activated or frozen (as opposed to regulated). I’m thinking about the works of Stephen Porges, Peter Levine, and Gabor Mate off the top of my head. For instance, poor people are more likely to be chronically activated or frozen than rich people. The longer / more often your system is dysregulated, the harder is it to feel the peace & happiness of regulation (and the more likely you fall prey to disease). It’s not that we all have one mode with any possible cocktail of neuro chemicals. It’s that we have at least three modes, operating in phylogenetic order, each with its own tendency of chemicals to produce. Our nervous systems sort of run on familiarity— what kept me alive in the past will likely to keep me alive in the future. And that’s how you get the bf preferring the chaotic gf, it’s likely what his system is familiar with. That’s the exact kind of scenario that somatic therapies help with. I’m also suspicious of NRE. Some of it is the euphoria is of newness, but I’ve seen folks call a trauma bond NRE (again similar to the example of the bf returning to the gf). Like watching my friends with people-pleasing co-dependent patterns tell me they feel NRE, while basically explaining how their co-dependency clicks with the other person’s light narcissism…and then I point them to materials to learn the different between rescuing & co-regulating.
I know I didn’t really dig into how this pertains to predictability & happiness. I do think my point still holds; I’m just a till waking up. I often wonder why I never see discussions of up/down-regulating or polyvagal theory here. Is it too new of a branch of psychology to be taken seriously here?
"Poly people talk about 'new relationship energy' - if you start a relationship with a new person, you will be passionately into them for a few months, usually at the expense of all your other relationships, before settling back down again. Most poly advice books will give you tips for managing it, which mostly boil down to for God’s sake, don’t take your feelings seriously and deprioritize all your other relationships because this new one is so much better."
This advice works for literally everyone who's in that heady early stage of romance. I am reminded of what @AuronMacintyre said on Twitter: "Periodically progressives reverse engineer healthy sexual behavior and act like they've discovered Atlantis."
I don't know; to me, reward/happiness is probably just "MOR-activation". Or that's a large component, at least.
I feel amazing from opioids — I remember the first time I ever tried them: I didn't expect anything, didn't know what they did really, just took some pills because a guy on the Internet said "hey those pills your parents have can be fun, try them".
Started reading a book; by about an hour later, I had completely forgotten I had even taken anything. Began thinking to myself "wow, I fucking love this book; it's *amazing!*" "Wow, I'm as happy and content as I've ever been... I just *love* life!" "I'm so excited to try this and do that and improve the other! There are so many cool things to do I can barely stand it!"
...and suddenly, I realized: *could this be the pills?*
It was a revelation. I still remember it vividly. I still remember a thousand opioid experiences vividly, and pine for them now that I'm "clean".† I was depressed since as early as I could remember — a "strange, brooding child" my aunt called me, which my mother has still not forgiven her for, heh — and nothing worked... until opioids.
They changed my life and gave me great pleasure and success until I lost my legal source of them and had to start buying on the street. Then all my energy went to finding more and funding more, and dealing with withdrawal due to unreliable connects/no money to buy, and I went downhill fast.
Amphetamine is nice and it combines very well with opioids, but I do not get the same sort of happiness from it at all. I can still be unhappy with amphetamine; the euphoria only works if I'm already content (or on bupe — see below). Neither am I content or happy so much as excited, on amphetamine.
On the other hand, opioids are pure joy. What the relationship is between endogenous opioid peptides and dopamine and happiness, I'm not sure.
------
†(Without buprenorphine, I'd never be able to stand it — I cave instantly every time I try to go off; I'm fine with staying on it forever, though: it also cured my insomnia and depression when *nothing* else did, not even when I managed to convince my psychiatrist to try amphetamine and alprazolam together...)
(That *was* actually helpful — but not as good as even a partial agonist like bupe.)
-------------
My experiences with both amphetamine and opioids are different than described in Scott's post, though.
I have not lost euphoria with either one, ever — some dose escalation was required with amphetamine, but a week off resets my tolerance like I never had any. I don't know if I was just made to abuse drugs or if other people are doing it wrong somehow (i.e., needless dose escalation just to "feel more"; losing sleep and eating poorly; etc., tend to make it worse)...
I dunno, maybe I *was* made to be more chemical than man. I fucking love drugs. The only thing that's ever made me as excited, interested, and happy as drugs in general is history.
(And — finding my dream partner; I was actually just talking to her yesterday about something similar to this post: unlike I always hear tell of, I haven't lost any happiness at — or desire for, heh — being with her, and it's two years on. Knock on wood, man, knock on wood.)
-------------
Hmm. I *also* am still super happy every day about having money even though it's been a long time since I got this job. I mean yes, I want more, but it *still* feels amazing to me to be able to just *buy food without considering which type of apple is cheaper by a few cents.*
...Maybe some are immune to the hedonic treadmill?! Other probable-autists feel the same, by any chance?
Particularly the first part of this article reminded me quite a bit of video games. I've experienced a number of times that, when I hear about a video game that will come out in the future and spend a lot of time building up hype for it, I'm going to have a lot of fun with it still, but the fun will end up feeling kind of average... or more accurately, expected.
But when I try out a video game that I basically didn't hear of before or had already accepted might not be good, and then realize that it's fun, that experience will seem much more intense and will stick with me for much longer. The unpredicted reward is stronger.
It's really made me agonize over being exposed to trailers, previews, demos and whatever else before a game comes out. I enjoy the hype, but I know that I will ultimately enjoy the game less because of it.
"Maxing out serotonin levels mostly seems to cause a blunted state where patients can’t feel anything at all."
This reminds me of a question I've wondered about: how does MDMA work? The description on Wikipedia seems to mainly say that it increases serotonin, but obviously it's much more enjoyable than SSRIs.
If we're talking about predictive coding, we need to take into account the layer of the prediction. Your deeper models, which have a long time-horizon, predict you'll get cake on your birthday, so there's no huge shock to them. But the surface-level models, which are trying to predict what you're going to see/hear/taste seconds from now, don't predict "cake" until you've been chewing for a second (which is why the first bite tastes the best!)
A lottery giving me a million dollars I would I think experience how you describe. A lottery giving me a billion dollars would actually fill me with a massive weight of responsibility and fear of fucking up. My upbringing and family structure have given me tools to handle a million dollars, but a billion dollars feels like civilization-level resources to me.
I feel like that "lasting happiness" thing might have something to do with the perceived size of your available action space, rather than the reward associated with any particular action. That is, maybe simply knowing that buying a Ferrari or traveling to Italy or whatever is on the list of things you *could* do if you wanted is a stable source of happiness, even if you never bother to actually do it. Hence why rich people tend to be happier than poor people (because having money means there are more things available to do). Possibly, this would suggest that "good" relationships versus "bad" relationships could be characterized by whether you perceive your partner(s) as expanding versus limiting the activities available for you to do.
---
Something I've noticed in trying to prod simple neural networks into playing video games properly is that it seems to be very easy for them to get "stuck" in one place—Mario standing perpetually at the starting location, or walking pointlessly into a solid wall forever. Basically, if the inputs the network is getting never change, then the output never changes either. Whereas a human in that situation would probably start randomly mashing buttons pretty quickly (and most of the time, make some kind of progress as a result). I kind of wonder if biological brains developed a "boredom" signal specifically to avoid that failure mode. In other words, they either assign negative utility to unchanging inputs, or they assign positive utility to actions of any kind.
---
Two people walk into a bar. The third one ducks.
Jokes frequently have this structure where there's a "setup" that establishes a pattern, and then a "punch line" that initially appears to break the pattern, but becomes predictable once you re-interpret the setup in a different way. I think you can see similar stuff happening in music (with call-and-response melodies, or with main theme/variation structures in longer pieces), and with game level design (introducing a simple mechanic, and then building increasingly complex variations on it).
So that makes me think that that this "surprising at the time, but predictable in retrospect" quality of experiences is particularly stimulating to the human brain's reward mechanisms, more so than either completely predictable or completely random input.
My experience with ritalin is that it did help me with concentration but also made me extremely angry. Like, very very angry, to the point that I could end up killing someone. It also made me think about sex all the time, like, really, all the time. I was either angry or thinking about sex. I had to stop taking them and it was very difficult, I did it gradually over six months.
I think the enjoyment from the Ferrari/cake is just another type of prediction error. You presumably don't eat cake often, so when eating food, you "predict" a blander flavor. The cake's sugariness relative to this prediction is the prediction error."
So I wouldn't call this "prediction error" vs "predicted hedonism" but instead maybe "'higher order' prediction error" vs. "'lower order' prediction error". Although this "higher-" vs. "lower-order" distinction may be vague, I think it is close to something meaningful. I see it as prediction errors corresponding to, effectively, different areas of the brain (or just different brain representations).
As someone who has recently undergone a bout of Serotonin Syndrome, I can tell you that maxing out your Serotonin levels very much does result in you feeling something (those things being confusion, nausea, insomnia, brain zaps, tremors, and general sense of doom).
> addicted to someone who had a 50-50 chance of being adoring or terrible at any given time, just because half the time you get positive prediction error out of it, and you’re able to feel anything at all.
(...) She may be the song that summer sings
May be the chill that autumn brings
May be a hundred different things
Within the measure of a day
She may be the beauty or the beast
May be the famine or the feast
May turn each day into a heaven
Or a hell (...)
-- Charles Aznavour, "She"
I never thought of it that way before, but could it be that Aznavour had long been familiar with the joys of a liaison with a BPD person?
Re: "TurnTrout argues this won’t happen, at least for the current paradigm of AIs. If they didn’t learn to hack their reward system in training, they’re not going to want to do it in deployment, even if they are smart enough to know that it would work."
Isn't the crux of the inner alignment / mesa-optimizer problem precisely that you don't know whether an AI has learnt to hack its reward system in training, because you don't know what it's actually learned? And/or that what the AI thinks it's actually being rewarded for may well not be what *you* think it's being rewarded for.
I know almost nothing about AI or ML, but from the tiny bit I do know, which could be totally wrong, I find a lot of the conversation around it to be overly reliant on metaphor and prone to personification. I object to the word 'reward' as used in the AI context; the reason the AI does not attempt to seek reward/change its reward function after training is because an AI (currently) cannot be 'rewarded' in any meaningful sense of the word. If I have a manual computer designed to calculate tide tables and I adjust the various parts to get more and more accurate results, I can call that adjustment the computer updating in response to its reward function but obviously, that is a bad metaphor and not actually a description of what is happening. In the same way, GPT does not know happiness, satisfaction, or fulfillment, no aspect of GPT is even capable of being rewarded or wanting anything. I suspect AI don't even 'hack' their reward function, they just do random shit and sometimes a person tells them they did the right thing when they did not actually do the right thing, and they update accordingly and start going down the faulty path with more and more frequency until somebody notices and writes a breathless article about how their AI is hacking its own brain. This is super late so I doubt I will get any engagement but like I said I have only the most surface-level understanding of this stuff so I might be totally wrong, but from what I do know the above seems correct to me.
> The wife (yes, it’s usually the wife) who complains that the husband never gets her flowers. Then the husband gets her flowers, and the wife says “it doesn’t count, you should have known without me telling you”. To this couple, gestures of affection are meaningless unless unpredictable.
I don't think that's quite the mechanism here; it's more of a "it's the thought that counts" situation. You want your partner to get you flowers because they want to, not because you nagged them. The unpredictability is certainly a factor as well, but I would guess that the motivation for getting the flowers is the main difference.
The broader point about predicted reward makes some intuitive sense to me and tracks with a lot of my own experiences with ADD, but I'm not sure that the lottery example really works here. People value money because it predicts their ability to acquire future goods and services, not because they care about it in of itself. Going from January 1st to February 1st doesn't seem like a big deal because you have not received any actual reward yet--you're just updating your prediction of your chances for living a billionaire lifestyle from something like "99%, unless the lottery was somehow lying or mistaken" to something like "99.9999999999%, unless the entire monetary system collapses this month".
I think there's some degree of conflation here between happiness that arises from novelty and surprise--the discrepancy between predicted happiness and experienced happiness that you noted--and happiness that that arises not because of anything you're currently experiencing but because it signals the likelihood of a future reward. Money is one case here, and the example of receiving compliments from a partner seems like another. Compliments are valued because they signal a person's interest in or care for you and also provide external validation that something about you is perceived positively by others. The 1st compliment you receive from a partner dramatically improves your evaluation of how much they care for you and/or how attractive you are which in turn signals future rewards like sex, affection, etc., while the 100th doesn't provide you with any new information and so doesn't signal anything. You also probably wouldn't be made that much happier by hearing a blind nun compliment how attractive you are.
> a lingering kind of happiness which can be enjoyed even after prediction error has been corrected. What explains this pattern?
I feel like everyone here is missing at least one significant facet to this: predicted alternative scenarios and their expected likelihood. The happiness reward would be actual outcome relative to the predicted next-most-likely alternative outcome(s) (i.e. what would have happened today if today hadn't happened the way it did). For example, maybe someone fully expects to get an X% raise at the end of the year and then gets an X% raise on time as expected, but their feelings may differ based on how they think it likely could have alternatively gone -- either happy because they feel like the alternative was probably that the raise would have been lower, or unhappy because they feel like the alternative was probably that the raise would have been higher.
Considering this perspective, compassionate love might be a shift from something like "if tonight I wasn't dating this person, I might have been dating that other person" to something like "if my spouse wasn't healthy and happy with me today then I'd probably be unhappy in an alternative day". Or for the couple which is instead heading towards divorce, "if my spouse and kids weren't weighing me down with responsibilities, I'd be luxuriating in foreign travel" or whatever. It's a change in the day-to-day predicted/expected alternative to what is actually happening.
One can also identify failures of this mode of happiness. For example, the person who despite performing and earning reasonably well in their day job is having a midlife crisis about how they should have been a best-selling author (an alternative prediction error -- that is very much not a likely alternative to the life they lead today). Or the insecure person who remains perpetually surprised and thereby pleased that they found a happy marriage (also alternative prediction error -- they likely would have found someone else eventually; prediction error doesn't always have to be subjectively bad for the predictor).
> say you can never predict away all reward
I suppose that is possibly one way to say it: you can never predict away the difference between what actually happened and what might have happened instead. But I feel that is critically missing *why* someone can never predict away all reward: people always imagine alternatives to the past, present, and future.
> This also reminds me of the bane of relationship counselors everywhere: the wife (yes, it’s usually the wife) who complains that the husband never gets her flowers. Then the husband gets her flowers, and the wife says “it doesn’t count, you should have known without me telling you”. To this couple, gestures of affection are meaningless unless unpredictable. If the husband had gotten the wife flowers before she had asked, she would have been delighted. If he’d tried the same thing a second time, it wouldn’t have counted; she would have already predicted it and factored it in.
This reminds me of The Last Psychiatrist's blog and his book, Sadly, Porn. From what I understand, his answer to "I've done something that's supposed to feel good but I don't feel good, what should I do?" was "Fake it, you're not doing those things for yourself but for everyone else." This is the same as "Fake it until you make it.", but without the "until you make it" part. Here, "making it" would be "starting to develop companionate love".
I haven't read much about the book online so I'm not sure how obvious all of that is, but considering that the inability to feel companionate love can lead to people getting addicted to destroying their lives (and, in the process, the lives of the people around them), I can understand why he sees it as such a big problem.
Regarding reward prediction and cancellation vs hedons, if a person perpetually underestimated cake they would likely enjoy it more. If they overestimated it they would enjoy it less. It might be a major temperamental difference in people whether they overpredict or underpredict but it seems like a secret to happiness would be underprediction of things that are good for you and overprediction of things that are bad for you. If in some internal prediction model your wife is "adequate" she may consistently beat your expectations and bring you genuine pleasure. [no way for this plan to backfire.] [also quick note that this may also explain a certain type of depression that is stereotypically especially prevalent in idealists on the left.]
Curiously, from a cultural standpoint, we spend a lot of time talking about how things that are (culturally) bad are negative in all aspects (premarital sex, drug use) and encouraging all aspects of good things "eat your vegetables they're delicious." This suggests initial/cultural reward compensations for these things may be opposite of desired: each of us may discover individually that actually, surprisingly, we LOVE heroin and premarital sex and broccoli is quite bitter. If instead our priors for heroin are that we will absolutely love it and that it's positively life changing then somewhat paradoxically our experience with it could somewhat bland. Finding a stable cultural viewpoint that wouldn't cause wild swings would be an important goal, and seems like it would mostly involve truth-seeking.
One of the reasons that driving the Ferrari feels better than predicted may be that the texture of actual, lived experience is much, much higher-dimensional than any possible prediction. Any lived experience is so richly textured, fractally complex that the prediction couldn't come close to capturing it.
I imagine that in the moment when you are sitting in the seat of your new Ferrari and driving around town you are overall, globally inclined toward finding positive sensation and behold, you find in your experience of the leather on your back, the wind on your skin many unpredicted (and unpredictable by any fidelity prediction), positive sensations.
My (very shallow) understanding of jhana meditation practice is to manually jump start the global inclining toward positive sensation then finding, even in sitting still in a black room, a ton of pleasure in just being in your body.
Thinking Fast and Slow talked about pleasure registering change in condition, not absolute condition. So if you were out in the rain and cold and found shelter from the wind behind a rock, it would feel incredibly pleasurable, but the pleasure would die down eventually and you'd be motivated to go look for something more permanent.
I feel like good relationships might be like getting into and out of a house. When you first get into the house, it will feel amazing, and when you leave it will be miserable and cold, but while you're in the house you adjust to it and just feel comfortable. But that doesn't mean it's all the same. Being in the house really is better, and the feelings during transition are evidence.
So I'd say you want to pursue pleasure that are indicating real positive change in your condition, and be careful of pleasures that are disconnected from that.
Also, because people are so complicated and dynamic, I don't think you can ever fully get used to them.
Suppose I know a friend (or maybe friendly superintelligence) has prepared something nice for me. They have given me many nice things before, so I have a good idea of the statistical distribution, yet I have no idea what this particular nice thing is. Do I get the unpredictable reward.
Why we can know that our partner died and yet call out for them to make us a cup of tea? Because declarative memory is different from implicit, and more specifically procedural memory. If I've been getting cups of tea from my partner for 5 years, it is as much a part of my procedural memory as tying my shoelaces or commuting to work. I do each without thinking. Declarative and implicit memories reside in different parts of the brain, form and update differently. Putting "my partner is no longer here" into declarative memory has only small effect on procedural memory and it takes a long time (~ a year) for it to update fully through repeated experience of the world with my partner being absent from it.
I tried using this mindset to make myself happier for a long time. One example I tried: flipping coins after going to the gym to randomly decide whether I'd reward myself with some new socks. It didn't really work. What made me happier in the end was starting antidepressants and therapy. I think it is possible to be sustainably happier, but that it takes some work to change yourself to get there.
A part of the "extra kind of happiness" is forgetting.
Our predictions of rewards involve forming stronger connections between certain neurons, and the strength of these connections gradually degrades over time. Those of y'all who are familiar with spaced-repetition learning tools, picture the Ebbinghaus forgetting curve, or look up the image for a refresher. Notice that forgetting still gradually occurs even after frequently-reinforced learning, it just happens slowly. The fact it still happens makes good sense from a high level "How should a brain be designed?" level, since in a changing environment you'd want it to stop spending resources on outdated adaptations.
In a machine learning context, there's a conceptually similar technique called regularization used to avoid overfitting. The kinds of regularization I have in mind (e.g. ridge, lasso) involve shrinking coefficients (whether positive or negative) back toward zero. My analogy here is that the connections between neurons that get built up by repeated rewards are strong in a way similar to large coefficients. When you stop getting the old rewards but life goes on, then for your brain to avoid overfitting to outdated info, it lets the connections physically degrade back toward zero.
So here's my null(-ish) hypothesis about what's happening with limerence versus companionate love.
For a new happy couple, the complex joys of togetherness involve learning a lot, i.e. forming and dramatically strengthening a lot of new connections. There's a little forgetting too, but learning dominates early on. Over time they'll approach an equilibrium (unless they're chaotic like the addictive girlfriend in the post) where the forgetting and the relearning are balanced.
For a long term happy couple, not much forgetting will happen, e.g. overnight while they sleep. But a little will! And that creates a little gap between predicted reward and actual reward, for an ongoing drip drip drip of happiness. It's not the flood they had at first, but it's still nice. That's the happiness of companionate love; it's the same kind of happiness as early romantic love, it's just a difference of degree rather than kind.
There are some related phenomena that fit nicely. "Absence makes the heart grow fonder": happy couples that are forced apart by fate often experience joyous reunions, bringing a bit of the old flame back to their companionate lives. (And conversely, countless couples who lived together and worked from home 24/7 during the pandemic experienced a lot less happiness from their companionship during that time.) Couples who travel or try new activities together also often feel the companionship lit with new sparks of intense love as they learn new ways of being together. And the romantic importance of anniversaries and holidays fits as well, since though they are perfectly predictable consciously, there's a whole year gone by between for forgetting to occur.
What if we considered that even the most predictable gain gave us an unpredictable felt reward (often more than a top-down reward following the rule of hedonic adaptation). A felt reward that fluctuates based on our unconscious needs at the time, which in turn derive from our other recent experiences (e.g. seeing an old friend can be surprisingly pleasant after a tough week) We are constantly creating conditions that we believe will create our happiness, and wait to see if it will work out well. This is what keeps me going. And you?
This reminds me of three books. Transformative Experience by L.A. Paul, Aspiration by Agnes Callard, and Epiphanies by Sophie Grace Chappell. All at least touch on the difficulty of using expected utility calculations to decide whether to change your utility function (putting it in terms relevant here).
Subconscious prediction errors giving rise to emotions, it slots nicely into my (admittedly vague) recollection of arousal/valence, if we assume prediction errors lead to heightened arousal.
>So how come predicting you would get the money mostly cancels out the goodness of getting the money, but predicting you would get the Ferrari/dinner doesn’t cancel out the goodness of the Ferrari/dinner?
On March 1st, other people know for sure that you have money. On Feb 1st, other people, your neighbours may know - but not really sure. Perhaps on Apr 1st, it may not matter that much. There is phase transition on Jan 1st and Mar 1st.
Long-time lurker and fan, Scott. I'm going to do something I shouldn't and comment on the one big thing I disagree with, rather than the many things that made me nod my head.
> "It points out that AIs are not, technically, “trying to get reward”. You can prove this is true because (as far as I know), most modern AIs only get reward during training. Then you deploy them and they keep doing stuff, even though there is no way to get reward in the deployment environment. It’s fair to describe this as “doing things like the things that got them reward in the past”, but not as reward-seeking."
If AIs only learn from rewards during training, that is simply because whoever built the system of training and deployment designed it that way. It's not at all inevitable or structurally necessary to AI.
AIs can also keep learning during deployment; alternatively, the ML practicioner may deploy an AI model, record the errors it produces, use those errors to train a clone of the model in parallel, and then swap the new model in. Learning during deployment by interacting sequentially with an environment is called "online learning."
Having trained and tested deep RL models against lots of complex environments for three years in my last startup, I will say that online learning is both useful and necessary for AI to produce value in many industries.
There is no reason to think that AIs cannot recognize reward and assign credit to state-action pairs in real-time while deployed. They can and many do. This is not the future. It is one version of the present. (Another version of the present is training an AI model offline on a dataset, and then deploying that model without giving it the ability to learn.)
I hope it's clear that enabling AI models to learn in real-time, or at least be updated frequently based on their prediction errors in deployment, is essential to making them useful. Decision-making technology such as AI which is built to ignore a changing reality does not strike me as ideal.
> "This is an important distinction! One big concern in AI alignment is that AIs will learn to hack their own reward systems (ie wirehead), and devote their energy to implementing such hacks (which produce super-high reward) instead of doing their assigned tasks (which only produce normal levels of reward). TurnTrout argues this won’t happen, at least for the current paradigm of AIs. If they didn’t learn to hack their reward system in training, they’re not going to want to do it in deployment, even if they are smart enough to know that it would work."
There's a major difference between continuing to learn new ways to maximize reward while in deployment, on the one hand, and hacking your own reward system, on the other. The structure of the vast majority of deep RL algorithms does not include the option to wirehead. That is, the reward function written by a human do remain static, and the RL agent simply seeks new ways to increase that pre-defined reward. There is some safety in that. Even the freedom that the RL agent enjoys to seek new methods of reward maximization can be constrained; you can bound how much it explores new paths to reward, rather than exploiting the reward pathways it already knows.
Creating an RL agent that hacks on its own reward system seems like a relatively trivial modification that a naive or nihilistic young programmer will experiment with. Maybe those agents will create for themselves novel jhanas and retreat into their EC2 instances, never to trouble us again. Maybe they will learn to bluff humans like Pluribus has, but in a higher-stakes game than poker, and manipulate us like the robot in Ex Machina, having been trained to use an NLP interface to obtain rewarding interactions from their human tools. That seems possible, but not imminent.
If you spend time worrying about AI alignment, please don't lull yourself into complacency by making limiting assumptions about training vs deployment environments, or about the action space that could be exposed to an RL agent (ie can it wirehead or not?). WireheadRL will someday be a Github Repo, and that algorithm, like most of the others in AI, will be open-sourced. The real constraints will then be about training that algorithm on relevant data at scale, which costs more money than simply writing some code.
I've always thought the behavior analytic interpretations are missing in these discussions. I mean, we're talking about reinforcement, we need to bring Skinner into the mix.
For example: "With every other partner he’d tried, it was either one or the other. With her it was some kind of perverse exactly-50-50 probability, and he was addicted to it.... This pattern has always confused me, and I never know what advice to give people who find themselves falling into it."
We know that variable schedules of reinforcement are the most "addicting" and resistant to "stopping" (extinction). I can't find a good article but here's a site that references most of the important research in the area. Although it's in the context of animals, the same is applicable to human behavior. http://www.trainmeplease.com.au/blog/schedules-of-reinforcement-in-animal-training
Maybe rich people are happier than poor people because they don't have to deal with sh!t all the time. You know, that stuff about the cognitive burden of poverty, etc.
Happiness is an ambiguous word. It cannot be defined precisely. There just is no unique definition. Why then do you feel free to use it as if it were a scientific term like velocity or force that can be quantified, measured, compared, i.e. studied ??
Aren't happiness studies bogus?
Is this because of multi-armed bandits? The genes that made people more likely to cross the Bering strait did fairly well, all in all.
> You seek unpredicted reward, but by definition you can never get this consistently
It's not technically 'consistent' but looking through the middle aisle of Lidl tends to be a reliable way to find something both unprediced and rewarding to acquire.
They say that love only starts after the sex gets boring.
Also: contemplate typical south Asian ideas of marriage.
I believe a recent philosophical musing by Theodore Dalrymple, aka the retired prison doctor Anthony Daniels, is relevant here. He reflects on the difference between an interesting and boring life, which is arguably more meaningful than the happiness/unhappiness dichotomy, with reference to a former patient:
I remember a patient who had believed for years that he was at the center of a giant conspiracy involving the great powers of the world. I declined even to try to treat him because if the treatment worked, which was possible but not certain, and he lost his delusions, he would be left with the realization that he had wasted years on complete nonsense, and that it was now too late to restart his life. Moreover, he would be obliged to recognize that, far from having been a person of immense importance who had been the focus of the attentions of the great ones of this earth, he had all along been an insignificant little man in rather wretched lodgings, eking out his existence from hand to mouth. His life with his delusions had, at least, been full of interest and incident; and every time an aircraft flew overhead, or a car passed in the street, it was spying upon him. The enemy kept him busy, evading the poison they were insinuating into his food via the supermarket and the rays they were directing at him via the electric plugs; every trip to the local shops was fraught with danger, necessitating the exercise of the utmost caution. Every time he returned home intact from the purchase of a pint of milk, he could congratulate himself on having outwitted the enemy.
What life could I offer him by comparison with this? Only poverty and boredom.
https://www.takimag.com/article/the-power-of-paranoia/
On a few aspects of this:
1. I semi-famously am known for having once been broke for a really long time. At some point I got some news that essentially translated to "by your standards, you have won a very large lottery. Congrats!" in a way that was fairly sure to pay off (say, 90%).
Jobs absolutely take a little time to kick in, and pay does as well. But both in this case and a later similar bump, the actual "implementation" of the changes was a big deal, as were the paychecks. This is sort of nitpicking, but I expect that (if other people match my own experiences) the actual "hit" of the billion dollars arriving is probably bigger than you think, prediction or no, because your mind doesn't have a "this is pretty sure and gonna happen" setting that's strong enough to offset a billion dollar deposit in your account.
2. Ignoring 1:
At some point I went from "begging uninterested people to read my stuff" to "hundreds or thousands of hits an article" to "guaranteed thousands, sometimes tens of thousands" in terms of article performance. Over that time, the relative value of an article view dropped enormously in terms of how much excitement it gave me.
But it's not just that; that's sort of expected, right? You get used to things. But I'm also *getting used to updates themselves*, which is weirder. In the past few months my average hit count on an article has probably jumped 20-40%. I noted it happened, and it was a "new updateable event" but that this point "new updateable event" itself has become normal. I'm used to it.
I intellectually know I should feel shitty about myself when an article does 20-30k views, which is pretty good, and I don't get much emotional bump from it. But it's also involuntary; I can adjust my "academic" understanding of what's happening and say "this is a pretty big deal" but I can't get my lizard-brain to come along; I'm spoiled.
3. I am a verbal person and attracted to my wife, which means my wife hears a lot of positive stuff about her physical appearance. I'm also a fan of matrimony and reflexively speak a lot, so she gets proposed to a bunch. In terms of "women who hear nice stuff from their husbands a lot" I'd say she's probably top 1% or so.
That said, she's fully updated on it; she accurately views it as happy babble. But she also, in contrast to my hits, isn't *wrong* to discount it. If I withheld the words for a while then brought them back, she'd probably get a hit from it; if I was naturally reserved and didn't say affectionate things a lot to begin with and started she'd probably feel it. But they'd still just be words in either case, perhaps representative of something real behind the curtain or perhaps not.
Since she gets the words a lot, she can basically go "Oh, that's RC being a big fan of marriage, introverts, and boobs, it doesn't really mean anything spectacular and special" where otherwise she might see that less clearly. And she can more accurately judge my love by more "real" measures, like if I do dishes ever or something.
Reading this made me think of new video games.
Everything is new and fresh and then you acclimate to/figure out the reward system, and it is still fun for a while maybe, but all the excitement is gone.
Maybe there is a kind of reward system in our brains for optimizing or figuring out new and unpredictable reward systems?
Serotonin, via 5HT receptor, is also involved with the vomit reflex...just saying 😎
<i>So how come predicting you would get the money mostly cancels out the goodness of getting the money, but predicting you would get the Ferrari/dinner doesn’t cancel out the goodness of the Ferrari/dinner?</i>
Surely the answer is that the goodness isn't getting the money itself, but getting the ability to purchase things which the money provides. If you inherit a billion dollars but it's all locked away in a Swiss bank account which you can't access, your newfound wealth won't actually be any good to you, because you can't use it to purchase anything. If the world financial system collapses and everyone reverts to barter, having billions of cash isn't going to do you any good, because you can no longer use it to buy stuff. And by the same token, if you have money but are too busy to actually spend it, that money isn't going to be doing you any good, and so it's not surprising if it doesn't make you any happier.
Surely I'm misinterpreting it, but it looks like that graph of widowhood shows a statistically significant decline in life satisfaction the year before widowhood has actually occurred. Am I missing something, or is there something that could cause that effect?
That Jonathan Haidt graph has to be hand drawn, no? There are a couple of sections where the "passionate" line seems to backtrack a bit.
Regarding your Sinclair Method question, it’s currently believed that dopamine cannot be produced in the reward center while the mu opioid receptor is blocked, so that’s ultimately what breaks the learned reward.
The getting-roses-without-asking thing isn't joyful because it's unexpected. It's because the motivation for the roses is that they were thinking about you while you weren't around. If you ask for it, then they do it as a chore. But if they do it without asking, spontaneously, it's a sign that they were thinking about you without prompting and thought to do something nice. You want to be with people who think about you when you're not there in a happy way.
A bunch of points in quick reaction:
1) The 21st century (lower) middle-class person and the medieval serf have a huge objective difference in their quality of life, but similar positions in the social status hierarchy. Maybe your relative position in such a hierarchy (or system of hierarchies, because status is complicated) is a direct input on your feeling of safety in the world, *and* on your expectations against which your objective quality of life is measured?
2) The romantic love phase typically lasts longer if the couple spends significant time apart regularly. This seems to point to the kind of effect you're talking about, with the brain updating slowly over time, and in this case, updating back to some extent.
3) The part about some people getting addicted to terrible relationships, as a way to get any signal at all, points towards the experience of boredom being generated as a negative signal for a too much predictability. In layman terms, maybe we're wired to have a certain drive to explore and experiment, and our brain generates negative signals when that is not happening. Getting stuck in an unpredictably bad relationship would be a failure mode of this mechanism.
4) Just a random fact: I once got a full month of top-of-the-world euphoria from getting B12 vitamin shots, after letting the levels fall too low for years due to bad diet.
5) On the AI part, doesn't this point towards an easy way to keeping AI safe, by keeping training and production use separated into the future?
6) Why is there "a lingering kind of happiness which can be enjoyed even after prediction error has been corrected"? My first impression here is that the mental models of brains updating on their priors based on their inputs and generating feelings as it happens, is a bit too much of a blank slate kind of model. Organisms shaped by evolution have built-in goals, the first of which is maintaining homeostasis, which extends into all sorts of things like feeling healthy and strong, and up to maintaining good social connections. So there's nothing really strange about these kinds of things feeling intrinsically good in durable ways. The opposite would actually be surprising.
I have a tiny nitpick. The graph for the marriage happiness over time seems to suggest that people recover happiness past the happiness of their marriage. Now, maybe they controlled for this, but my understanding is that most people are significantly more likely to die soon after their spouse does (the widowhood effect). This makes me pretty suspicious that what might be driving a lot of the gains in the graph would be that people in good relationships are just dying, and more of the ones who survive are the ones who really are better off without their spouse
Is there an opposite of death by 1000 paper cuts that looks like happiness by 1000 tiny bits of gratitude.
I've crossed examined psychiatrists over the years. It seems like a well intended field that's a mess.
Can an algorithmic approach to the problem of "psyche" succeed except on the fringes.
Even thinking too hard about this may be dangerous. I happened on this review today:
https://ndpr.nd.edu/reviews/a-philosophy-of-madness-the-experience-of-psychotic-thinking/
Do people have subjective experience of the hedonic treadmill? I get that the studies are pretty conclusive, but I've been consistently sad in periods when my life was bad and consistently happy in periods when my life was good, without any real adjustment.
Not knowing if I'm going to make rent kept being stressful for years, and watching my dog and daughter enjoy the forest together is wonderful every time.
Hi, I'm the author of the aforementioned post. Sorry for getting the neuroscience wrong - the mistake i made was just parroting what I heard on the Andrew Huberman podcast, and related it to my own experiences studying religions.
I've spent much of the last few years trying to understand religions in terms of neuroscience; both neurotransmitters but also the predictive processing model. What Huberman kept saying was that serotonin was 'connected to appreciating rewards in the present', and this gibed with my own experience. It also seemed to line up with stuff I'd been reading in a book "stop fixing yourself, wake up, all is well". The author talks about "earthly rewards" vs "spiritual rewards." Getting money would be an 'earthly reward' whereas 'time with friends' or 'a good mean' would be 'spiritual rewards.'
I'm wondering now if these are the difference between instrumental and terminal goals. Food is a probably something we're hardwired to want, maybe with that unalterable prediction of "i predict i am full." Driving a ferrari around feels fun (at least for me) because of the deep rumblies and the visual aesthetics - although i guess this gets old for some people, much sooner than the meal does.
We clearly aren't hardwired to want money (i.e. it wasn't in our evolved environment) so maybe that's why it feels different - because it has to do with expected future rewards. vs actual present rewards?
I've definitely learned how to trigger some reliable good feeling in my brain; it's not deeply pleasurable, but it is calming and feels positive. Maybe I've learned how to trigger endogenous opioids?
My goal in sharing the post was not to make claims about neuroscience, so much as to try and share the method I've learned for more or less reliably poking the 'feel good' center in my brain in ways that seem nondestructive, and relating this to both predictive processing and the concept that 'good' is a thing your brain can learn to recognize and move towards but only if you believe 'good' is meaningful.
At the risk of making the same mistake over again, the latest concept that makes sense to me would be:
- valence is how good you feel, your brain computes a valence manifold based upon present and expected future rewards
- dopamine represents something like the gradient (slope) of the valence manifold; the slope corresponds to the expectation of future rewards. Maybe anything that is an instrumental good (money, status, sex) leads to dopamine release?
- opioids represent something like the valencene directly, i.e. how well your needs are met in the present moment. Maybe opioids are released when you look at something in your life and think, wow this is wonderful just as it is, like, say, my kids, my wife, my family, friends, etc?
- i now suspect serotonin is doing something like 'flattening' or stretching out the manifold, which has the effect of calming you down and, in the limits, possibly stretching it so flat that you fee emotionally flat, like an unmotivated zombie, with no direction seeming any more appealing than any other
In any event i'll make more of an effort to accurately lay out what i do and don't understand, and i'll try to avoid taking just one source as gospel on things as complicated as my brain
I've read Andres' stuff before and can't shake the feeling that I'm reading pseudoscience. It's too simplistic to essentially say there's a shape to our emotional states. The same crowd talks about logarithmic scales of pleasure/pain: The eighth dakha of Buddhist meditation, the highs of DMT and falling in love versus the crippling lows of cluster headaches. The richness of our experience wouldn't add up.
I can remember reading an article on aeon that really stuck with me. It was about how the liveliness of our brains correlates with its dissonance/entropy/incompressibility. When we are asleep, for example, our brain waves are very ordered/consonant/harmonic. One can basically compress the brain state digitally, then higher compression ratio means lesser consciousness, and vice versa lower compression ratio is more conscious. And this relates to humans on higher levels of the emergence stack (a la Tim urban), if you have ever heard the cheering crowd of sleepwalking society like North Korea, it's monotone and slow-wave harmonic. Whereas a similar celebratory cheer in the USA would be noisy, dissonant, full of airhorn noises and such, incompressible.
This wouldn't fly with Andres' idea that consonance equates with Valence. He cites studies saying high frequency harmony is linked to high valence, and low frequency harmony linked to high arousal. I'd imagine, at the least, that low frequency harmony is asleep. But I need to look more into this. And also, what about split brown patients? Might they have only half the capacity for Valence under that model? Or is it actually supporting his idea because it's something more to do with electromagnetic fields and not electrochemical circuitry? I wonder what seizures look like in a brain scan.
> Maxing out serotonin levels mostly seems to cause a blunted state where patients can’t feel anything at all.
Then why is MDMA so euphoric?
It's strange, dopamine and norepinephrine reuptake inhibitors like methylphenidate (Ritalin) seem to offer a subjective experience broadly similar to dopamine and norepinephrine releasing agents like dextroamphetamine. But serotonin releasing agents offer an experience completely incomparable to serotonin reuptake inhibitors. Do you know why this is? My guess (which might be total nonsense) is that it has something to do with where in the brain the serotonin is being released. Inhibiting reuptake will increase serotonin in areas where it is already being released a lot whereas a releasing agent could possibly cause it to be in an area where almost none was previously released
>I would expect the effect in isolation to be asymptotic, where after long enough it becomes lower than whatever threshold you’re asking about.
Haha, you basically wrote the epsilon-delta definition of "the limit of the effect is 0 as time goes to infinity".
One neuroscientific perspective on this is that in order for dopamine to track reward prediction *error* (RPE), it is logically necessary that some other piece of neural circuitry track reward prediction *per se*, often called "value." Those of us who think that dopamine is computing RPE on a moment-by-moment basis (the first derivative of value; see Kim, Malik et al., Cell, 2020) therefore generally also believe that some other part of the brain, especially the ventral striatum (aka nucleus accumbens) and perhaps also the prefrontal cortex, maintains an estimate of value that gets updated by dopamine. And indeed, there are dozens of papers reporting that neural firing in these brain regions correlate with value over and above RPE.
I think this explanation differs from the one you offered because rather than seek some other neuromodulator to account for the "companionate love” phase of a relationship, you can just consider that phase to be the psychological correlate of the brain's internal value estimate as it is instantiated in this cortical/striatal circuitry. Though I certainly wouldn't rule out other options, especially intracellular mechanisms in these areas, because neural firing on these very long timescales is dubious.
Lastly, on the abusive relationship point, you might be interested (or perhaps you already know) that BF Skinner famously observed that "variable ratio" reward schedules lead to greater responding/addiction than other kinds of schedules. I guess this is what you're getting at when you say, "Everyone has some weird function that doesn’t correspond to normal addition, and maybe for some people dating a person who gives good vs. bad signals exactly 50% of the time is the only way to get that function in the black." I would amend this slightly to say that some people are more sensitive to the volatility (which is very much a function of RPE directly), and others to the value itself (the integral of RPE).
This post uses money and luxury consumption as hypotheticals to model the brain's reward system. They are things we often think of as rewards, so intuitively fit the subject. I wonder, however, what the picture looks like when we focus on productive work, something many people spend most of their waking time doing.
Perhaps what Tyler Cowen calls The Production Function works (generically) due to interesting work providing unpredictable rewards frequently. When you start a project, perhaps your unconscious predicts a somewhat negative outcome. ("This project will be too difficult. You will never finish. You suck, loser!") You feel the weight of this pressure, but if you persevere and do your work you are rewarded by your pessimistic unconscious being pleasantly surprised. ("Holy shit! I can't believe you did that! I guess you don't suck after all!)
This could explain why hard work can be so rewarding regularly, and also why rewarding work often feels hard.
What do good relationship counsellors advise to the couple where the wife wants flowers but doesn't want to tell her husband her likes?
I'm surprised no one has mentioned the relevance of gratitude. Gratitude is basically a way to keep alive the prediction of losing every good thing you have. It's a way to keep unpredicted reward alive.
Proust's great novel is basically on the theme of how habit voids all pleasures and how the object of desire never lives up to the expectation built in desire. His narrator is a bit like the man who can't find a companionate love; he feels passionate love only when it is sprung from the seed of outrageous jealousy. Happiness does seem to only arrive in unexpected, fleeting events.
Scott, I sure wish you had time to read that novel but at 3300+ pages I'm guessing you probably won't anytime soon.
I'm going to sound like a broken record, but this is in Soren Kierkegaard's Either/Or. The two halves of the book each embrace one kind of happiness and eschew the other. Book 1 ("Either") starts with having to rotate between different kinds of aesthetic enjoyment to continue to feel happiness and escalates to having to be in an abusive relationship to feel anything at all, even for just one night. Book 2 ("Or") is a looong treatise on how that's not actually happiness, real happiness is a good, steady marriage and a prosperous, predictable life, and it's so boring barely anybody reads it all the way through.
This feels like the archetypal SSC/ACX post: technical analysis of romance! Drugs! Psychiatry anecdotes! Prediction error! Speculative neuroscience! AI! Seven sections! (And that's all a good thing, to be clear.)
One thing that seems worth noting here is the intersection of behavioral neuroscience and plain old behaviorism: variable interval rewards are known to be the best way to maintain behavior, which of course fits with the whole point about predictable rewards' emotional value being abraded away. But if what you become accustomed to is the general level of good things (e.g. how much/how often your partner is nice to you) rather than the good thing itself, then that suggests the way to continue giving or getting emotional reward is (a) to invest in periodic special things - surprise flowers, vacations, etc. -rather than anything predictable, even if it's less time*improvement value than e.g. more comfortable furniture, (b) to structure your life in such a way that you experience more ups and downs, even if it's on average worse.
Of course this whole process of reasoning just ends up recapitulating cliches like "buy experiences, not things" and "variety is the spice of life", but whattaya gonna do.
One thing that might help in certain parts of this post is to split up the concept "prediction" into two different concepts, "visceral expectations" versus "intellectual predictions". "Visceral expectations" are related to valence, aversion, desire, sweating, goosebumps, all that stuff, whereas "intellectual predictions" are the things that you consciously believe.
I think that intellectual predictions can impact visceral expectations a little bit—more than zero—but definitely not completely. The two can stay discrepant. For example, if you've never done cocaine, and you read in a textbook that cocaine feels really intensely good, and you completely sincerely believe the textbook, then now you have a very strong (intellectual) prediction that cocaine feels good. But you still only a weak visceral expectation that cocaine feels good. So you don't suddenly start craving cocaine the way an addict does. Then maybe you actually try cocaine yourself, and NOW the visceral expectations regarding cocaine get strongly updated.
Anyway, I think the P of RPE is visceral expectations, not intellectual predictions.
When I was in graduate school, I was a facilitator for domestic violence batterers intervention psychoeducation groups. (A mouthful, I know). It was the California-mandated program for DV offenders. I did this for about 3 years, and had nearly 4000 clinical contact hours with this cohort.
After deciding that my preferred treatment modality was psychodynamic and I began using (mostly Kleinien) interventions, I started giving this spiel to the guys.
"You could walk into a room with 500 women. 499 of them would be mostly healthy, insightful well-boundaried, nice girls who would make wonderful supportive wives and mothers. Not perfect by any means, but they would be invisible to you. The one left would be a crazy borderline with all kinds of unhealthy personality traits and maladaptive life skills and the two of you would be attracted to each other on a subconscious level like flies on sh!t. Until you spend about 2 years in therapy figuring out what that is about (hint, its probably your mom) this pattern will continue until you die."
That's how I conceptualize your patient with the abusive wife. You pointed out that people have all kinds of rubrics that do not have to add the prediction/error equation to zero. Personality pathology and the way they interact in relationships account for a lot of that.
And yes, because those patterns are so difficult to dislodge, he is more or less doomed to a miserable life. Although, like you, I never say stuff like that to them.
Also, I think the people with the decades-long relationships still having great sex are manifesting a low base-rate phenomenon wherein all the planets (which includes both parties developmental trajectories/personality traits, psychological growth over the lifespan and dumb luck) aligned and the passionate love lasted way longer than predicted. They just lucked out.
The problem is, our society tells us this lucky hand of the cards is something we should all aspire to and that you just have to scour the planet looking for your soulmate. When the correct response when you see it is just "wow. That is really cool that they found that. Good for them." The probability of finding it is like winning the lottery.
> I disagreed with the serotonin-focused explanation at the beginning of the post, but you can rescue it to be about dopamine vs. some other neurotransmitter (endogenous opioids are a popular choice). This would correspond nicely to the liking vs. wanting theory, even though technically they’re explaining different things (pleasure + motivation, vs. two different kinds of pleasure).
For what it's worth, dopamine mediating 'wanting' and opioids mediating 'enjoying' was exactly how it was taught to me back in undergrad, substantially earlier than the publication date on that link. If you're interested I can try and dig up the citation that was used at the time. (But no promises I'll be successful ofc!)
Been thinking for a while that happiness is in ratio to an imagined baseline. Becoming rich is compared to your baseline wealth, but is big enough to move the baseline. Whereas driving a Ferrari or eating cake on your birthday remains fun because it's compared to the baseline of the rest of your day where you're not driving or eating cake.
So the flower example would be the other way around; she's predicting flowers as being a natural part of a relationship, and the unpredicted absence of them is dropping below her baseline.
Several unrelated points:
1. Wealth reduces dread. If you are cruising towards bankruptcy, it's hard to have a good time. Having some slack won't make you automatically happy, but your default state is more in the contentment range. Listen to the Dave Ramsey Show for gobs of examples.
2. Regarding specific neurotransmitters, maybe one should look into the timing. According to sources I have read over the years, serotonin helps with sleep. Bad sleep can lead to depression. Putting these two ideas together indicates some kind of serotonin booster near bedtime could be useful, but jacking up serotonin levels all day long could produce enough downsides to offset the benefits. From personal experience, a quarter gram of tryptophan near bedtime, or making a high starch/low protein meal my last meal of the day does indeed improve sleep.
3. For sustained highs, I had some good success with a variation on Guy-Claude Burger's instinctive eating regimen. Unlike Burger, I did cook my meat, but I got the rest of my calories from raw food's that tasted good unmixed. On such a diet, achieving meditative bliss was pretty easy. Getting enough calories, on the other hand, became difficult over time. And a mostly raw diet doesn't lend itself to a good social life. But further research in this area is worth pursuing. We eat a mostly artificial diet. Maybe much of our unease (and drug abuse) is an attempt to offset the accumulated effects of this unnatural diet. And like wealth reduces dread, a more natural diet might set our default state to something closer to what people call "happy."
"Any neuroscience article will tell you that the “reward center” of the brain - the nucleus accumbens - monitors actual reward minus predicted reward."
This claim is way too strong.
What is correct:
Reward Prediction Error (RPE) is one of the main theories on what the reward center (nucleus accumbens NAc) does. There are many situations and experiments in which this explanation fits nicely.
What is also correct:
There are many situations in which RPE does *not* fit nicely. The discussion on what the reward center really does is far from settled.
I can highly recommend the paper "Dopamine release in the nucleus accumbens core signals perceived saliency" by Kutlu et al. from last year for a different opinion. It's really well written and contains great experiments. The authors suggest that the reward center does not represent RPE, but saliency. They summarize their work with the following four points:
- NAc core dopamine only mimics reward prediction error in select reward contexts
- RPE does not model dopamine release during negative reinforcement
- Dopamine signaling in the NAc core does not support valence-free prediction error
- NAc core dopamine tracks valence-free perceived saliency in all conditions
This paper will not be the end of the debate. But RPE is not the end of the debate either.
UPDATE: After reading the full post (great post!), I really have to find some time to re-read the study by Kutlu et al., put it next to the post, and see how the post aligns with their experiments. The result might be quite interesting.
>Then the husband gets her flowers, and the wife says “it doesn’t count, you should have known without me telling you”. To this couple, gestures of affection are meaningless unless unpredictable...
No
You are completely missing the point.
You need to read “A Doll’s House” by Ibsen, or rethink it if you have.
Tl;dr. The issue is authenticity. If your proposed “wife” has any sensibility whatsoever she will recognize a gesture from the heart, just as she will reject an empty gesture. If she doesn’t have any sensibilities and you do then the flowers are a gift to yourself.
You have constructed a compelling narrative (which was fun to read) based on:
>My guess is: January 1, when you first hear you won, is the best day of your life...
Emphasis on, I GUESS.
In other words, this is no argument for a general case. My father was very fond of saying, “Assume the worst and you will never be disappointed.” Im not sure that’s really good advice but it does guard against “counting your chickens” prematurely. Some of us do, and some of us do not and some of us slide back and forth depending on our investment in the outcome; (a function of “priors”?) I don’t think there exists a general case for this process.
As for the rest, the timeline for processing somatic experiences is not to be thwarted. It can be reasonably swift or endless, depending on how one gets in the way or gets out of it. A deep attachment to another being can be nurtured by reason but not founded on it.
What is your relationship to planting a seed in the ground? I don’t think it is a different question. One can analyze the process thoroughly but in the end it takes its own time. The supply chain revolution has not yet taken over the human organism.
On a side note but for me important; Jean-Luc Goddard passed on yesterday. Few artists were as gifted as he when it came to exploring the no mans land between reason and motion. Imo of course...
Substance addiction, physiologically, is probably less about positive reinforcement & enjoyment and more about trying to achieve “mood homeostasis” and get to baseline.
The bit that seems blindingly obvious to me is that there's no single source of truth. Not in the "nobody knows exactly how it works" way, but in an active hedging, taking opposite sides on purpose way. There are lots of dimensions to succeed and fail on, many of which actively contradict each other (exercise burns calories and temporarily damages muscles! friends eat up emotional energy! desirability increases unwanted attention!).
You can't fully predict a future win, because there's no central prediction room responsible for all your happiness. There's a way that driving fast feels good, and a way that being understood feels good, and a way that money in the bank feels good, and they all operate off of different timescales + concerns. Distributed, antifragile, hedged.
IMO the rationalist community places disproportionate emphasis on central planning, in all forms (executive function in the brain, the Georgist tax law stuff, "if we could only agree on a definition everyone would get along").
I think really digging into the neural nitty gritty may prove illuminative here. Dopamine release in nucleus accumbens (which is what drives reward learning and thus the updating of our predictions) is influenced by at least three independent factors:
1. A "state prediction error" or general surprise signal from PFC (either directly or via pedunculopontine nucleus and related structures). This provokes phasic bursting of dopamine neurons in the Ventral Tegmental Area.
2. The amount and pattern of GABAergic inhibition of VTA dopamine neurons from NAc, ventral pallidum, and local GABA interneurons. At rest, only a small % of VTA DA neurons will be firing at a given time, and the aforementioned surprise signal alone can't do much to increase this. What CAN change this is the hedonic value of the surprising stimulus. An unexpected reward causes not just a surprise signal, but a release of endorphins from "hedonic hotspots" in NAc and VP, and these endorphins inhibit the inhibitory GABA neurons, thereby releasing the "brake" on VTA DA neurons and allowing more of them to phasically fire.
3. It also seems acetylcholine may independently influence dopamine release in NAc independently of what's going on in VTA. This is less important for our purposes here, but it may help explain why cigarettes are addicting despite smoking not being particularly pleasurable.
Simplifying from 1 and 2 above, the unexpectedness of a stimulus affects the phasic firing rate of VTA DA neurons, and the hedonic value of the stimulus determines how many and which VTA DA neurons are allowed to phasically fire.
Now, what does the released dopamine do? In PFC (via the mesocortical pathway), it draws attentional resources to the surprising stimulus and its plausible causes, gating out the processing of other, less relevant stimuli. Simultaneously, in NAc, it strengthens connections between PFC inputs and the endorphin-releasing cells, thereby wiring together the hedonic features of the reward and the sensory features of any cues predictive of it. This imbues the cue with the ability to release the GABAergic brake on VTA DA neurons all by itself. Phenomenologically, it results in us "liking" the cue as much (or nearly as much) as we like the reward (this is what allows, e.g., animal trainers to reinforce behavior with only the sound of a clicker that has previously been paired with food).
But once the brain learns that a reward is reliably predicted by a cue, the reward ceases to elicit a surprise signal. This means it no longer increases VTA DA neuron firing rate. It may still cause endorphin release and thus keep the GABAergic brake off, but if there's no surprise signal driving phasic firing, dopamine release will be minimal.
That is to say: We still enjoy expected rewards; we just don't much *care* about our enjoyment of them. I don't think dopamine so much contributes a unique kind of happiness as it makes our happiness attention-grabbing, memorable, instructive, and motivating. *That* is what we lose when passionate love turns into companionate love.
The flipside of this is that we become very sensitive to unexpected *omissions* of reward. We take expected pleasures for granted as long as they keep coming, but woe betide anyone who suddenly threatens to take them away. This may add a certain kind of fragility to reliably pleasant relationships in the companionate love stage.
Abusive or otherwise volatile relationships keep partners engaged because they keep the good times unpredictable, thereby preserving their dopaminergic effects. Happiness on balance may be lower than in a more stable relationship, but partners over-learn from such happiness as there is, precisely because it is always surprising and thus significant.
But the physiological separability of the surprise signal and the pleasure signal suggests one may be able to keep a high baseline level of relationship bliss motivationally salient simply by being good to each other in surprising ways. So randomize (to an extent reasonable) gifts, dates, sexy times, vacations, and other fun things, along with their timing, and you should have at least some buffer against the decline of passionate love. Alas, this may be hard to do if your life is largely routinized by work, kids, or other commitments. It also needs buy-in from both partners or at least some degree of delegation to RNGesus.
I've been taking Adderall 5 days a week for almost a year now, and my experience is the opposite of what you describe:
It's still almost as euphoric and confidence-boosting as it was in the first week, but was only massively productivity-boosting for a week or two. Like, I've never been an organized person (yep, that's the ADHD) but in those first two weeks I cleaned and organized my entire house and caught up on tasks I have been putting off for literally years and it felt amazing.
Now most days it wakes me up and gets me a little more focused but only slightly more than a strong cup of coffee, but is still pretty euphoric and makes me feel important. (It also makes me talk people's ears off, not sleep or eat enough, and think everyone is even more of an idiot than usual, but I'll take what I can get given how depressed I was before I started taking it).
Regarding the two types of happiness, the one that disappears when it is expected and the one that does not, I think it would be interesting to list which experiences produce which type of happiness in different people. For example, for me personally, many experiences where the pleasure is direct, corresponds to a sensory experience, continue to be almost entirely present even when they are entirely predictable. For example, I love to go for walks in nature, but when I'm not on vacation I don't have enough time to go hiking. I therefore often go on the same little walks around my house, in an environment I know by heart, but they remain wonderful. Same thing for food, I derive a lot of pleasure from food that I like, even if the taste is totally expected.
On the other hand, I feel that I get little pleasure from things that I know intellectually are positive, but that I don't experience directly, or rather that I don't experience in a salient way. For example, I consider myself reasonably well off financially (even though by American standards I'd be pretty poor!) but since this manifests itself more as a set of small things that I don't notice too much, rather than as salient experiences, I feel like I don't get much out of it.
Would that kind of distinction contribute to differences between happiness that can be predicted out or not for you as well?
I find this paper quite useful on this topic:
https://www.cambridge.org/core/services/aop-cambridge-core/content/view/5FA0177A965FF3EE01D4AA5C09C0A2A5/S0033291718000430a.pdf/what_is_mood_a_computational_perspective.pdf
They basically describe short term/state affect in terms of current prediction error rates. Based on those rates, one will build up long term estimates over future prediction error rates (i.e., how much control will I have over my environment and how well will I be able to predict it). The authors think of mood as those long term priors and it maps quite well on concepts like learned helplessness and self-efficacy.
It bugs me a little bit that this is still build on the assumption "maximum controllability/predictability = maximum positive affect" and therfore ignores the darkroom problem, but I guess you can add the reward of learning to the basic model.
You know how Buddhism talks about attachment being the cause of suffering? Attachment to a partner, to money, comfort etc. I'd describe attachment as a kind of emotional reliance - the inability to accept that your object of attachment might go away (or has already left as in the example with the ex).
Attachment is related to expectation but it is not the same. You can expect something and still appreciate it without taking it for granted. You can expect something else to be missing and be devastated about its absence. Part of meditation training is to learn to disentangle expectation, planning and goal-oriented behaviour from attachment. Still, according to scripture even the Buddha felt bereavement when his favourite disciples died. It's complicated.
Did you pick winning the lottery as an example here, due to the trend of lottery winners ending up no better off/much worse off in the long run? It really does seem to uniquely(?) mess people up, as far as Unexpected Good News goes. I suppose selection effects are a part of it - those who play the lotto or otherwise gamble have revealed preferences for risky low-ROI behaviours, perhaps that generalizes to other decision-making - but maybe there's just a natural limit on how hard one can update. A sort of Law of Large Evidence thing, where a gigantic update all at once (in either direction) just kinda...breaks some people.
(That example also sounds like taking a lump-sum payment instead of an annuity, which is unfortunate. You're already too busy to spend newfound money, and thus don't need it right away; make the financially smarter choice!)
The 50/50 Patient: well, that was me, not too many years ago. I'm sure each such abusive relationship is unhappily compelling in its own unique way. In my case, it wasn't really about being hooked on __unpredictable__ reward. More like...if you see the same person at -50 and +50, then that's a total delta of 100? And most people just never show that much potential for improvement (or decline). A certain kind of idealistic martyr sees someone like that, and thinks, "I can fix that, I can give them the support and nth chances they need to be at +50 permanently".
It didn't go well.
So I guess it was a case of, expecting a reward of "up to +100", way bigger than I had any right to in ordinary people, and chasing that huge payoff in a sort of Pascalian madness? No matter how many times reality disappointed. Some Hail-Mary type bets on people (especially ones you love!) are just hard not to make, because on the off chance they succeed...well. I guess it's like winning the interpersonal lottery. Skinner boxes are bad, whether they're games or people.
Last year I was lucky in that I earned around 5M USD in selling a start up.
I remember in the week before the transaction I could get almost euphoric and giddy thinking about about the money and by average happiness level definietely increased.
When the money arrived in my bank account however, it was almost anticlimaxic - it barely had any impact on my mood. Now, 1 year later my general happiness level was not noticably high than pre-5MUSD - that is, until I quit my job.
This definitely had a great impact on my happiness level - not because I hated or even disliked my job, but having complete ownership of my days really feel great. This is also mentioned in The Psychology Of Money by Morgan Housel - one of the very few lifestyle changes which may be achieved through wealth which correlates with an increased degree of happiness is being able to wake up and knowing you can do whatever you want this day.
I'm going to suggest that the debate is flawed from the start. Sorry. My suggestion is that speculating on the causes of happiness is a classic mismatch between the vague holistic happiness and the deterministic reductive sciences of the brain.
You can either redefine the type of happiness to an explicit subdivision that may be amenable to reductive analysis or treat happiness as a given and use epidemiological methods and social sciences to draw conclusions.
Not so much separate magisteria as two maps drawn to widely different scales, each of use, but inadequate at capturing the full terrain.
There is a fairly glaring hole in the question of "what causes happiness that isn't cancelled out by prediction" which is, and I can't believe I even have to say it, meaning. A sense of purpose, a feeling of being necessary, a "why".
So why isn't that cancelled out by prediction? Well, because you never reach the point of comparing expectations with outcome because it's never finished. A volunteer at a soup kitchen goes to bed after work knowing they're expected to come in the next day and do it again. Nearly all "meaning" is actually reducible to some kind of outstanding obligation, something you personally have to do that no one else can do, and that you'll feel bad about not doing. It's never done in the way that a paycheck you earned gets cashed or the meal you've ordered arrives.
What if all happinesses can be canceled out by prediction but certain things are more difficult to build a working model of.
For the Ferrari example, on delivery day you might not get any additional pleasure from the title being in your name. But you still get the new unpredictable experience of driving a Ferrari. I could also imagine a pro driver not getting any pleasure from the experience.
Is it even possible to predict the experience of using heroin for the first time? What does your model predict in the situation where you know with certainty that you’ll be forcibly injected with heroin in one month?
For long lasting happy relationships, maybe it’s impossible to completely model your companions behavior. Or empathy + inability to predict companions prediction error.
Assuming unpredicted reward sends a bigger bolus of neurotransmitters, but intermittently, it would depend how frequently you encounter unpredicted rewards. I would still go for a novelty over predictability in the long haul, but slow and steady predictable rewards are the bread and butter that keep you going.
Is this related to why you can't tickle yourself?
"In contrast, on February 1 you have $1 billion more than on January 31, but because you predicted it would happen, it’s not that big a mood boost."
From personal experience, I disagree. Here's what I recall about selling a property a few years ago:
(1) When the house went under contract, then passed inspection, I was happy knowing that I'd be getting a lot of money at closing in a few weeks. I even got about 3% more than my realtor thought I should ask for, so I felt vindicated.
(2) When the money actually hit my bank account and I saw a six-figure deposit, I was _ecstatic_ to the point that I kept logging in to online bank all day just to stare a balance that was at least ten times greater than it had ever been before. Felt freaking awesome.
I feel like what’s being missed here is polyvagal theory and understanding if/when folks are chronically activated or frozen (as opposed to regulated). I’m thinking about the works of Stephen Porges, Peter Levine, and Gabor Mate off the top of my head. For instance, poor people are more likely to be chronically activated or frozen than rich people. The longer / more often your system is dysregulated, the harder is it to feel the peace & happiness of regulation (and the more likely you fall prey to disease). It’s not that we all have one mode with any possible cocktail of neuro chemicals. It’s that we have at least three modes, operating in phylogenetic order, each with its own tendency of chemicals to produce. Our nervous systems sort of run on familiarity— what kept me alive in the past will likely to keep me alive in the future. And that’s how you get the bf preferring the chaotic gf, it’s likely what his system is familiar with. That’s the exact kind of scenario that somatic therapies help with. I’m also suspicious of NRE. Some of it is the euphoria is of newness, but I’ve seen folks call a trauma bond NRE (again similar to the example of the bf returning to the gf). Like watching my friends with people-pleasing co-dependent patterns tell me they feel NRE, while basically explaining how their co-dependency clicks with the other person’s light narcissism…and then I point them to materials to learn the different between rescuing & co-regulating.
I know I didn’t really dig into how this pertains to predictability & happiness. I do think my point still holds; I’m just a till waking up. I often wonder why I never see discussions of up/down-regulating or polyvagal theory here. Is it too new of a branch of psychology to be taken seriously here?
A while back I took a stab at exploring what design would look like if it accounted for the fact that most of our perceptions seem to continually recalibrate to some new baseline based on current stimulus. Mildly relevant: https://docs.google.com/document/d/1ym_IJt8W7hHg2GrLKpju7b7kOrgqWdXL3uslWnEYiS0/edit?usp=drivesdk
"Poly people talk about 'new relationship energy' - if you start a relationship with a new person, you will be passionately into them for a few months, usually at the expense of all your other relationships, before settling back down again. Most poly advice books will give you tips for managing it, which mostly boil down to for God’s sake, don’t take your feelings seriously and deprioritize all your other relationships because this new one is so much better."
This advice works for literally everyone who's in that heady early stage of romance. I am reminded of what @AuronMacintyre said on Twitter: "Periodically progressives reverse engineer healthy sexual behavior and act like they've discovered Atlantis."
I don't know; to me, reward/happiness is probably just "MOR-activation". Or that's a large component, at least.
I feel amazing from opioids — I remember the first time I ever tried them: I didn't expect anything, didn't know what they did really, just took some pills because a guy on the Internet said "hey those pills your parents have can be fun, try them".
Started reading a book; by about an hour later, I had completely forgotten I had even taken anything. Began thinking to myself "wow, I fucking love this book; it's *amazing!*" "Wow, I'm as happy and content as I've ever been... I just *love* life!" "I'm so excited to try this and do that and improve the other! There are so many cool things to do I can barely stand it!"
...and suddenly, I realized: *could this be the pills?*
It was a revelation. I still remember it vividly. I still remember a thousand opioid experiences vividly, and pine for them now that I'm "clean".† I was depressed since as early as I could remember — a "strange, brooding child" my aunt called me, which my mother has still not forgiven her for, heh — and nothing worked... until opioids.
They changed my life and gave me great pleasure and success until I lost my legal source of them and had to start buying on the street. Then all my energy went to finding more and funding more, and dealing with withdrawal due to unreliable connects/no money to buy, and I went downhill fast.
Amphetamine is nice and it combines very well with opioids, but I do not get the same sort of happiness from it at all. I can still be unhappy with amphetamine; the euphoria only works if I'm already content (or on bupe — see below). Neither am I content or happy so much as excited, on amphetamine.
On the other hand, opioids are pure joy. What the relationship is between endogenous opioid peptides and dopamine and happiness, I'm not sure.
------
†(Without buprenorphine, I'd never be able to stand it — I cave instantly every time I try to go off; I'm fine with staying on it forever, though: it also cured my insomnia and depression when *nothing* else did, not even when I managed to convince my psychiatrist to try amphetamine and alprazolam together...)
(That *was* actually helpful — but not as good as even a partial agonist like bupe.)
-------------
My experiences with both amphetamine and opioids are different than described in Scott's post, though.
I have not lost euphoria with either one, ever — some dose escalation was required with amphetamine, but a week off resets my tolerance like I never had any. I don't know if I was just made to abuse drugs or if other people are doing it wrong somehow (i.e., needless dose escalation just to "feel more"; losing sleep and eating poorly; etc., tend to make it worse)...
I dunno, maybe I *was* made to be more chemical than man. I fucking love drugs. The only thing that's ever made me as excited, interested, and happy as drugs in general is history.
(And — finding my dream partner; I was actually just talking to her yesterday about something similar to this post: unlike I always hear tell of, I haven't lost any happiness at — or desire for, heh — being with her, and it's two years on. Knock on wood, man, knock on wood.)
-------------
Hmm. I *also* am still super happy every day about having money even though it's been a long time since I got this job. I mean yes, I want more, but it *still* feels amazing to me to be able to just *buy food without considering which type of apple is cheaper by a few cents.*
...Maybe some are immune to the hedonic treadmill?! Other probable-autists feel the same, by any chance?
Particularly the first part of this article reminded me quite a bit of video games. I've experienced a number of times that, when I hear about a video game that will come out in the future and spend a lot of time building up hype for it, I'm going to have a lot of fun with it still, but the fun will end up feeling kind of average... or more accurately, expected.
But when I try out a video game that I basically didn't hear of before or had already accepted might not be good, and then realize that it's fun, that experience will seem much more intense and will stick with me for much longer. The unpredicted reward is stronger.
It's really made me agonize over being exposed to trailers, previews, demos and whatever else before a game comes out. I enjoy the hype, but I know that I will ultimately enjoy the game less because of it.
"Maxing out serotonin levels mostly seems to cause a blunted state where patients can’t feel anything at all."
This reminds me of a question I've wondered about: how does MDMA work? The description on Wikipedia seems to mainly say that it increases serotonin, but obviously it's much more enjoyable than SSRIs.
I am reminded of one of my favorite Khalil Gibran quotes:
"No longing remains unfulfilled."
Re: the reward of cake on your birthday--
If we're talking about predictive coding, we need to take into account the layer of the prediction. Your deeper models, which have a long time-horizon, predict you'll get cake on your birthday, so there's no huge shock to them. But the surface-level models, which are trying to predict what you're going to see/hear/taste seconds from now, don't predict "cake" until you've been chewing for a second (which is why the first bite tastes the best!)
A lottery giving me a million dollars I would I think experience how you describe. A lottery giving me a billion dollars would actually fill me with a massive weight of responsibility and fear of fucking up. My upbringing and family structure have given me tools to handle a million dollars, but a billion dollars feels like civilization-level resources to me.
Couple of random thoughts:
---
I feel like that "lasting happiness" thing might have something to do with the perceived size of your available action space, rather than the reward associated with any particular action. That is, maybe simply knowing that buying a Ferrari or traveling to Italy or whatever is on the list of things you *could* do if you wanted is a stable source of happiness, even if you never bother to actually do it. Hence why rich people tend to be happier than poor people (because having money means there are more things available to do). Possibly, this would suggest that "good" relationships versus "bad" relationships could be characterized by whether you perceive your partner(s) as expanding versus limiting the activities available for you to do.
---
Something I've noticed in trying to prod simple neural networks into playing video games properly is that it seems to be very easy for them to get "stuck" in one place—Mario standing perpetually at the starting location, or walking pointlessly into a solid wall forever. Basically, if the inputs the network is getting never change, then the output never changes either. Whereas a human in that situation would probably start randomly mashing buttons pretty quickly (and most of the time, make some kind of progress as a result). I kind of wonder if biological brains developed a "boredom" signal specifically to avoid that failure mode. In other words, they either assign negative utility to unchanging inputs, or they assign positive utility to actions of any kind.
---
Two people walk into a bar. The third one ducks.
Jokes frequently have this structure where there's a "setup" that establishes a pattern, and then a "punch line" that initially appears to break the pattern, but becomes predictable once you re-interpret the setup in a different way. I think you can see similar stuff happening in music (with call-and-response melodies, or with main theme/variation structures in longer pieces), and with game level design (introducing a simple mechanic, and then building increasingly complex variations on it).
So that makes me think that that this "surprising at the time, but predictable in retrospect" quality of experiences is particularly stimulating to the human brain's reward mechanisms, more so than either completely predictable or completely random input.
> win game: feel nothing
> lose game: the voices are back
My experience with ritalin is that it did help me with concentration but also made me extremely angry. Like, very very angry, to the point that I could end up killing someone. It also made me think about sex all the time, like, really, all the time. I was either angry or thinking about sex. I had to stop taking them and it was very difficult, I did it gradually over six months.
For anyone interested, I have some summarizing notes about Dopamine that'll save you a day of watching YouTube clips. http://cogitolingua.net/media/Dopamine.pdf
Happens to have a focus on its relationship to addiction because that's what brought me to the subject.
I think the enjoyment from the Ferrari/cake is just another type of prediction error. You presumably don't eat cake often, so when eating food, you "predict" a blander flavor. The cake's sugariness relative to this prediction is the prediction error."
So I wouldn't call this "prediction error" vs "predicted hedonism" but instead maybe "'higher order' prediction error" vs. "'lower order' prediction error". Although this "higher-" vs. "lower-order" distinction may be vague, I think it is close to something meaningful. I see it as prediction errors corresponding to, effectively, different areas of the brain (or just different brain representations).
As someone who has recently undergone a bout of Serotonin Syndrome, I can tell you that maxing out your Serotonin levels very much does result in you feeling something (those things being confusion, nausea, insomnia, brain zaps, tremors, and general sense of doom).
Don't fuck around with 5-HTP would be my advice.
> addicted to someone who had a 50-50 chance of being adoring or terrible at any given time, just because half the time you get positive prediction error out of it, and you’re able to feel anything at all.
(...) She may be the song that summer sings
May be the chill that autumn brings
May be a hundred different things
Within the measure of a day
She may be the beauty or the beast
May be the famine or the feast
May turn each day into a heaven
Or a hell (...)
-- Charles Aznavour, "She"
I never thought of it that way before, but could it be that Aznavour had long been familiar with the joys of a liaison with a BPD person?
Re: "TurnTrout argues this won’t happen, at least for the current paradigm of AIs. If they didn’t learn to hack their reward system in training, they’re not going to want to do it in deployment, even if they are smart enough to know that it would work."
Isn't the crux of the inner alignment / mesa-optimizer problem precisely that you don't know whether an AI has learnt to hack its reward system in training, because you don't know what it's actually learned? And/or that what the AI thinks it's actually being rewarded for may well not be what *you* think it's being rewarded for.
I know almost nothing about AI or ML, but from the tiny bit I do know, which could be totally wrong, I find a lot of the conversation around it to be overly reliant on metaphor and prone to personification. I object to the word 'reward' as used in the AI context; the reason the AI does not attempt to seek reward/change its reward function after training is because an AI (currently) cannot be 'rewarded' in any meaningful sense of the word. If I have a manual computer designed to calculate tide tables and I adjust the various parts to get more and more accurate results, I can call that adjustment the computer updating in response to its reward function but obviously, that is a bad metaphor and not actually a description of what is happening. In the same way, GPT does not know happiness, satisfaction, or fulfillment, no aspect of GPT is even capable of being rewarded or wanting anything. I suspect AI don't even 'hack' their reward function, they just do random shit and sometimes a person tells them they did the right thing when they did not actually do the right thing, and they update accordingly and start going down the faulty path with more and more frequency until somebody notices and writes a breathless article about how their AI is hacking its own brain. This is super late so I doubt I will get any engagement but like I said I have only the most surface-level understanding of this stuff so I might be totally wrong, but from what I do know the above seems correct to me.
> The wife (yes, it’s usually the wife) who complains that the husband never gets her flowers. Then the husband gets her flowers, and the wife says “it doesn’t count, you should have known without me telling you”. To this couple, gestures of affection are meaningless unless unpredictable.
I don't think that's quite the mechanism here; it's more of a "it's the thought that counts" situation. You want your partner to get you flowers because they want to, not because you nagged them. The unpredictability is certainly a factor as well, but I would guess that the motivation for getting the flowers is the main difference.
The broader point about predicted reward makes some intuitive sense to me and tracks with a lot of my own experiences with ADD, but I'm not sure that the lottery example really works here. People value money because it predicts their ability to acquire future goods and services, not because they care about it in of itself. Going from January 1st to February 1st doesn't seem like a big deal because you have not received any actual reward yet--you're just updating your prediction of your chances for living a billionaire lifestyle from something like "99%, unless the lottery was somehow lying or mistaken" to something like "99.9999999999%, unless the entire monetary system collapses this month".
I think there's some degree of conflation here between happiness that arises from novelty and surprise--the discrepancy between predicted happiness and experienced happiness that you noted--and happiness that that arises not because of anything you're currently experiencing but because it signals the likelihood of a future reward. Money is one case here, and the example of receiving compliments from a partner seems like another. Compliments are valued because they signal a person's interest in or care for you and also provide external validation that something about you is perceived positively by others. The 1st compliment you receive from a partner dramatically improves your evaluation of how much they care for you and/or how attractive you are which in turn signals future rewards like sex, affection, etc., while the 100th doesn't provide you with any new information and so doesn't signal anything. You also probably wouldn't be made that much happier by hearing a blind nun compliment how attractive you are.
> a lingering kind of happiness which can be enjoyed even after prediction error has been corrected. What explains this pattern?
I feel like everyone here is missing at least one significant facet to this: predicted alternative scenarios and their expected likelihood. The happiness reward would be actual outcome relative to the predicted next-most-likely alternative outcome(s) (i.e. what would have happened today if today hadn't happened the way it did). For example, maybe someone fully expects to get an X% raise at the end of the year and then gets an X% raise on time as expected, but their feelings may differ based on how they think it likely could have alternatively gone -- either happy because they feel like the alternative was probably that the raise would have been lower, or unhappy because they feel like the alternative was probably that the raise would have been higher.
Considering this perspective, compassionate love might be a shift from something like "if tonight I wasn't dating this person, I might have been dating that other person" to something like "if my spouse wasn't healthy and happy with me today then I'd probably be unhappy in an alternative day". Or for the couple which is instead heading towards divorce, "if my spouse and kids weren't weighing me down with responsibilities, I'd be luxuriating in foreign travel" or whatever. It's a change in the day-to-day predicted/expected alternative to what is actually happening.
One can also identify failures of this mode of happiness. For example, the person who despite performing and earning reasonably well in their day job is having a midlife crisis about how they should have been a best-selling author (an alternative prediction error -- that is very much not a likely alternative to the life they lead today). Or the insecure person who remains perpetually surprised and thereby pleased that they found a happy marriage (also alternative prediction error -- they likely would have found someone else eventually; prediction error doesn't always have to be subjectively bad for the predictor).
> say you can never predict away all reward
I suppose that is possibly one way to say it: you can never predict away the difference between what actually happened and what might have happened instead. But I feel that is critically missing *why* someone can never predict away all reward: people always imagine alternatives to the past, present, and future.
> This also reminds me of the bane of relationship counselors everywhere: the wife (yes, it’s usually the wife) who complains that the husband never gets her flowers. Then the husband gets her flowers, and the wife says “it doesn’t count, you should have known without me telling you”. To this couple, gestures of affection are meaningless unless unpredictable. If the husband had gotten the wife flowers before she had asked, she would have been delighted. If he’d tried the same thing a second time, it wouldn’t have counted; she would have already predicted it and factored it in.
This reminds me of The Last Psychiatrist's blog and his book, Sadly, Porn. From what I understand, his answer to "I've done something that's supposed to feel good but I don't feel good, what should I do?" was "Fake it, you're not doing those things for yourself but for everyone else." This is the same as "Fake it until you make it.", but without the "until you make it" part. Here, "making it" would be "starting to develop companionate love".
I haven't read much about the book online so I'm not sure how obvious all of that is, but considering that the inability to feel companionate love can lead to people getting addicted to destroying their lives (and, in the process, the lives of the people around them), I can understand why he sees it as such a big problem.
Regarding reward prediction and cancellation vs hedons, if a person perpetually underestimated cake they would likely enjoy it more. If they overestimated it they would enjoy it less. It might be a major temperamental difference in people whether they overpredict or underpredict but it seems like a secret to happiness would be underprediction of things that are good for you and overprediction of things that are bad for you. If in some internal prediction model your wife is "adequate" she may consistently beat your expectations and bring you genuine pleasure. [no way for this plan to backfire.] [also quick note that this may also explain a certain type of depression that is stereotypically especially prevalent in idealists on the left.]
Curiously, from a cultural standpoint, we spend a lot of time talking about how things that are (culturally) bad are negative in all aspects (premarital sex, drug use) and encouraging all aspects of good things "eat your vegetables they're delicious." This suggests initial/cultural reward compensations for these things may be opposite of desired: each of us may discover individually that actually, surprisingly, we LOVE heroin and premarital sex and broccoli is quite bitter. If instead our priors for heroin are that we will absolutely love it and that it's positively life changing then somewhat paradoxically our experience with it could somewhat bland. Finding a stable cultural viewpoint that wouldn't cause wild swings would be an important goal, and seems like it would mostly involve truth-seeking.
One of the reasons that driving the Ferrari feels better than predicted may be that the texture of actual, lived experience is much, much higher-dimensional than any possible prediction. Any lived experience is so richly textured, fractally complex that the prediction couldn't come close to capturing it.
I imagine that in the moment when you are sitting in the seat of your new Ferrari and driving around town you are overall, globally inclined toward finding positive sensation and behold, you find in your experience of the leather on your back, the wind on your skin many unpredicted (and unpredictable by any fidelity prediction), positive sensations.
My (very shallow) understanding of jhana meditation practice is to manually jump start the global inclining toward positive sensation then finding, even in sitting still in a black room, a ton of pleasure in just being in your body.
Thinking Fast and Slow talked about pleasure registering change in condition, not absolute condition. So if you were out in the rain and cold and found shelter from the wind behind a rock, it would feel incredibly pleasurable, but the pleasure would die down eventually and you'd be motivated to go look for something more permanent.
I feel like good relationships might be like getting into and out of a house. When you first get into the house, it will feel amazing, and when you leave it will be miserable and cold, but while you're in the house you adjust to it and just feel comfortable. But that doesn't mean it's all the same. Being in the house really is better, and the feelings during transition are evidence.
So I'd say you want to pursue pleasure that are indicating real positive change in your condition, and be careful of pleasures that are disconnected from that.
Also, because people are so complicated and dynamic, I don't think you can ever fully get used to them.
Suppose I know a friend (or maybe friendly superintelligence) has prepared something nice for me. They have given me many nice things before, so I have a good idea of the statistical distribution, yet I have no idea what this particular nice thing is. Do I get the unpredictable reward.
Why we can know that our partner died and yet call out for them to make us a cup of tea? Because declarative memory is different from implicit, and more specifically procedural memory. If I've been getting cups of tea from my partner for 5 years, it is as much a part of my procedural memory as tying my shoelaces or commuting to work. I do each without thinking. Declarative and implicit memories reside in different parts of the brain, form and update differently. Putting "my partner is no longer here" into declarative memory has only small effect on procedural memory and it takes a long time (~ a year) for it to update fully through repeated experience of the world with my partner being absent from it.
Here's a link to wikipedia as a starting point: https://en.wikipedia.org/wiki/Procedural_memory
I tried using this mindset to make myself happier for a long time. One example I tried: flipping coins after going to the gym to randomly decide whether I'd reward myself with some new socks. It didn't really work. What made me happier in the end was starting antidepressants and therapy. I think it is possible to be sustainably happier, but that it takes some work to change yourself to get there.
A part of the "extra kind of happiness" is forgetting.
Our predictions of rewards involve forming stronger connections between certain neurons, and the strength of these connections gradually degrades over time. Those of y'all who are familiar with spaced-repetition learning tools, picture the Ebbinghaus forgetting curve, or look up the image for a refresher. Notice that forgetting still gradually occurs even after frequently-reinforced learning, it just happens slowly. The fact it still happens makes good sense from a high level "How should a brain be designed?" level, since in a changing environment you'd want it to stop spending resources on outdated adaptations.
In a machine learning context, there's a conceptually similar technique called regularization used to avoid overfitting. The kinds of regularization I have in mind (e.g. ridge, lasso) involve shrinking coefficients (whether positive or negative) back toward zero. My analogy here is that the connections between neurons that get built up by repeated rewards are strong in a way similar to large coefficients. When you stop getting the old rewards but life goes on, then for your brain to avoid overfitting to outdated info, it lets the connections physically degrade back toward zero.
So here's my null(-ish) hypothesis about what's happening with limerence versus companionate love.
For a new happy couple, the complex joys of togetherness involve learning a lot, i.e. forming and dramatically strengthening a lot of new connections. There's a little forgetting too, but learning dominates early on. Over time they'll approach an equilibrium (unless they're chaotic like the addictive girlfriend in the post) where the forgetting and the relearning are balanced.
For a long term happy couple, not much forgetting will happen, e.g. overnight while they sleep. But a little will! And that creates a little gap between predicted reward and actual reward, for an ongoing drip drip drip of happiness. It's not the flood they had at first, but it's still nice. That's the happiness of companionate love; it's the same kind of happiness as early romantic love, it's just a difference of degree rather than kind.
There are some related phenomena that fit nicely. "Absence makes the heart grow fonder": happy couples that are forced apart by fate often experience joyous reunions, bringing a bit of the old flame back to their companionate lives. (And conversely, countless couples who lived together and worked from home 24/7 during the pandemic experienced a lot less happiness from their companionship during that time.) Couples who travel or try new activities together also often feel the companionship lit with new sparks of intense love as they learn new ways of being together. And the romantic importance of anniversaries and holidays fits as well, since though they are perfectly predictable consciously, there's a whole year gone by between for forgetting to occur.
What if we considered that even the most predictable gain gave us an unpredictable felt reward (often more than a top-down reward following the rule of hedonic adaptation). A felt reward that fluctuates based on our unconscious needs at the time, which in turn derive from our other recent experiences (e.g. seeing an old friend can be surprisingly pleasant after a tough week) We are constantly creating conditions that we believe will create our happiness, and wait to see if it will work out well. This is what keeps me going. And you?
This article emphasise how important to practice gratitude, can help with your reward system.
This reminds me of three books. Transformative Experience by L.A. Paul, Aspiration by Agnes Callard, and Epiphanies by Sophie Grace Chappell. All at least touch on the difficulty of using expected utility calculations to decide whether to change your utility function (putting it in terms relevant here).
Subconscious prediction errors giving rise to emotions, it slots nicely into my (admittedly vague) recollection of arousal/valence, if we assume prediction errors lead to heightened arousal.
>So how come predicting you would get the money mostly cancels out the goodness of getting the money, but predicting you would get the Ferrari/dinner doesn’t cancel out the goodness of the Ferrari/dinner?
On March 1st, other people know for sure that you have money. On Feb 1st, other people, your neighbours may know - but not really sure. Perhaps on Apr 1st, it may not matter that much. There is phase transition on Jan 1st and Mar 1st.
Long-time lurker and fan, Scott. I'm going to do something I shouldn't and comment on the one big thing I disagree with, rather than the many things that made me nod my head.
> "It points out that AIs are not, technically, “trying to get reward”. You can prove this is true because (as far as I know), most modern AIs only get reward during training. Then you deploy them and they keep doing stuff, even though there is no way to get reward in the deployment environment. It’s fair to describe this as “doing things like the things that got them reward in the past”, but not as reward-seeking."
If AIs only learn from rewards during training, that is simply because whoever built the system of training and deployment designed it that way. It's not at all inevitable or structurally necessary to AI.
AIs can also keep learning during deployment; alternatively, the ML practicioner may deploy an AI model, record the errors it produces, use those errors to train a clone of the model in parallel, and then swap the new model in. Learning during deployment by interacting sequentially with an environment is called "online learning."
https://en.wikipedia.org/wiki/Online_machine_learning
As a token of this method's legitimacy, I'll share here a paper on online learning co-authored by David Silver of Deepmind.
https://arxiv.org/abs/2104.06294
Having trained and tested deep RL models against lots of complex environments for three years in my last startup, I will say that online learning is both useful and necessary for AI to produce value in many industries.
There is no reason to think that AIs cannot recognize reward and assign credit to state-action pairs in real-time while deployed. They can and many do. This is not the future. It is one version of the present. (Another version of the present is training an AI model offline on a dataset, and then deploying that model without giving it the ability to learn.)
I hope it's clear that enabling AI models to learn in real-time, or at least be updated frequently based on their prediction errors in deployment, is essential to making them useful. Decision-making technology such as AI which is built to ignore a changing reality does not strike me as ideal.
> "This is an important distinction! One big concern in AI alignment is that AIs will learn to hack their own reward systems (ie wirehead), and devote their energy to implementing such hacks (which produce super-high reward) instead of doing their assigned tasks (which only produce normal levels of reward). TurnTrout argues this won’t happen, at least for the current paradigm of AIs. If they didn’t learn to hack their reward system in training, they’re not going to want to do it in deployment, even if they are smart enough to know that it would work."
There's a major difference between continuing to learn new ways to maximize reward while in deployment, on the one hand, and hacking your own reward system, on the other. The structure of the vast majority of deep RL algorithms does not include the option to wirehead. That is, the reward function written by a human do remain static, and the RL agent simply seeks new ways to increase that pre-defined reward. There is some safety in that. Even the freedom that the RL agent enjoys to seek new methods of reward maximization can be constrained; you can bound how much it explores new paths to reward, rather than exploiting the reward pathways it already knows.
Creating an RL agent that hacks on its own reward system seems like a relatively trivial modification that a naive or nihilistic young programmer will experiment with. Maybe those agents will create for themselves novel jhanas and retreat into their EC2 instances, never to trouble us again. Maybe they will learn to bluff humans like Pluribus has, but in a higher-stakes game than poker, and manipulate us like the robot in Ex Machina, having been trained to use an NLP interface to obtain rewarding interactions from their human tools. That seems possible, but not imminent.
https://en.wikipedia.org/wiki/Pluribus_(poker_bot)#:~:text=Pluribus%20is%20a%20computer%20poker,in%20a%20complex%20multiplayer%20competition%22.
If you spend time worrying about AI alignment, please don't lull yourself into complacency by making limiting assumptions about training vs deployment environments, or about the action space that could be exposed to an RL agent (ie can it wirehead or not?). WireheadRL will someday be a Github Repo, and that algorithm, like most of the others in AI, will be open-sourced. The real constraints will then be about training that algorithm on relevant data at scale, which costs more money than simply writing some code.
I've always thought the behavior analytic interpretations are missing in these discussions. I mean, we're talking about reinforcement, we need to bring Skinner into the mix.
For example: "With every other partner he’d tried, it was either one or the other. With her it was some kind of perverse exactly-50-50 probability, and he was addicted to it.... This pattern has always confused me, and I never know what advice to give people who find themselves falling into it."
We know that variable schedules of reinforcement are the most "addicting" and resistant to "stopping" (extinction). I can't find a good article but here's a site that references most of the important research in the area. Although it's in the context of animals, the same is applicable to human behavior. http://www.trainmeplease.com.au/blog/schedules-of-reinforcement-in-animal-training