Part Of: Neural Oscillators sequence
Related To: The Theta-Gamma Neural Code, Complementary Learning Systems
Content Summary: 3200 words, 16 min read

The Biology of Ripple Sequences

Last time, we learned that the brain represents behavior on two different timescales.

1. During locomotion, place cells fire in behavioral sequences that change approximately every 1-2s. 
2. The same trajectories are simultaneously played out in theta sequences every 100-200ms.

But neuroscientists noticed bursts of activity that tend to bracket these sequences. When you zoom in on these short-lived events, you find that they contain the exact same trajectory as the canonical theta sequence. These sharp wave ripples (SWRs) are a brief high-frequency oscillation (~100 ms duration, 80-140 Hz):

You’ll notice two different ripple sequences in the above, which bracket the behavioral sequence in a palindrome:

• Forward preplays, which compress the forthcoming sequence.
• Reverse replays, which recapitulate past behavior in reverse order.

Why do some occur in reverse order? I personally suspect the answer may be related to orthogonalization. Libby & Bushchman (2021) found that the brain rotates memories to save them from interference from new percepts. 

SWRs occur in other contexts besides exploration. In fact, they occur most frequently during sleep, and rest.

SWR rate intensifies in novel environments, and during consumption of reward:

During sleep, ripples are phase locked to two other electrophysiological phenomena: slow oscillations and spindles. 

Before sleep ripples, there is activity in sensory areas; afterwards, there is activity in the prefrontal cortex. This temporal structure suggests a cortico-hippocampal-cortical loop of information transmission (Rothschild 2017). 

SWRs are anticorrelated with theta. SWRs primarily occur during slow wave sleep (SWS) and restful waking, theta primarily occurs during REM and exploration. A walking rat expresses theta; a stationary rat expresses SWRs. Just before a rat expresses exploratory SWRs (i.e., forward replays), it slows down for a couple seconds (O’Neill et al 2006). 

Ripples are brief, they are immersed in a resting oscillatory pattern called N waves (Kay & Frank 2018). N-Waves are not continuously oscillatory, expressed at 1-4 Hz (delta range), and anatomically correlate with a sleep pattern known as microarousal.

The anti-correlation between SWRs and theta is partially mediated by acetylcholine (Ach). Theta is promoted, and SWRs are suppressed, by selective optogenetic activation of Ach neurons in the medial septum (Vandecasteele et al 2014).

Memory Consolidation and Self-Projection

Hippocampal lesions create graded retrograde amnesia, where very old memories are preserved but recent memories are lost. The complementary learning systems model holds this to be evidence of consolidation: a small capacity, highly plastic cache slowly copying memories into a large capacity, less plastic storage in neocortex. 

Most models of hippocampal memory are based on autoassociative neural networks (Hopfield 1982). But memory consolidation in such networks requires repeated exposures to the same episode to tune the synaptic matrix. Memories are initially encoded during online experience, but consolidated through offline retrieval. On this two stage model (Marr 1971), these retrieval events should ideally occur during periods of rest, so consolidation does not interfere with online behavior. 

But hippocampal lesions don’t just produce amnesia. Patients like H.M also struggled with the imagination of novel and future events (Buckner & Carroll 2006), confabulating or just remaining silent when asked to do so. In fact, the hippocampal system activates during four distinct cognitive abilities: episodic memory, prospection (planning for the future), theory of mind, and navigation. These four abilities are collectively described as self-projection:  the ability to shift perspective from the immediate present.

On this view, memory is not simply optimized for retrospective accuracy. Rather, these systems are used to guide prospective decision making. Counterfactual scenarios are simulated, and different episodes are creatively merged to forge novel predictions. This may explain results from Tolman & Gleitman (1949), with rats combining their spatial knowledge of a T-maze with a later, separate event conditioning them to fear one of its two arms:

More generally, we expect the hippocampus to behave something like Alpha-Go, which taught itself chess not through expensive supervision, but rather through self-play. If it did not improve your decision-making, we would experience amnesia. 

The real importance of mental time travel applies to travel into the future rather than the past; that is, we predominantly stand in the present facing the future rather than looking back to the past.. The constructive element in episodic recall is adaptive in that it underlies our ability to imagine possible scenarios rather than the actual ones Since the future is not an exact replica of the past, simulations of future episodes may require a system that can draw on the past in a manner that flexibly extracts and recombines elements of previous experiences – a constructive rather than a reproductive system. 

In the next section, we identify SWRs with memory consolidation. Later we will also discuss the constructive role of the SWR system.

SWRs as Consolidation Device

Sharp-wave ripples are natural candidates for retrieval-based consolidation. They preferentially occur during rest, and they engage vast swathes of neocortex. Most ripples preferentially activate the default mode network (DMN) (Norman et al 2021) which is associated with daydreaming and recollection.

Wilson & McNaughton (1994) were among the first to demonstrate that sleep SWRs replay past experiences. The correlations between place cells that manifest during running are preferentially reproduced in subsequent sleep. Skaggs & McNaughton (1996) demonstrated that these recapitulations occurred in the same temporal order. 

When SWRs are disrupted in sleep after learning, but not after a random foraging task, there is a subsequent increase in their rate (Girardeau et al 2014). This suggests homeostatic control of SWRs that is consistent with their role in the facilitation of learning.

Girardeau et al (2009) provides loss of function evidence that suppression of sleep SWRs dramatically impairs subsequent memory recall for spatial tasks. This finding is nicely complemented by several gain of function experiments. Fernandez-Ruiz et al (2019) were able to prolong ripple duration, and showed that prolongation improved performance, and conversely that shortening impaired performance. This may explain why novel situations naturally evince longer duration ripples.

Gillespie et al (2021) found one category that SWRs tend to prefer: the most recent rewarded location. This relationship held regardless of whether the rat was actively searching for its next reward, or exploiting a reward that it had just found. 

You may have noticed a very slight enrichment of current goal arms in the above. This enrichment was accentuated as the rat continued being rewarded during the repeat phase. From a chronological perspective, replays began to prioritize rewarded locations near the end of the repeat phase, and continued to represent that reward location for hours afterward. 

SWRs did represent arms not recently associated with reward. In fact, these non-enriched ripples were found to prioritize the arm that was retrieved furthest in the past (Gillespie et al 2021). This suggests a role for consolidation, and is consistent with spaced repetition solutions to the well-known forgetting curve (Murre & Dros 2015).

Most SWR retrieval events are not conscious. But retrieval appears to be a prerequisite of conscious recollection. When Malach Norman asked neurosurgery, electrode-implanted patients to recall pictures presented several minutes ago, he found that SWRs reliably fired immediately before producing the verbal report.

Norman et al (2019) also found that pictures that were recalled featured more frequent ripples while the brain:

Not all ripples correspond to conscious recollection. Buzsaki (2015) proposes that they are the vehicle of subconscious priming, a subset of which are selected for global broadcast to the conscious workspace. To my knowledge, this hypothesis has not yet been tested. 

Credit Assignment

Forward replays preferentially fire before a run; reverse replays fire afterward. Why?

Recall the distinction between model-based vs. model-free learners. Model-based learners build explicit maps of their environment. When they discover reward prediction error, this information must be propagated backwards (via Bellman backups) to the rest of its cognitive map. This suggests 

So how does the hippocampus decide which locations to replay? Mattar & Daw (2018) answer that it should refresh memories with the highest action relevance. We can model this utility as the product of two terms:

1. Gain: update the locations which reduce uncertainty the most.
2. Need: update the locations the animal is likely to use soon.

Such a model is consistent with much of our data. 

• We should expect replay to be biased towards relevant locations, such as the agent’s position (high need) and recent reward sites (high gain). 
• We should expect replay to be biased forward at the beginning of a trial (high need) and backwards at the end of a trial (high gain).

This model also predicts that reverse replay (driven by gain) should be uniquely sensitive to the magnitude of the reward prediction error. And Ambrose et al (2016) found precisely this: unexpectedly large rewards increased the rate of reverse replay (red bars, top row), and vice versa (blue bars, bottom row).

Ripple Subtypes

The hippocampus supports a bewildering number of functions. Ranganath & Ritchey (2012) present evidence for two distinct subsystems:

• Posterior-Medial PM System (pHC, MEC, PHC/RSC) as the substrate of episodic memory and egocentric navigation.
• Anterior-Temporal AT System (aHC, LEC, PRC) as the substrate of semantic memory, allocentric navigation, and stress/anxiety.

Grid cell spacing and place field size both increase as one travels from posterior to anterior hippocampus (dorsal to ventral in rodents). This observation is consistent with aHC abstracting over specific details in the pHC (Stranger et al 2014). From an ICN perspective, the PM system has been linked to parietal memory and FPCN, and the AT system has been linked to contextual association and DMN (Zheng et al 2021). For a consistent, yet more detailed hypothesis about anatomical function, see Ritchey & Cooper (2020).

The longitudinal axis is immediately relevant to ripples. Posterior SWRs have been found to propagate to the prefrontal cortex. But posterior and anterior SWRs are uncorrelated in time, suggesting their independence. Further, Sosa et al (2020) found they engage different nucleus accumbens (nAC) subpopulations, and inhibit each other’s effects. This suggests ripples may consolidate episodic and semantic memories separately. It also explains how Tingley et al (2021) discovered that SWRs are involved in homeostatic regulation of glucose. All of this underscores our impoverished view of subtypes.

Location isn’t everything, however. Besides longitudinal subtypes, four other distinctions have been identified:

1. Directionality subtypes. As discussed above, forward and reverse ripples respond differently to reward and novelty.
2. Consciousness subtypes. Unit recording evidence suggests that PFC neurons respond differently to sleep vs awake SWRs.
3. Frequency subtypes. Ramirez-Villegas et al (2015) found four subtypes of ripple frequency signatures, each with a different coupling between sharp waves and ripples. Ripples that occurred during the peak of the sharp wave strongly activate neocortical regions; ripples synchronized at the trough communicated more with subcortical regions.
4. Content subtypes. Denovellis et al (2021) found roughly three categories of ripple. Stationary ripples occur 20% of the time, are overwhelmingly local, with no speed. Continuous ripples occur 15% of the time, are usually remote (50cm away) and very high speed (10 m/s). Finally, continuous-stationary ripples occur 50% of the time, are usually remote (60cm away) and real-world speeds (17 cm/s). Jai et al (2017) found that stationary and continuous ripples exert different effects on the prefrontal cortex. 

Identifying homologies across these categories would improve our ability to understand ripple function. To this question we turn next.

Homologies across Subtypes

SWRs are brief. Extended experiences are represented by joint replays, with multiple SWRs each representing one area of the maze (Davidson et al 2009). Longer maze segments require more ripples to represent, and these ripples are of longer duration.  Wu & Foster (2014) noticed that joint replays tend to cleave at choice points:

Joint replays tended to begin on the current arm and proceed in the reverse order, before switching at the choice point to proceed along either of the two other arms in forward order. This organization suggests that reverse and forward replays may have different functions, with reverse replay representing a rewind of the immediate past, and forward replay representing the exploration of alternative futures, perhaps for the purposes of planning future behavioral trajectories. 

Joint replays seem awfully reminiscent of theta sequences. Per the Separate Phases for Encoding and Retrieval (SPEAR) model, we might associate forward replays with retrieval theta, and reverse replays with encoding theta. The SPEAR model associates slow gamma with memory retrieval. Carr et al (2012) note that, when SWRs are synchronized to slow gamma, memory replay is of higher quality.

So which is it? Do forward ripples read values to simulate future behavior (as predicted by SPEAR), or do they update values to calibrate behavior (as predicted by Mattar & Daw 2018)?

Perhaps there is a way to reconcile these accounts. But that still leaves open the question of substrate. There seems to be two competing hypotheses:

1. Laminar Hypothesis. Wang et al (2020) and de la Prida (2020) present evidence that deep CA1 pyramidal cells not only drive reverse theta sequences (i.e., past encoding), but these same anatomical circuits also contribute to reverse ripples. Similarly, superficial CA1 is held to drive forward theta and forward ripples. The analogy might be strengthened by noting that 70% of SWRs are forward ripples, and theta sequences are asymmetric, devoting more time to forward sweeps.
2. Longitudinal Hypothesis. Novel and rewarding experiences strongly enhance the rate of posterior SWRs, but have no effect on anterior SWRs (Strange et al 1999; Sosa et al 2019). Reverse ripples bear a similar relationship to forward ripples.

I favor the laminar hypothesis. This synthesis binds together three distinctions in the literature. 

There are hints at relationships amongst the unsolved pieces. 

• SWRs in slow wave sleep are predominantly forward, and not backward, replays (Wikenheiser & Reddish 2013). 
• Slower mixture ripples may be driven by CA3, while the faster moving ripples are more externally driven by MEC (Denovellis et al 2021).

But a synthesis has not yet been forged. A puzzle for another decade.

The Constructivity of Ripples

So far, we’ve mostly discussed how ripples participate in memory consolidation. But lesion data, neuroimaging, and functional considerations all suggest that the hippocampus underlies self-projection, counterfactual simulations, and creativity. Is the literature consistent with ripples playing a constructive role?

Most SWRs occur during slow-wave sleep, and sleep seems to preferentially sustain creative insight. Ellenbogen et al (2007) found that sleep facilitates the discovery of latent relations in a hierarchical stimulus. Likewise, in a mathematical quiz with a hidden yet elegant transformation rule, Wagner et al (2004) found that subjects allowed to sleep on the problem more than doubled their chances of gaining insight into the shortcut. 

Karlsson & Frank (2009) found that sleep replays tend to be lower fidelity than awake replay, which has prompted speculation that this difference may enact a form of generalization. 

Constructivity data need not be confined to sleep. Experimental suppression of awake SWRs during a spatial alternation task, for example, failed to impair memory consolidation, but instead degraded decision making (Jadhav et al 2012). This is causal evidence for constructivity.

Some studies report that ripple content can predict future choice. But in a recent study, Gillespie et al (2021) found that the content of remote ripples don’t predict the rat’s future decisions! Even if you compare correct vs incorrect trials, there is no change in ripples associated with the prospective behavioral choice. 

Tellingly, awake replay can link familiar paths into novel trajectories never experienced before (Gupta et al 2010). While this line of enquiry is still young, ripples do seem to play a role for constructive, creative exploration of memory.

Ripples and theta are expressed during rest and activity, respectively. It seems likely that theta sequences may also participate in constructive functions, perhaps in joint sequences, just as theta does. However, interplay and potential handoffs between theta and ripples have not yet been fully explored.

Until next time.

References

1. Ambrose et al (2016). Reverse Replay of Hippocampal Place Cells Is Uniquely Modulated by Changing Reward
2. Buckner & Carroll (2006). Self-projection and the brain
3. Buckner (2010). The Role of the Hippocampus in prediction and imagination
4. Buzsaki (2015). Two-stage model of memory trace formation: a role for “noisy” brain states.
5. Carr et al (2012). Transient slow gamma synchrony underlies hippocampal memory replay
6. Davidson et al (2009). Hippocampal replay of extended experience
7. de la Prida (2020). Potential factors influencing replay across CA1 during sharp-wave ripples
8. De la Prida (2019). Potential factors influencing replay across CA1 during sharp-wave ripples
9. Denovellis et al (2021). Hippocampal replay of experience at real world speeds
10. Diba & Buzsaki (2007). Forward and reverse hippocampal place-cell sequences during ripples
11. Ellenbogen et al (2007) Human relational memory requires time and sleep.
12. Fernandez-Ruiz et al (2019). Long-duration hippocampal sharp wave ripples improve memory.
13. Gillespie et al (2021). Hippocampal replay reflects specific past experiences rather than a plan for subsequent choice
14. Girardeau et al (2009). Selective suppression of hippocampal ripples impairs spatial memory.
15. Girardeau et al (2014). Learning induced plasticity regulates hippocampal sharp wave ripple drive. 
16. Gupta et al (2010). Hippocampal replay is not a simple function of experience
17. Hopfield (1982). Neural networks and physical systems with emergent collective computational abilities
18. Jadhav et al (2012). Awake Hippocampal Sharp-Wave Ripples Support Spatial Memory
19. Jai et al (2017). Distinct hippocampal-cortical memory representations for experiences associated with movement versus immobility
20. Joo & Frank (2018). The hippocampal sharp wave-ripple in memory retrieval for immediate use and consolidation
21. Karlsson & Frank (2009). Awake replay of remote experiences in the hippocampus
22. Kay & Frank (2018). Three brain states in the hippocampus and cortex 
23. Libby & Buschman (2021). Rotational dynamics reduce interference between sensory and memory representations
24. Marr (1971). Simple memory: a theory for archicortex
25. Mattar & Daw (2018). Prioritized memory access explains planning and hippocampal replay
26. Mizuseki & Buzsaki (2013). Preconfigured, skewed distribution of firing rates in the hippocampus and entorhinal cortex
27. Murre & Dros (2015). Replication and Analysis of Ebbinghaus’ Forgetting Curve
28. Norman et al (2019). Hippocampal sharp-wave ripples linked to visual episodic recollection in humans
29. Norman et al (2021). Hippocampal ripples and their coordinated dialogue with the default mode network during recent and remote recollection
30. O’Neill et al (2006). Place-selective firing of CA1 pyramidal cells during sharp wave/ripple network patterns in exploratory behavior
31. Ramirez-Villegas et al (2015). Diversity of sharp-wave–ripple LFP signatures reveals differentiated brain-wide dynamical events
32. Ritchey & Cooper (2020). Deconstructing the Posterior Medial Episodic Network.
33. Rothschild (2017). A cortical–hippocampal–cortical loop of information processing during memory consolidation
34. Skaggs & McNaughton (1996). Replay of neuronal firing sequences in rat hippocampus during sleep following spatial experience
35. Sosa et al (2020). Dorsal and Ventral Hippocampal Sharp-Wave Ripples Activate Distinct Nucleus Accumbens Networks
36. Strange et al (1999). Segregating the functions of human hippocampus
37. Tingley et al (2021). A metabolic function of the hippocampal sharp wave-ripple
38. Tolman & Gleitman (1949). Studies in learning and motivation: I. Equal reinforcements in both end-boxes, followed by shock in one end-box.
39. Vandecasteele et al (2014). Optogenetic activation of septal cholinergic neurons suppresses sharp wave ripples and enhances theta oscillations in the hippocampus
40. Wagner et al (2004). Sleep inspires insight.
41. Wang et al (2020). Alternating sequences of future and past behavior encoded within hippocampal theta oscillations.
42. Wikenheiser & Redish (2013) The balance of forward and backward hippocampal sequences shifts across behavioral states
43. Wilson & McNaughton (1994). Reactivation of Hippocampal Ensemble Memories During Sleep
44. Wu & Foster (2014). Hippocampal Replay Captures the Unique Topological Structure of a Novel Environment
45. Zheng et al (2021). Parallel hippocampal-parietal circuits for self- and goal-oriented processing