Heterogeneity in striatal dopamine circuits: Form and function in dynamic reward seeking

2.1 The striatum

While VTA/SNc dopamine neurons project broadly throughout the forebrain (Breton et al., 2019; Fields et al., 2007; Fuxe, 1965; Haber & Knutson, 2010; Swanson, 1982), their primary target is the striatum. Dopamine neurons are somewhat unique in that they almost exclusively project unilaterally to one target region (although see Fox et al., 2016), which results in a structured, parallel innervation pattern into the striatum. The striatum is historically divided into ventral and dorsal sections, based on unique anatomical connectivity and cytoarchitecture (Groenewegen, Wright, Beijer, & Voorn, 1999; Voorn, Vanderschuren, Groenewegen, Robbins, & Pennartz, 2004). The ventral striatum, which receives dopaminergic input from the VTA, is divided into the nucleus accumbens core (NAC) and shell (NAS). The dorsal striatum, receiving dopaminergic input from the SNc, is typically divided into the dorsal lateral (DLS) and dorsal medial (DMS) striatum. The striatum also receives a number of topographically organized non-dopaminergic inputs from thalamus, cortex, and other regions (Groenewegen et al., 1999; Haber & Knutson, 2010; Heilbronner, Rodriguez-Romaguera, Quirk, Groenewegen, & Haber, 2016). Approximately 90% of all striatal neurons are GABAergic medium spiny neurons (MSNs), which are the primary output. MSNs exist in two categories, containing either D1 or D2 dopamine receptors, which constitute the “direct” and “indirect” pathways as conceptualized by canonical basal ganglia literature (Albin, Young, & Penney, 1989; Bolam, Hanley, Booth, & Bevan, 2000; Gerfen, 1992a). Finally, a number of interneuron populations form local microcircuits within the striatum, which can modulate glutamatergic inputs, dopamine release, and MSN activity (Bolam et al., 1986; Burke, Rotstein, & Alvarez, 2017; Wilson, Chang, & Kitai, 1990).

2.2 Ventral striatum

The ventral striatum is at the center of the classic brain reward network, receiving the densest dopaminergic input, as well as strong innervation from prefrontal cortical areas, midline thalamus, amygdala, and hippocampus (Cardinal, Parkinson, Hall, & Everitt, 2002; Haber & Knutson, 2010; Ikemoto, 2007; Mogenson, Jones, & Yim, 1980). Two populations of MSNs, D1- or D2-containing, provide the main output source from the striatum (Le Moine & Bloch, 1995). Within the NAC, a selective D1-containing MSN population projects back to predominantly GABAergic neurons within the VTA (Heimer, Zahm, Churchill, Kalivas, & Wohltmann, 1991; Kalivas, Churchill, & Klitenick, 1993; Xia et al., 2011), while both D1- and D2-containing MSNs project to the ventral pallidum (Lu, Ghasemzadeh, & Kalivas, 1998; Tripathi, Prensa, Cebrián, & Mengual, 2010), which sends projections to the mediodorsal thalamus (Kupchik et al., 2015; Tripathi, Prensa, & Mengual, 2013). NAS has similar MSN populations projecting to the VTA, lateral hypothalamus, ventral pallidum and brainstem regions (Gangarossa et al., 2013; Heimer et al., 1991; Zahm & Brog, 1992). The NAS is complicated by a higher proportion of coexpressing D1 and D2 receptors (Bertran-Gonzalez, Hervé, Girault, & Valjent, 2010). The NAS also has distinct dopaminergic subregions: the medial and lateral NAS compartments (Lammel et al., 2008; Lammel, Ion, Roeper, & Malenka, 2011). Within the medial NAS, the dorsal and ventral portions also exhibit unique connectivity (Castro & Bruchas, 2019).

2.3 Dorsal striatum

The dorsal striatum receives dense dopaminergic innervation from the SNc and lateral portion of the VTA (Haber, Fudge, & McFarland, 2000; Ikemoto, 2007; Lerner et al., 2015), as well as strong projections from somato-motor cortical areas and lateral thalamic motor nuclei (Smith, Raju, Pare, & Sidibe, 2004). Projections from the dorsal striatum include D1-containing MSNs projecting to the internal globus pallidus (GPi) and the substantia nigra pars reticulata (SNr; Aubert, Ghorayeb, Normand, & Bloch, 2000; Gerfen et al., 1990; Matamales et al., 2009). D2-containing MSNs indirectly project to the GPi and SNr via the globus pallidus external (GPe; Gerfen, 1992b). These populations generally delineate the “direct” and “indirect” pathways, but a small proportion of D1- and D2-coexpressing populations project both directly and indirectly to the GPi and SNr (Lévesque, Bédard, Cossette, & Parent, 2003; Nicola, Surmeier, & Malenka, 2000; Surmeier, Song, & Yan, 1996).

Recent focus on the posterior “tail” portion of the dorsal striatum has shown its unique anatomical properties (Griggs et al., 2017; Watabe-Uchida & Uchida, 2018). The rodent striatal tail receives input from dopamine neurons located in the most lateral portion of the SNc, and unique non-dopaminergic inputs from sensory regions, like the visual, auditory, and gustatory cortices, the dorsal raphe, basolateral amygdala, and subthalamic nucleus (Griggs et al., 2017; Jiang & Kim, 2018; Kim, Ghazizadeh, & Hikosaka, 2014; Menegas et al., 2015). Interestingly, midbrain neurons that project to the tail do not cluster in an organized manner, in contrast to canonical projection populations to the ventral or dorsal regions (Menegas et al., 2015). In primates, the tail of the caudate also has unique connectivity, compared to the rest of the striatum, including inputs from the temporal and occipital cortex (Griggs et al., 2017; Kemp & Powell, 1970; Yamamoto, Monosov, Yasuda, & Hikosaka, 2012).

2.4 Striatal microcircuitry

A variety of anatomical mechanisms can shape MSN activity and afferent dopamine signals at axon terminals in the striatum. Glutamatergic input from the cortex, thalamus, and other limbic structures, for example, can modulate dopamine release (Floresco, Yang, Phillips, & Blaha, 1998; Jones et al., 2010; Krebs et al., 1991). Several interneuron populations also exist in the striatum, creating a local microcircuitry that is critical to shaping dopamine signals. These include the so-called “fast-spiking” interneurons, which contain parvalbumin, and a number of other cell types that have unique electrophysiological properties (Berke, 2011; Burke et al., 2017; Owen, Berke, & Kreitzer, 2018; Wilson et al., 1990). A prominent group, the cholinergic interneurons (CINs) comprise only ~2% of neurons in the striatum (Bolam et al., 1986; Oorschot, 1996; Phelps, Houser, & Vaughn, 1985) but are well-situated to affect dopamine signaling, as they can regulate plasticity at glutamatergic inputs and also synapse densely with dopamine terminals (Descarries & Mechawar, 2000; Zhou, Wilson, & Dani, 2003). Nicotinic and muscarinic receptors located directly on dopamine terminals (Jones, Bolam, & Wonnacott, 2001) regulate dopamine release (Cachope et al., 2012; Exley & Cragg, 2008; Exley et al., 2011; Threlfell et al., 2012). In the rat, CINs share similar morphological features between the dorsal and ventral regions on the striatum (Ma et al., 2014; Phelps et al., 1985; Phelps & Vaughn, 1986). Through a number of mechanisms, terminal modulation of dopamine release regulates dopamine actions on MSN populations to influence local signaling dynamics and function, reviewed in detail below.

2.5 Midbrain-striatal connectivity

The VTA and SNc not only project to the striatum in a topographic manner, they also receive anatomically organized inputs from the striatum. Striatal MSNs project back to the VTA and SNc, creating a midbrain-striatum-midbrain network that is thought to be is organized as a series of loops (Haber et al., 2000). Medially located dopamine neurons in the VTA project to the most medial and ventral portion of the striatum. There, some MSNs they target project back to the VTA near dopamine neurons that innervate slightly more lateral and dorsal parts of the striatum. This continues, back and forth, until the most lateral portion of the SNc, which innervates the DLS and tail portion. This loop architecture has been demonstrated anatomically in both non-human primates and rodents (Aoki et al., 2019; Haber et al., 2000; Ikemoto, 2007).

2.6 Open questions and future directions

3.1 The midbrain: SNc and VTA

In the context of reward learning, midbrain dopamine neurons assign motivational value to stimuli and actions. From an extensive literature, we know that deficits in VTA and SNc dopamine signaling typically impair learning and reward-directed behaviors, or movement planning, execution, and vigor, respectively. Exaggerated VTA and SNc dopamine signaling, conversely, underlies compulsive motivation and behavioral inflexibility, which occur in diseases that sensitize the dopamine system, such as addiction (Cardinal et al., 2002; Everitt & Robbins, 2005; Robinson & Berridge, 1993; Wise, 2004). Recent work taking advantage of modern neural targeting methods has begun to isolate specific functions to different populations of dopamine neurons.

Activation of dopamine neurons has long been implicated in reinforcement, as demonstrated by intracranial electrical self-stimulation studies (Corbett & Wise, 1980; Fibiger, LePiane, Jakubovic, & Phillips, 1987; Shizgal & Murray, 1989; Wise, 2004). In recent years, optogenetics allowed for confirmation of a number of hypotheses about basic dopamine functions generated from those studies. For example, activation of dopamine neurons reinforces instrumental actions (Ilango, Kesner, Keller, et al., 2014; Kim et al., 2012; Steinberg et al., 2014; Witten et al., 2011). Phasic stimulation of dopamine neurons drives Pavlovian learning to create conditioned stimuli that can elicit behaviors on their own (Saunders, Richard, Margolis, & Janak, 2018; Sharpe et al., 2017), supports conditioned place preference (Tsai et al., 2009), and strengthens ongoing reward-seeking actions (Adamantidis et al., 2011). Activation of dopamine neurons as animals make choices can bias and accelerate behavioral output (Hamid et al., 2016; Howard, Li, Geddes, & Jin, 2017; Soares, Atallah, & Paton, 2016), while activation as animals consume rewards can drive learning about predictive cues (Steinberg et al., 2013). Conversely, inhibition of dopamine neurons drives extinction learning, while excitation blunts extinction of cue-reward associations (Chang, Gardner, Tillio, & Schoenbaum, 2017; Fischbach-Weiss, Reese, & Janak, 2018; Steinberg et al., 2013). Together these studies clearly demonstrate the widely accepted view that, broadly, dopamine neurons contribute to learning and motivation. It is becoming clear, however, that there is considerable heterogeneity in dopamine neuron function depending on their anatomical location and projection target (Howe & Dombeck, 2016; Parker et al., 2016; Saunders et al., 2018).

A series of recent studies (Saunders et al., 2018) revealed a key distinction in the function of SNc and VTA dopamine neurons. Optogenetic activation of either VTA or SNc dopamine neurons, in the absence of delivery of an external reward, is sufficient to turn a neutral predictive cue into a conditioned stimulus, with important region-specific differences. Cues that predict activation of VTA dopamine neurons evoke cue-directed approach behavior and acquire conditioned value that reinforces instrumental actions in the absence of stimulation. SNc cues, conversely, evoke vigorous but undirected movement, and they do not reinforce actions on their own. This suggests a selective role for VTA dopamine neurons in Pavlovian learning that assigns incentive salience to environmental cues, which is critical to persistent and adaptive reward pursuit (Berridge, 2007; Flagel et al., 2011; Saunders & Robinson, 2013). Furthermore, these data show that SNc dopamine neurons can increase responding to a conditioned cue without increasing the value of that cue, suggesting dissociable dopamine contributions to learning versus motivation.

The ability of VTA dopamine neurons to assign value to reward cues may stem from their ability to signal information about reward identity (Stalnaker et al., 2019; Takahashi et al., 2017) and learned “hidden states” (Starkweather, Babayan, Uchida, & Gershman, 2017), in contrast to SNc dopamine neurons (Keiflin, Pribut, Shah, & Janak, 2019), whose role in learning specific cue-reward associations, independent of nonspecific movement invigoration, is less clear. Whether reward identity is fully represented in the firing patterns of VTA dopamine neurons, or in downstream activity, is also unclear.

VTA and SNc dopamine neurons have different roles in spontaneous movement generation independent of learning (Coddington & Dudman, 2019). Optogenetic activation of SNc dopamine neurons evokes movement, and their activity encodes self-paced movement initiation but not specific movement sequences (da Silva, Tecuapetla, Paixão, & Costa, 2018; Dodson et al., 2016), suggesting they contribute to general movement invigoration more than goal-directed pursuit (Saunders et al., 2018). VTA dopamine neurons are less responsive to spontaneous movements (Howe & Dombeck, 2016), and are instead engaged when animals emit cue- or goal-directed movements. Thus, in isolation, dichotomous VTA and SNc roles in behavior are clear. However, outside of targeted optogenetic manipulations, in intact behaving animals, VTA and SNc dopamine neurons appear to send parallel signals to assign value and invigorate movement for dynamic navigation of the environment.

These results show dissociable fundamental functions for VTA and SNc dopamine neurons in reward prediction, value assignment, and action invigoration (summarized in Figure 1). It remains unclear exactly how VTA and SNc dopamine neurons signal heterogeneous functions, given that in vivo measurements of their activity often suggest relatively uniform encoding of behavior, at least for simple learning paradigms, in head-fixed recording preparations (Coddington & Dudman, 2018; Eshel, Tian, Bukwich, & Uchida, 2016; Schultz, Dayan, & Montague, 1997). Other evidence suggests, however, that dopamine's multiple functions stem from a combination of diverse electrophysiological properties (Farassat et al., 2019; Lammel et al., 2011; Lerner et al., 2015; Margolis, Mitchell, Ishikawa, Hjelmstad, & Fields, 2008), firing patterns and release during behavior (da Silva et al., 2018; Engelhard et al., 2019; Howe & Dombeck, 2016; Matsumoto & Hikosaka, 2009), and subregional specialization within the striatum (Howe & Dombeck, 2016; Lerner et al., 2015; Parker et al., 2016; Saunders et al., 2018; Yang et al., 2018; Yuan, Dou, & Sun, 2019).

3.2 Ventral striatum

The ventral striatum is conceptualized as a “limbic-motor interface” (Mogenson et al., 1980), where a variety of affective and reward-related information streams converge to generate behavioral output. Broadly, dopamine in the ventral striatum contributes to the formation of stimulus-outcome associations to guide flexible, goal-directed reward-seeking behavior. Within the ventral striatum, considerable attention is given to functional distinctions between the NAC and NAS subregions.

The medial portion of the NAS receives dopaminergic input from the ventromedial VTA (Breton et al., 2019; Ikemoto, 2007; Saunders et al., 2018). Diverse functions are ascribed to NAS dopamine, but a general consensus is that it is critical to signaling reinforcement associated with the receipt of rewards, and representing ongoing motivational drive in reward seeking, rather than associative learning per se (Bassareo & Di Chiara, 1999; Kelley & Delfs, 1991; Saddoris, Cacciapaglia, Wightman, & Carelli, 2015; Saunders et al., 2018; Wyvell & Berridge, 2000). Overall, the NAS is heterogeneous compared to the rest of the ventral striatum. A larger portion of medial NAS-projecting dopamine neurons co-release glutamate, which contributes to aversive motivational states (Morales & Root, 2014), and positions them to have unique functions compared to other striatal subregions. Within the medial portion of the NAS, there is also considerable heterogeneity in reward-related functions across dorsal/ventral and rostral/caudal axes, reflecting hotspots of endogenous opioid activity and dopamine signaling (Al-Hasani et al., 2015; Castro & Berridge, 2014; Richard & Berridge, 2011). It remains unclear to what extent dopamine signals, and/or multiplexed signaling of multiple transmitters with dopamine, contribute to medial NAS heterogeneity. There is also some evidence that the lateral portion of the NAS, which lies ventral to the NAC subregion, has unique functions and connectivity within the ventral striatum (Lammel et al., 2011; Lammel, Lim, & Malenka, 2014; Yang et al., 2018). Lateral NAS projecting dopamine neurons sit in the ventrolateral portion of the VTA, contiguous with the SNc (Lammel et al., 2014; Yang et al., 2018), and may be positioned to relay information between the ventral and dorsal striatum, which could be important for learning-related plasticity as part of the striatal loop system.

In comparison to the medial NAS, the NAC receives input from dopamine neurons positioned more dorsally and laterally within the VTA (Breton et al., 2019; Ikemoto, 2007; Lammel et al., 2014; Saunders et al., 2018). Critically, across all brain regions receiving VTA input, the VTA-NAC projection is the only one for which a majority of neurons are dopaminergic (Breton et al., 2019). As such, most of the broad hypotheses about dopamine function in associative learning stem from measurement and manipulation of NAC dopamine. Generally, NAC dopamine signaling is critical to Pavlovian cue-reward learning and cue-directed or evoked motivation (Day, Roitman, Wightman, & Carelli, 2007; Di Ciano, Cardinal, Cowell, Little, & Everitt, 2001; Flagel et al., 2011; Hamid et al., 2016; Mohebi et al., 2019; Saddoris et al., 2015; Saunders et al., 2018; Saunders & Robinson, 2012; Wassum, Ostlund, Loewinger, & Maidment, 2013).

3.3 Dorsal striatum

The dorsal striatum is critical to associating stimuli, actions, and outcomes. This is particularly apparent in the context of instrumental learning, where dopamine release functions in concert with sensory-motor cortical inputs to the dorsal striatum. Within the dorsal striatum, there is considerable functional heterogeneity across the DMS and DLS subregions. Several studies show that dopamine in the DMS contributes to learning associations between actions and outcomes (Gremel & Costa, 2013; Hilario, Holloway, Jin, & Costa, 2012; Yin, Ostlund, Knowlton, & Balleine, 2005). This supports the so-called “goal-directed” behaviors, or those that are sensitive to changes in the value or content of subsequent outcomes. The DLS, in contrast, generally contributes to stimulus-response learning, which supports the so-called “habit-like” behaviors, those that are insensitive to changes in outcome value or delivery conditions (Balleine, Delgado, & Hikosaka, 2007; Dickinson, 1994). Blockade of dopamine signaling within the DMS and DLS generally impairs goal-directed and habitual responding, respectively, under a variety of conditions (Balleine et al., 2007; Balleine, Liljeholm, & Ostlund, 2009; Vandaele & Janak, 2018).

The distinction between goal-directed and habit-like behavioral control for DMS and DLS is not always clear-cut (Malvaez et al., 2018; Vandaele & Janak, 2018), and recent functional anatomical work adds further complexity to our understanding of dorsal striatal function. Dopamine neurons projecting to the DMS and DLS are organized topographically across a medial-to-lateral gradient within the SNc (Lerner et al., 2015). DMS and DLS projecting dopamine neurons are similarly activated by reward receipt but show opposing responses to aversive stimuli. DLS dopamine neurons are activated by aversive stimuli, suggesting that they have unique functional relevance outside of a typical reward-learning framework (Lerner et al., 2015).

Striatal tail-projecting dopamine neurons, unlike many others, show relatively little value-related encoding and instead have activity that scales with the novelty and magnitude of stimuli. Unlike dopamine projections to the dorsal and ventral striatum (Ilango, Kesner, Keller, et al., 2014; Saunders et al., 2018; Witten et al., 2011), tail-projecting dopamine neurons do not support positive reinforcement, as measured by optogenetic self-stimulation procedures. Activation of tail dopamine neurons reinforces the avoidance of potentially threatening novel stimuli, and tail dopamine stimulation can reinstate avoidance of a familiar object (Menegas, Akiti, Amo, Uchida, & Watabe-Uchida, 2018). The tail of the striatum has been studied more thoroughly in non-human primates, where it has distinct motor control functions. For example, in monkeys, the tail portion of the caudate integrates the visual streams to guide visual saccades based on reward preference (Griggs et al., 2017; Hikosaka, Takikawa, & Kawagoe, 2000; Yamamoto et al., 2012). Whether the tail dopamine neuron threat responses and DLS dopamine neuron activity in response to aversive stimuli (Lerner et al., 2015; Matsumoto & Hikosaka, 2009) are functionally similar, or distinct, remains to be determined.

3.4 Striatal microcircuitry

Local striatal microcircuits, which can modulate glutamatergic inputs, dopamine release, and MSN activity, also influence learning, motivation, and movement. Much of the research in this domain has focused on the function of cholinergic interneurons. For example, CIN activity can inhibit MSNs in vivo (Witten et al., 2010), and a pause in CIN activity induces learning-related plasticity mechanisms (Nair, Gutierrez-Arenas, Eriksson, Vincent, & Hellgren Kotaleski, 2015). Striatal CIN activity becomes synchronized during reward learning, and activity patterns correlate with aspects of associative learning (Aosaki, Kimura, & Graybiel, 1995; Aosaki, Tsubokawa, et al., 1994; Apicella, 2007; Goldberg & Reynolds, 2011; Kimura, Rajkowski, & Evarts, 1984). Within the ventral striatum, CINs regulate the motivational impact of reward-paired cues. Optogenetic stimulation of CINs concurrent with reward-paired cue presentations inhibits the ability of that cue to invigorate reward seeking (Collins, Aitken, Greenfield, Ostlund, & Wassum, 2016). Brief suppression of ventral striatal CIN activity can also enhance the acquisition of Pavlovian fear learning (Brown et al., 2012), suggesting a broader role in learning beyond appetitive motivation.

The functional significance of dorsal striatal cholinergic-dopamine microciruitry is less clear, but some evidence suggests acetylcholine activity in the dorstal striatum is important for behavioral flexibility under changing reward contingencies (Ragozzino, Mohler, Prior, Palencia, & Rozman, 2009). Activity in dorsal striatal CINs and dopamine terminals encode distinct features of spontaneous movement (Howe et al., 2019). Given acetylcholine's ability to modulate dopamine release, this suggests that dorsal striatum CINs can shape features of nigrostriatal dopamine-mediated actions. Dorsal striatal CINs may organize large-scale striatal activity, based on evidence that CIN activation can spread in wave-like patterns spanning considerable anatomical distances (~1 mm) across the dorsal striatum (Rehani et al., 2019). Recent data suggests that dopamine terminal activity in the dorsal striatum can also organize into wave formations (Hamid, Frank, & Moore, 2019), and MSNs show ramping activity associated with reward seeking (Cui et al., 2013; London et al., 2018). Thus, a possible role for dorsal striatal CINs is to coordinate dopamine release and MSN activity across medial–lateral/anterior–posterior domains of the striatum. In the DMS, CIN activity tracks decision state (Stalnaker, Berg, Aujla, & Schoenbaum, 2016), consistent with a role in flexible behavior. Given the reciprocal connectivity of the nigrostriatal system and divergent functions for DMS and DLS (Gremel & Costa, 2013; Haber et al., 2000; Lerner et al., 2015), coordinated CIN activity may be important for engaging spatially distinct goal-directed and habitual striatal modules, to dynamically shift between decisions states in reward seeking.

Local sculpting of dopamine signals within the ventral and dorsal striatum can modulate cue-triggered motivation, reward-related learning, movement invigoration, and action selection. Together, these data suggest that one powerful contributor to dopamine function is moment-to-moment control of its release by, and its effects on, local microcircuitry in striatum.

3.5 Circuit-level functional connectivity

The looping architecture of the midbrain-striatal system is thought to have a functional role in reward learning and decision-making over time. In this framework, experience-dependent plasticity results in a progressive engagement of the network, from initial reliance on ventral striatal dopamine to form cue-reward associations and assign value to stimuli to the formation of DLS-dependent stimulus-response associations (Keiflin & Janak, 2015). Dopamine-mediated learning potentiates MSNs in the ventral striatum that project back to GABA neurons in the midbrain (Bocklisch et al., 2013; Ikemoto, 2007). Hypothetically, as learning continues, becoming more streamlined and efficient, DLS projecting dopamine systems become engaged through recruitment of these loops. This framework is consistent with general notions of the functional role of the DLS in the formation of habits and the execution of skilled motor plans, which usually occur after extended experience (Everitt & Robbins, 2005).

This functional organization of midbrain-striatal circuits is generally accepted, but surprisingly few studies have directly tested it. In vivo evidence for a progressive engagement of ventral to dorsal striatal systems, potentially via a serial looping architecture, comes from measurements of striatal dopamine release, using electrochemical recording methods. These studies show that initial responding for rewards, including food and cocaine, evokes large dopamine responses in the NAC (Aragona et al., 2009; Clark, Collins, Sanford, & Phillips, 2013; Flagel et al., 2011; Willuhn, Burgeno, Everitt, & Phillips, 2012). After extended training, cue or outcome-related ventral striatal dopamine signals generally diminish (Clark et al., 2013; Day et al., 2007), and in some cases, emerge in the dorsal striatum (Brown, McCutcheon, Cone, Ragozzino, & Roitman, 2011; Willuhn et al., 2012; Willuhn, Burgeno, Groblewski, & Phillips, 2014). Furthermore, dopamine signaling in the ventral striatum becomes less important for the execution of some instrumental behaviors after extended training, while dorsal striatal dopamine signaling becomes more important for well-learned instrumental actions (Belin & Everitt, 2008). How this happens is unclear, but cocaine exposure is known to potentiate the activity of D1 MSNs in the nucleus accumbens that synapse onto GABAergic VTA neurons, which could disinhibit more dorsal projecting dopamine neurons (Bocklisch et al., 2013). It is unclear how that plasticity actually affects SNc dopamine projections, or if a similar mechanism occurs in non-drug learning. In contrast to instrumental learning, ventral striatal dopamine signaling remains critical to the execution of Pavlovian conditioned behaviors, and functional control does not emerge in the dorsal striatum, even after extended training (Fraser & Janak, 2017), underscoring that distinct engagement of striatal networks occurs for different forms of learning. This issue is ripe for further study but, together, these results also highlight an important dissociation: cues, actions, and rewards may engage striatal circuits, but that does not necessarily mean that that signal has functional importance for the behavior.

Other work suggests that learning-related plasticity in the striatum does not always follow in a ventromedial-to-dorsolateral progression. Within the nigrostriatal pathway, dopamine neurons projecting to the DMS and DLS are reciprocally connected to MSNs projecting back to SNc, but some lateral SNc dopamine neurons also project to the DMS (Lerner et al., 2015), suggesting that information can flow in a lateral-to-medial direction across the striatum. The behavioral conditions in which this occurs are unclear, but it is important to note that the transition to habit-like behavior is not permanent. Depending on the behavioral conditions, animals can shift dynamically between states that are goal-directed (i.e., outcome-sensitive) and habitual (i.e., outcome-insensitive; Barker, Glen, Linsenbardt, Lapish, & Chandler, 2017; Gremel & Costa, 2013). Switching from habit-like to goal-directed behavior, at least in some situations, involves simultaneous disengagement of DLS activity and upregulation of DMS activity (Gremel & Costa, 2013). Therefore, well-learned goal-directed behaviors could still be influenced by lateral SNc dopamine neurons, as predicted by serial spiraling architecture, but via their signaling into the DMS. If information flows lateral-to-medial across nigrostriatal circuits, this could be important for toggling between habitual and goal-directed behavioral states, to allow for adaptive action selection. Notably, in sequence learning, DMS and DLS activity evolves in parallel, rather than in series (Thorn, Atallah, Howe, & Graybiel, 2010), and extended training does not necessarily result in progressive loss of DMS activity (Vandaele et al., 2019), suggesting that behavioral features (e.g., responding to well-learned but changing contingencies vs. progressing in skill learning) determine how the dorsal striatum is engaged.

While there is an anatomical basis for striatal loop architecture, and some functional evidence, there is as yet no clear demonstration that serial plasticity across a hypothetical loop starting in ventromedial striatum and ending in dorsolateral striatum occurs during learning. Broadly, it is unclear how VTA, SNc, and ventral and dorsal striatum all interact in vivo, and under what changes in ventral versus dorsal striatal dopamine signaling, as described above, require plasticity outside the striatum, such as engagement of corticostriatal or thalamostriatal inputs (Aoki et al., 2019; Athalye, Santos, Carmena, & Costa, 2018; Balleine & O'Doherty, 2010; Cover et al., 2019; Gremel & Costa, 2013; Mandelbaum et al., 2019; Rothwell et al., 2015; Smith & Graybiel, 2013). Notably, relatively little work has examined the central portion of the striatum. Investigations there could be informative for testing the serial midbrain-striatal loop architecture, which would implicate the central striatum as a transition zone between ventral and dorsal. If this was the case, we would expect dopamine signals there to emerge after the ventral striatum, but before the dorsal striatum. Alternatively, the central striatum could have unique functions and dopamine signaling dynamics during learning, which might suggest that ventral and dorsal striatal dopamine systems are comprised of parallel, but independent, loops rather than one continuous serial network.

3.6 Open questions and future directions

4.1 Tonic activity and release

Low-frequency tonic dopamine cell body activity occurs in the 2–5 Hz range (Goto, Otani, & Grace, 2007). Tonic dopamine release induced by an increase in population activity is distinct from neurotransmission induced by burst firing (Floresco, West, Ash, Moore, & Grace, 2003). However, recent work showed that tonic-scale dopamine release is not necessarily due to changes in tonic cell body activity (Mohebi et al., 2019), illustrating that additional refinement of the signal occurs at the terminal. One regulatory mechanism occurs via cholinergic signaling, as blockade of β2 nicotinic receptors located on dopamine terminals suppresses tonic release (Zhang, Doyon, Clark, Phillips, & Dani, 2009). Many studies report a slow rise in striatal dopamine levels during extended reward seeking (Cousins, Trevitt, Atherton, & Salamone, 1999; McCullough, Cousins, & Salamone, 1993; Ostlund, Wassum, Murphy, Balleine, & Maidment, 2011; Sokolowski, Conlan, & Salamone, 1998). This slow increase in dopamine release correlates with the amount of behavior output, but not amount of reward delivered (Sokolowski et al., 1998), suggesting a role in mediating effort expenditure. NAC dopamine depletions suppress effortful reward seeking for a high-palatable food when a less palatable, low-effort option is also available (Sokolowski & Salamone, 1998). Chronic dopamine depletions alter reward seeking particularly when the effort required to pursue reward is high (Salamone, Correa, Farrar, & Mingote, 2007). Additionally, knocking down striatal dopamine transporters (DATs), which increases tonic activity without altering phasic activity, increases reward seeking (Cagniard et al., 2006).

Tonic dopamine is thought to encode net costs and benefits of the current situation, tracking reward rate and any associated costs over long time periods (Niv, Daw, Joel, & Dayan, 2007). In line with this theory, a change in tonic dopamine negatively correlates with effort required to obtain a reward (Ostlund et al., 2011). Tonic dopamine also tracks motivational state information, as tonic dopamine efflux is reduced when animals are sated, compared to a hungry state (Ostlund et al., 2011). Together, these data suggest that slower, tonic changes in dopamine concentrations in the ventral striatum control motivation broadly, to regulate effort expenditure based on physiological state and work demands (Figure 2c).

4.2 Phasic activity and release

Many studies show that phasic midbrain dopamine neuron activity (Cohen, Haesler, Vong, Lowell, & Uchida, 2012; Eshel et al., 2016; Keiflin & Janak, 2015; Schultz et al., 1997; Waelti, Dickinson, & Schultz, 2001) and phasic striatal dopamine release in the NAC (Day et al., 2007; Hart, Rutledge, Glimcher, & Phillips, 2014) closely correlate with the reward-prediction error term proposed by temporal difference reinforcement learning theories to mediate some forms of associative learning. As an animal learns that cues predict rewards, dopamine activity diminishes to the reward and emerges to the predictive cue (Cohen et al., 2012; Day et al., 2007; Schultz et al., 1997), although not always (Coddington & Dudman, 2018). In situations where multiple cues predict rewards, with extended training, phasic dopamine release diminishes to the cues most proximal to reward (Clark et al., 2013) and emerges to earlier cues, to contribute to reward expectation (Collins, Greenfield, et al., 2016).

Recent work has emphasized that the fundamentals of this system can operate independently of sensory input that typically accompanies physical interaction with rewards. Direct optogenetic stimulation of dopamine neurons, in the absence of an external reward, elicits phasic dopamine neuron activity (Saunders et al., 2018) and phasic dopamine release in the NAC (Covey & Cheer, 2019) that back propagates to a cue-predicting stimulation. In natural cue-reward learning, expectation-related suppression of reward-evoked dopamine signals partially arises from plasticity that recruits local GABA neurons in the midbrain, which inhibit dopamine neurons during the period of expectation after cue delivery (Eshel et al., 2015). Remarkably, dopamine release in response to direct optogenetic dopamine cell body stimulation, which largely circumvents the pre-synaptic inhibitory mechanism within the VTA (Covey & Cheer, 2019), diminishes when it becomes well predicted, suggesting that expectation-related suppression of dopamine release evoked by learned rewards likely also occurs via mechanisms at the terminal.

The role of burst activity or rapid release patterns in response to aversive events is informative to this notion, as the presence of an unexpected aversive outcome is thought to be encoded by this system the same way as the absence of an expected appetitive outcome, that is, “worse than expected.” In line with this, dopamine neuron activity within the VTA generally decreases to a noxious stimulus (Ungless, Magill, & Bolam, 2004), though this is not entirely uniform within VTA, as some neurons are activated by the same noxious stimuli (Brischoux, Chakraborty, Brierley, & Ungless, 2009). Phasic release to aversive events is also heterogeneous depending on ventral striatal subregion. For example, dopamine release is suppressed in the NAC and lateral NAS, but increased in the medial NAS, by foot shock. Interestingly, after learning, a cue-predicting shock also increases medial NAS dopamine release (Badrinarayan et al., 2012), suggesting that cue-evoked dopamine may not always encode the valence of a prediction error, but rather an absolute change between expectation and outcome (Yang et al., 2018).

In addition to mediating learning about predictive cues, phasic activity (Ilango, Kesner, Broker, Wang, & Ikemoto, 2014; Jin & Costa, 2010; Zweifel et al., 2009) and release (Phillips, Stuber, Heien, Wightman, & Carelli, 2003; Roitman, Stuber, Phillips, Wightman, & Carelli, 2004; Wassum, Ostlund, Balleine, & Maidment, 2011; Wassum, Ostlund, & Maidment, 2012) encode key aspects of reward-seeking actions. Phasic NAC dopamine release precedes self-initiated action and relates to both effort and vigor of responding (Collins, Greenfield, et al., 2016; Ko & Wanat, 2016; Syed et al., 2016). NAC dopamine release preceding action initiation is observed under circumstances where the action is self-guided, as opposed to driven by external cues, wherein behavioral movement precedes dopamine release (Coddington & Dudman, 2019; Satoh, Nakai, Sato, & Kimura, 2003; Syed et al., 2016). Additionally, phasic NAC dopamine release encodes the value of impending rewards but not costs to obtain them (Gan, Walton, & Phillips, 2010).

4.3 Ramp-like release

A majority of studies investigating phasic or tonic dopamine profiles use experimental designs in which the actions and cues are closely linked to rewards, either temporally or spatially. Recent reports have profiled a potential third type of dopamine release in the ventral striatum that gradually increases as an animal navigates their environment or completes a series of actions to procure a distal reward, peaking near reward consumption (Collins, Greenfield, et al., 2016; Hamid et al., 2016; Howe et al., 2013). This ramp-like profile is more sustained than typical (<1sec) phasic release profiles but is contained within a single reward-seeking behavioral sequence, in contrast to a classic tonic dopamine profile.

Ramping release profiles may provide the motivational drive necessary to continue goal pursuit, as they scale with distance to reward (Howe et al., 2013). In addition to spatial navigation, dopamine ramps occur during instrumental action sequences, in which persistent motivation is also required to earn a distal reward (Collins, Greenfield, et al., 2016; Hamid et al., 2016). In this situation, ramping dopamine release occurs over time scales that bridge choices. Within ramps, dopamine release is dynamic with learning and expectation violation (Collins, Greenfield, et al., 2016), and overall ramping profiles encode a multiplexed signal that is thought to include prediction errors and current motivational state (Hamid et al., 2016). It is interesting to note that with extended instrumental training, the ramp-like profile of dopamine release disappears and only re-emerges when an animal takes an atypical path to navigate through a sequence task, which has implications for the role of these signals in motivation and value encoding with stereotyped actions (Collins, Greenfield, et al., 2016).

It remains unclear how the dopamine system generates an extended, but not exactly tonic, release profile. Understanding this is important, because dopamine ramps most directly coincide with the expression of reward-seeking behavioral sequences. One potential explanation is ramping release follows from a ramping in cell body activity. Indeed, single-unit dopamine neuron activity can show a positive slope ramp-like profile during a probabilistic Pavlovian task, where reward delivery is uncertain (Fiorillo, Tobler, & Schultz, 2003) and in a self-guided instrumental task (Romo & Schultz, 1990). Other studies, however, have found a negatively sloped ramp-like change in activity when the probability of reward delivery is uncertain (Bromberg-Martin et al., 2010a; Fiorillo, Newsome, & Schultz, 2008; Starkweather et al., 2017; Starkweather, Gershman, & Uchida, 2018). Given the uncertainty of reward delivery, this negative slope has been interpreted to reflect a gradual negative reward prediction error, encoding the accumulation of “worse than expected” across trials. That negatively sloped ramps encode a reward prediction error, as opposed to motivational or value signal, suggests a distinction between ramping profiles of dopamine cell activity and ramp-like release signals. Several other reports do not demonstrate a ramping-up profile in single-unit activity (Bromberg-Martin & Hikosaka, 2009; Hollerman & Schultz, 1998; Hollerman, Tremblay, & Schultz, 1998; Matsumoto & Hikosaka, 2007, 2009; Mirenowicz & Schultz, 1996; Morris, Arkadir, Nevet, Vaadia, & Bergman, 2004; Satoh et al., 2003), and recent work showed that dopamine release ramps do not correspond to phasic cell body activity (Berke, 2018; Mohebi et al., 2019). Thus, the evidence for single-unit dopamine neuron ramping activity is mixed. Population-level activity may instead produce ramps, which could reflect sequential, overlapping activation patterns across single units that individually encode distinct aspects of a behavioral sequence, such as spatial, sensory, movement, and decision features (Engelhard et al., 2019). Collectively, such a serial cell body activation pattern could produce a ramp in dopamine release in the ventral striatum that encodes multiple aspects of the ongoing behavior.

4.4 Local modulation for diverse dopaminergic profiles

Critically, dopamine cell body firing does not always track with or explain release (Cachope & Cheer, 2014; Mohebi et al., 2019; Sulzer, Cragg, & Rice, 2016), and thus an important feature of phasic/ramping release dynamics likely occurs via modulation at the terminal. Dopamine terminals contain a number of mechanisms for local modulation, including dopamine reuptake by DATs (Giros, Jaber, Jones, Wightman, & Caron, 1996; Jones et al., 1998, 1999; Rice & Cragg, 2008) and suppression of release via activation of D2 dopamine autoreceptors (Beaulieu & Gainetdinov, 2011; Ford, 2014; Missale, Nash, Robinson, Jaber, & Caron, 1998; Romanelli, Williams, & Neve, 2010).

Local interneuron circuits near dopamine terminals also play a key role in control of dopamine release dynamics. Recent work implicates striatal CINs in a variety of dopamine-regulatory mechanisms. Critically, the dynamics of cholinergic regulation of dopamine release at the terminal are dependent on the activity state of both neuronal populations, which has occluded a clear mechanistic understanding. In anesthetized rats, optogenetic stimulation of CINs in the NAC produces dopamine release independent of dopamine neuron stimulation (Cachope & Cheer, 2014), suggesting that acetylcholine release facilitates striatal dopamine release. The activity state of dopamine neurons matters, however, as a series of in vitro studies suggests a more dynamic mechanism (Exley & Cragg, 2008; Rice & Cragg, 2004; Threlfell et al., 2012; Zhang & Sulzer, 2004; Zhang et al., 2009). Blockade of nicotinic receptors coupled with low-frequency dopamine neuron stimulation suppresses dopamine release, in line with the notion that acetylcholine facilitates dopamine release. In contrast, blockade of nicotinic receptors coupled with high-frequency stimulation increases dopamine release, suggesting that acetylcholine can serve a gating function to suppress release when dopamine neurons burst fire (Exley & Cragg, 2008; Sulzer et al., 2016; Threlfell & Cragg, 2011). Indeed, disrupting nicotinic receptor signaling can actually increase high-frequency dopamine release (Exley et al., 2008; Rice & Cragg, 2004). Together, this work suggests that, based on the current firing mode of a dopamine neuron, acetylcholine release in the ventral striatum can bi-directionally modulate dopamine release. While the details will require direct testing, Figure 2 summarizes our working model of how dopamine release in the ventral striatum might be regulated by CINs across different behavioral and neuronal activity states.

A caveat of the in vitro work is that it assumes high-frequency electrical stimulation of dopamine neurons effectively mimics burst firing of cell bodies that occurs in vivo. Until recently it was largely unclear how acetylcholine modulates dopamine release in freely behaving animals during reward seeking. One hurdle is difficulty in examining the effects of acetylcholine receptor activity on dopamine release locally within the striatum, without simultaneously affecting dopamine cell bodies. A recent paper circumvented this issue by combining local pharmacological manipulation of acetylcholine signaling with simultaneous electrochemical measurements of dopamine release during behavior (Collins, Aitken, et al., 2016). This work demonstrated that intra-NAC blockade of nicotinic receptors can enhance dopamine release to an unexpected reward, as well as to a reward-associated cue, and also increases the ability of that cue to invigorate reward-seeking actions (Collins, Aitken, et al., 2016), suggesting that a pause in CIN transmission allows for dopamine neuron burst activity to produce phasic dopamine release and promote cue-motivated behavior (Figure 2a). Given that reward-paired cues elicit burst firing from dopamine neurons (Schultz et al., 1997), this result suggests that acetylcholine suppresses dopamine release in vivo when dopamine neurons are in a high, but not low, activity mode (Figure 2b,c), in line with the in vitro data. Additionally, rapid cholinergic modulation of striatal dopamine release has important behavioral consequences. Brief optogenetic stimulation of CINs in the NAC during presentation of a reward-paired cue suppresses the ability of that cue to invigorate reward seeking (Collins et al., 2019). This effect occurs via local acetylcholine action, as simultaneous blockade of β2 nicotinic receptors in NAC reversed the behavioral effect of optogenetic stimulation (Collins et al., 2019).

How dopamine and acetylcholine interact to produce heterogeneous functional profiles of release is not fully understood. Acetylcholine, via β2 activation, can suppress dopamine release, but if burst dopamine activity occurs prior to that, there is a rapid shutdown of the CINs via D2 receptor activity (DeBoer & Abercrombie, 1996; Maurice et al., 2004), suggesting that reciprocal suppression of either acetylcholine or dopamine release can occur depending on the current activity state of either cell type. Optogenetically stimulated acetylcholine induces dopamine release in vivo in anesthetized subjects (Cachope & Cheer, 2014), suggesting that activation of nicotinic receptors on dopamine terminals enhances release when dopamine neurons are firing in a low-frequency state, while blockade of β2 nicotinic receptors suppresses tonic release (Zhang et al., 2009). This is a potential mechanism by which the dopaminergic ramp-like profile could be generated at the terminal, given that release ramps dissociate from high-frequency dopamine neuronal firing (Mohebi et al., 2019). After an initial burst of dopamine neuron activity in response to a cue, reflecting expectation, dopamine neurons shift to a low-firing rate. If sustained motivation is then necessary to acquire the reward, CINs could sustain release at the terminal, creating a ramping release profile (Figure 2d).

It remains unclear how specific activity patterns of ventral striatal CINs could contribute to a ramp-like profile of dopamine release, as there is no direct evidence that CINs themselves display a ramp activity pattern. In the ventral striatum, most CIN studies to date involve single cell recordings, preventing understanding of the broader population activity that could regulate dopamine release, and CIN activity patterns typically are not measured in behavioral tasks where sustained motivation is required, that is, the conditions where dopamine ramps are likely to emerge. This requires further study, along with the possibility that acetylcholine release dissociates from action potentials in CINs, as it appears to do in dopamine neurons (Mohebi et al., 2019; Trulson, 1985). In the dorsal striatum, CIN and dopamine terminal activity can propagate in waves across relatively large swaths of tissue (Hamid et al., 2019; Rehani et al., 2019). It is unknown if ventral striatal CINs can also act synchronously, or if ventral striatal dopamine terminals activate in waves. One possibility, which requires direct testing, is that the consequences of dopamine–acetylcholine interactions in dorsal and ventral striatum are broadly distinct where, in dorsal striatum they result in wave patterns, and in ventral striatum they result in ramp patterns. Alternatively, given the similar activity-dependent functioning of α6/α4β2 nicotinic receptors across striatal subregions (Threlfell & Cragg, 2011), if dopamine ramps also occur in the dorsal striatum, CINs there may have a similar role in local modulation.

How are ventral striatal CINs regulated to facilitate or suppress dopamine release? CINs fire at low frequencies, from 3 to 10 Hz, maintained by intrinsic pacemaker functions (Bennett & Wilson, 1999; Wilson & Goldberg, 2006). CIN activity is also altered in response to a variety of inputs. Stimulation of glutamatergic thalamic inputs induces a brief increase of CIN activity followed by a cessation, a “burst-pause” profile (Ding, Guzman, Peterson, Goldberg, & Surmeier, 2010). Interestingly, recent work demonstrated that CIN pauses can also be driven by a decline of excitatory drive (Zhang, Yamada, Gomeza, Basile, & Wess, 2002). Lesion of dopaminergic inputs prevents CIN pauses (Aosaki, Graybiel, & Kimura, 1994), while an increase in dopaminergic activity can induce a pause in CINs via D2 receptor activation (DeBoer & Abercrombie, 1996; Maurice et al., 2004). This varies by striatal subregion. In the dorsal striatum, dopamine stimulation induces a pause in CINs, whereas in the NAS, a burst-pause activity pattern is produced, occurring via co-release of glutamate (Chuhma, Mingote, Moore, & Rayport, 2014; Chuhma et al., 2018). However, when measured together in vivo, CIN pause is not differentially regulated by dopamine neuron firing rate, suggesting this mechanism may be limited in its ability to induce dynamic pauses during behavior (Joshua, Adler, Mitelman, Vaadia, & Bergman, 2008; Morris et al., 2004). Long-range GABAergic projections from the VTA are also capable of inducing a pause in CINs (Brown et al., 2012). Additionally, an increase in acetylcholine can produce a suppression of CIN activity via M4 autoreceptors on CINs (Calabresi et al., 1998; Yan & Surmeier, 1996; Zhang et al., 2002). A number of other specific signaling molecules also influence CINs. Corticotrophin-releasing factor in the NAC or dorsal striatum, for example, can enhance CIN activity (Lemos, Shin, & Alvarez, 2019). That multiple mechanisms modulate CINs suggests striatal circuitry is setup to tightly regulate acetylcholine signaling, allowing for behavioral effects via enhancement or suppression of dopamine release and MSN activity.

Broadly, dopaminergic signaling profiles, including kinetics, volume, and duration, differ across striatal subregions (Brown et al., 2011; Cragg, Hille, & Greenfield, 2000; Howe & Dombeck, 2016; Jones, Garris, Kilts, & Wightman, 1995; Sulzer et al., 2016), reflecting the combination of a number of factors, including differences in the intrinsic properties of dopamine neuron subpopulations, and subregional diversity in local microcircuitry control (Burke et al., 2017; Margolis et al., 2008; Threlfell & Cragg, 2011). A major influence is the distribution of DAT, which is in higher concentrations in the dorsal striatum (Ciliax et al., 1995). This means that on average, dorsal striatum dopamine is removed from extracellular space faster, relative to the ventral striatum, which has consequences for release profile dynamics (Calipari, Huggins, Mathews, & Jones, 2012; Cragg & Rice, 2004; Garris & Wightman, 1994; Jones et al., 1995). For the striatal cholinergic system, the distribution and function of nicotinic (Exley & Cragg, 2008; Exley et al., 2011) and muscarinic (Threlfell et al., 2010) acetylcholine receptors differs across dorsal and ventral striatum (Threlfell & Cragg, 2011). M2 and M4 acetylcholine receptors, which can act as autoreceptors on CINs, are necessary for cholinergic regulation of dopamine release within the dorsal striatum, but only M4 receptors play a role in the ventral striatum (Threlfell et al., 2010). Other gross feature distinctions between the dorsal and ventral striatum, such as the organization of striosome and matrix compartments (Graybiel, Ragsdale, Yoneoka, & Elde, 1981), which exhibit unique local circuitry, connectivity, and function (Crittenden & Graybiel, 2011; Smith et al., 2016), create microenvironments with unexplored functional dynamics.

4.5 Open questions and future directions