Basic model formulation

Hidden Markov models (HMMs) are a class of statistical models for sequential data, in most instances related to systems evolving over time. The system of interest is modelled using a state process (or system process; Table 1), which evolves dynamically such that future states depend on the current state. Many ecological phenomena can naturally be described by such a process (Fig. 1). In an HMM, the state process is not directly observed – it is a ‘hidden’ (or ‘latent’) variable. Instead, observations are made of a state-dependent process (or observation process) that is driven by the underlying state process. As a result, the observations can be regarded as noisy measurements of the system states of interest, but they are typically insufficient to precisely determine the state. Mathematically, an HMM is composed of two sequences:

In most applications, the indices refer to observations made over time at a regular sampling interval (e.g. daily or annual rainfall measurements), but they can also refer to position (e.g. in a sequence of DNA; Henderson et al., 1997; Eddy, 2004) or order (e.g. in a sequence of marine mammal dives; DeRuiter et al., 2017). HMMs can also be formulated in continuous time (Jackson et al., 2003; Amoros et al., 2019), but these models have tended to be less frequently applied in ecology (but see Langrock et al., 2013; Choquet et al., 2017; Olajos et al., 2018). Among the many HMM formulations of relevance to ecology that we highlight in ECOLOGICAL APPLICATIONS OF HIDDEN MARKOV MODELS, some example observation sequences and underlying states include:

•  Observation of feeding/not feeding, with underlying state Hungry or sated;
•  Count of individuals, with underlying state True population abundance; and
•  Daily rainfall measurement, with underlying state Wet or dry season.

Unlike the larger class of state-space models (see Box 1), the state process within an HMM can take on only finitely many possible values: for . The basic HMM formulation further involves two key dependence assumptions: (1) the probability of a particular state being active at any time is completely determined by the state active at time (the so-called Markov property); and (2) the probability distribution of an observation at any time is completely determined by the state active at time (Fig. 2). The latter assumption is a conditional independence property, as this implies that is conditionally independent of past and future observations, given . Whether these simplifying assumptions can faithfully characterise the underlying dynamics for the system of interest must be carefully considered (see Challenges and pitfalls).

As a consequence of these assumptions, HMMs generally facilitate model building and computation that might otherwise be intractable. A basic -state HMM that formally distinguishes the state and observation processes can be fully specified by the following three components: (1) the initial distribution, , specifying the probabilities of being in each state at the start of the sequence; (2) the state transition probabilities, , specifying the probability of switching from state at time to state at time and usually represented as an state transition probability matrix:

where ; and 3) the state-dependent distributions, , specifying the probability distribution of an observation conditional on the state at time and usually represented as an diagonal matrix:

or, equivalently, for computational purposes (see Inference). These distributions can pertain to discrete or continuous observations and are generally chosen from an appropriate distributional family. For example, behavioural observation could be modelled using a categorical distribution (MacDonald and Raubenheimer, 1995), count using a non-negative discrete distribution (e.g. Poisson; Besbeas and Morgan, 2019, and measurement using a non-negative continuous distribution (e.g. zero-inflated exponential; Woolhiser and Roldan, 1982). After specifying , and in terms of the particular system of interest, one can proceed to drawing inferences about unobservable state dynamics from the observation process.

We note that Markov models (Grewal et al., 2019) are commonly used for inferring community- or ecosystem-level dynamics (Waggoner and Stephens, 1970; Wootton, 2001; Tucker and Anand, 2005; Breininger et al., 2010) and providing measures of stability, resilience or persistence (Li, 1995; Pawlowski and McCord, 2009; Zweig et al., 2020), especially in systems composed of sessile organisms such as plant (Horn, 1975; van Hulst, 1979; Usher, 1981; Talluto et al., 2017, but see Chen et al., 2013) or benthic communities (Tanner et al., 1994; Hill et al., 2004; Lowe et al., 2011). A Markov model can simply be viewed as an HMM where it is assumed that the state process is perfectly observed, that is, with a matrix with entry one in row , column , and otherwise zeros. For example, patch dynamics HMMs (MacKenzie et al., 2003) are simply generalisations of well-known Markov models for patch dynamics (Hanski, 1994; Moilanen, 1999) for cases when presence–absence data are subject to imperfect detection. Likewise, any Markov model can naturally be embedded as the state process within an HMM for less observable phenomena.

Inference

In addition to the ease with which a wide variety of ecological state and observation process models can be specified (see ECOLOGICAL APPLICATIONS OF HIDDEN MARKOV MODELS), a key strength of the HMM framework is that efficient recursive algorithms are available for conducting statistical inference. Here we will briefly outline some of the most common inferential techniques for HMMs, but motivated readers can find additional technical material and a worked example on model fitting, assessment and interpretation in the Supplementary Tutorial. Using the forward algorithm (also known as filtering), the likelihood as a function of the unknown parameters given the observation sequence can be calculated at a computational cost that is (only) linear in . The parameter vector , which is to be estimated, contains any unknown parameters embedded in the three model-defining components , and . Made possible by the relatively simple dependence structure of an HMM, the forward algorithm traverses along the time series, updating the likelihood step-by-step while retaining information on the probabilities of being in the different states (Zucchini et al., 2016, pp. 37–39). Application of the forward algorithm is equivalent to evaluating the likelihood using a simple matrix product expression,

(1)

where 1 is a column vector of ones (see Supplementary Tutorial for technical derivation).

In practice, the main challenge when working with HMMs tends to be the estimation of the model parameters. The two main strategies for fitting an HMM are numerical maximisation of the likelihood (Myung, 2003; Zucchini et al., 2016) or Bayesian inference (Ellison, 2004; Gelman et al., 2004) using Markov chain Monte Carlo (MCMC) sampling (Brooks et al., 2011). The former seeks to identify the parameter values that maximise the likelihood function (i.e. the maximum likelihood estimates ), whereas the latter yields a sample from the posterior distribution of the parameters (Ellison, 2004). Specifically for the maximum likelihood (ML) approach, the forward algorithm makes it possible to use standard optimisation methods (Fletcher, 2013) to directly numerically maximise the likelihood (eqn 1). An alternative ML approach is to employ an expectation–maximisation (EM) algorithm that uses similar recursive techniques to iterate between state decoding and updating the parameter vector until convergence (Rabiner, 1989). For MCMC, many different strategies can be used, but these tend to differ in appropriateness and efficiency in a manner that can strongly depend on the specific model and data at hand (Gilks et al., 1996; Gelman et al., 2004; Brooks et al., 2011; Robert and Casella, 2004).

The forward algorithm and similar recursive techniques can further be used for forecasting and state decoding, as well as to conduct formal model checking using pseudo-residuals (Zucchini et al., 2016, Chapters 5 & 6). State decoding is usually accomplished using the Viterbi algorithm or the forward–backward algorithm (also known as smoothing), which respectively identify the most likely sequence of states or the probability of each state at any time , conditional on the observations. Fortunately, practitioners can often use existing software for most aspects of HMM-based data analyses and need not dwell on many of the more technical details of implementation (see IMPLEMENTATION, CHALLENGES AND PITFALLS and Supplementary Tutorial).

To illustrate some of the basic mechanics, we use a simple example based on observations of the feeding behaviour of a blue whale (Balaenoptera musculus; cf. DeRuiter et al., 2017). Suppose we assume that observations of the number of feeding lunges performed in each of consecutive dives ( for ) arise from states of feeding activity. Building on Fig. 2, we could for example have:

Fig. 3 displays the results for this simple two-state HMM assuming Poisson state-dependent (observation) distributions, for , when fitted to the full observation sequence via direct numerical maximisation of eqn 1. The rates of the state-dependent distributions were estimated as and , suggesting states 1 and 2 correspond to ‘low’ and ‘high’ feeding activity respectively. The estimated state transition probability matrix,

suggests interspersed bouts of ‘low’ and ‘high’ feeding activity, but with bouts of ‘high’ activity tending to span fewer dives. The estimated initial distribution suggests this individual was more likely to have been in the ‘low’ activity state at the start of the sequence. Most ecological applications of HMMs involve more complex inferences related to specific hypotheses about system state dynamics, and a great strength of the HMM framework is the relative ease with which the basic model formulation can be modified to describe a wide variety of processes (Zucchini et al., 2016, Chapters 9–13). Next we highlight some extensions that we consider to be highly relevant in ecological research.

Extensions

The dependence assumptions made within the basic HMM are mathematically convenient, but not always appropriate (see Box 2). The Markov property implies that the amount of time spent in a state before switching to another state – the so-called sojourn time – follows a geometric distribution. The most likely length of any given sojourn time hence is one unit, which may not be realistic for certain state processes. The obvious extension is to allow for th-order dependencies in the state process (Fig. 4a), such that the state at time depends on the states at times . An alternative assumes the state process is ‘semi-Markov’ with the sojourn time flexibly modelled using any distribution on the positive integers (Choquet et al., 2011; van de Kerk et al., 2015; King and Langrock, 2016).

HMMs are often used to infer the drivers of ecological state processes by relating the state transition probabilities to explanatory covariates (Fig. 4b). Indeed, any of the parameters of a basic HMM can be modelled as a function of covariates (e.g. sex, age, habitat type, chlorophyll-a) using an appropriate link function (McCullagh and Nelder, 1989). Link functions can relate the natural scale parameters to a design matrix of covariates and -vector of working scale parameters such that (see White and Burnham, 1999; MacKenzie et al., 2002; Patterson et al., 2009, for common examples of link functions in HMMs). When simultaneously analysing multiple observation sequences, heterogeneity across the different sequences can be modelled through explanatory covariates or mixed HMMs that include random effects (Altman, 2007; Schliehe-Diecks et al., 2012; Towner et al., 2016).

At the level of the observation process, it is relatively straightforward to relax the conditional independence assumption. For example, it can be assumed that the observation at time depends not only on the state at time but also the observation at time (Fig. 4c; Langrock et al., 2014b; Lawler et al., 2019). It is also straightforward to model multivariate observation sequences using multivariate state-dependent distributions (Choquet et al., 2013; Phillips et al., 2015; van Beest et al., 2019), where it is often assumed that the different variables observed are conditionally independent and a univariate distribution is specified for each of the variables (Fig. 4d). Owing to the Markov property, this does not imply that the individual components are serially independent or mutually independent (Zucchini et al., 2016, Chapter 9). However, this assumption is not required and will not always be appropriate, in which case a multivariate distribution should be considered.

Individual level

Existential state

At the level of an individual organism, a fundamental measure of existence is to be alive or not (i.e. dead or unborn). We will therefore begin by demonstrating that one of the oldest and most popular inferential tools in wildlife ecology, the Cormack-Jolly-Seber (CJS) model of survival (Williams et al., 2002), is a special case of an HMM. The CJS model estimates survival probabilities () from capture–recapture data. Capture–recapture data consist of sequences of encounter histories for marked individuals collected through time, where for each individual the observed data are represented as a binary series of ones and zeros. For the CJS model, indicates a marked individual was alive and detected at time , while indicates non-detection. Marked individuals can either be alive or dead at time , but the ‘alive’ state is only partially observable and the ‘dead’ state is completely unobservable. Under this observation process, if it is known that the individual survived from time to time (with probability ) and was detected with probability . However, when there are two possibilities: (1) the individual survived to time (with probability ) but was not detected (with probability ); or (2) the individual did not survive from time to time (with probability ).

Although not originally described as such, the CJS model is simply a two-state HMM that conditions on first capture. Framing the observed and hidden processes within the dependence structure of a basic HMM (Fig. 2), we could for example have:

The state-dependent observation distribution for is a simple Bernoulli (i.e. a coin flip) with success probability if alive and success probability if dead:

We thus have the initial distribution

state transition probability matrix

and state-dependent observation distribution matrix

The CJS model is thus a very simple HMM with an absorbing ‘dead’ state and only two unknown parameters ( and ). As an HMM, it can not only be used to estimate survival, but also the point in time when any given individual was most likely to have died (based on local or global state decoding; see Table 1).

The classic Jolly-Seber capture–recapture model and its various extensions (Pradel, 1996; Williams et al., 2002) go a step further by incorporating both birth and death processes. It simply involves extending the two-state model to an additional ‘unborn’ (UB) state. We could for example now have:

To formulate a three-state HMM with an additional ‘unborn’ state, we must extend our components for the hidden and observed processes accordingly:

and

where

is the probability that an individual was already in the population at the beginning of the study, is the probability that any given individual was born at time , and is the probability that an individual entered the population on occasion given it had not already entered up to that time. Importantly, note that the two-state and three-state HMMs rely on the exact same binary data , but we are able to make additional inferences in the three-state model by re-formulating the observed and hidden processes in terms of both birth and death. While we have employed these well-known individual-level capture–recapture models to initially demonstrate the key idea of linking observed state-dependent processes to the underlying state dynamics via HMMs, these types of inferences are not limited to traditional capture–recapture observation processes. For example, telemetry and count data can also be used in HMMs describing individual-level birth and death processes (Schmidt et al., 2015; Cowen et al., 2017).

Developmental state

Individual-level data often contain additional information about developmental states such as those related to size (Nichols et al., 1992), reproduction (Nichols et al., 1994), social groups (Marescot et al., 2018) or disease (Benhaiem et al., 2018). However, assigning individuals to states can be difficult when traits such as breeding (Kendall et al., 2012), infection (Chambert et al., 2012), sex (Pradel et al., 2008) or even species (Runge et al., 2007) are ascertained through observations in the field. This difficulty has motivated models for individual histories that can not only account for multiple developmental states (Lebreton et al., 2009), but also uncertainty arising from partially or completely unobservable states (Pradel, 2005). Such multi-state models can be used for testing a broad range of formal biological hypotheses, including host–pathogen dynamics in disease ecology (Lachish et al., 2011), reproductive costs in evolutionary ecology (Garnier et al., 2016) and social dominance in behavioural ecology (Dupont et al., 2015). For example, it is straightforward to extend the capture–recapture HMM to multiple ‘alive’ states parameterised in terms of state-specific survival probabilities and transition probabilities between these ‘alive’ states . Consider a -state HMM for capture–recapture data that incorporates reproductive status, where indicates ‘alive and breeding’ and indicates ‘alive and non-breeding’:

and

where is an indicator function taking the value 1 when and 0 otherwise. To assess the costs of reproduction, a biologist will be interested in the probability of breeding in year , given breeding or not in year , as well as assessing any differences in survival probability between breeders and non-breeders . By simply re-expressing the , and components in terms of the specific state and observation processes of interest, such models can be used to infer the dynamics of conjunctivitis in house finches (Conn and Cooch, 2009), senescence in deer (Choquet et al., 2011), reproduction in Florida manatees (Kendall et al., 2012), interspecific competition between ungulates (Gamelon et al., 2020) and life-history trade-offs in elephant seals (Lloyd et al., 2020). Similar HMMs can also be used to investigate relationships between life-history traits and demographic parameters that are important in determining the fitness of phenotypes or genotypes (Stoelting et al., 2015). Several measures of individual fitness have been proposed, but one commonly used for field studies is lifetime reproductive success (Rouan et al., 2009; Gimenez and Gaillard, 2018). These approaches can be readily adapted to quantify other measures of fitness (McGraw and Caswell, 1996; Link et al., 2002; Coulson et al., 2006; Marescot et al., 2018).

Inferences about developmental states are of course not limited to traditional capture–recapture data, and significant advancements in animal-borne biotelemetry technology have brought many new and exciting opportunities (Cooke et al., 2004; Hooten et al., 2017; Patterson et al., 2017). For example, telemetry location data can be used to identify migratory phases (Weng et al., 2007), predation events (Franke et al., 2006) or the torpor-arousal cycle of hibernation (Hope and Jones, 2012). The multi-state (i.e. hidden Markov) movement model is often used to infer these types of movement behaviour modes from trajectories in two-dimensional space, where the observations are typically expressed in terms of the bivariate sequence of Euclidean distances (or ‘step lengths’) and turning angles between consecutive locations (Franke et al., 2004; Morales et al., 2004). For a model involving states that assumes conditional independence between step length (; in meters) and turning angle (; in radians) as in Fig. 4d, we could for example have:

These states could correspond to ‘resident’ (state 1) and ‘transient’ (state 2) behavioural phases, such that within state 2 the movements tend to be longer and directionally persistent (i.e. with turning angles concentrated near zero). When assuming conditional independence of the observations, the bivariate state-dependent distribution for is simply the product of two univariate state-dependent distributions,

These univariate distributions are typically assumed to be the gamma or Weibull distribution for step length and the von Mises or wrapped Cauchy distribution for turning angle. Unlike our previous examples so far, the number of underlying states in these types of HMMs is generally not clear a priori and needs to be selected based on both biological and statistical criteria (Pohle et al., 2017). Another difference is that there is often no predetermined structure in the state transition probability matrix,

and all entries are freely estimated (but still subject to ). As a consequence, the characteristics of the model states as represented by the state-dependent distributions are fully data driven, and hence may not correspond exactly to biologically meaningful entities (see IMPLEMENTATION, CHALLENGES AND PITFALLS).

Similar HMMs for animal movement have been used, inter alia, to identify wolf kill-sites (Franke et al., 2006), the relationship between southern bluefin tuna behaviour and ocean temperature (Patterson et al., 2009), activity budgets for harbour seals (McClintock et al., 2013), hunting strategies of white sharks (Towner et al., 2016), the behavioural response of northern gannets to frontal activity (Grecian et al., 2018) and how common noctules adjust their space use to the lunar cycle (Roeleke et al., 2018). Driven by the influx of new biotelemetry sensor technology, HMMs have also been used to analyse the sequences of dives of marine animals (Hart et al., 2010; Quick et al., 2017; DeRuiter et al., 2017; van Beest et al., 2019). The remote collection of activity data at potentially very high temporal resolutions using accelerometers is another emerging application area (Diosdado et al., 2015; Leos-Barajas et al., 2017b; Papastamatiou et al., 2018a,b; Adam et al., 2019b). These HMM formulations are conceptually very similar to the movement model outlined above, with the state process corresponding to behavioural modes (or at least proxies thereof), and the activity data represented by the state-dependent process. Fig. 5 illustrates a possible workflow for inferring four behavioural modes from high-resolution accelerometer data collected from a striated caracara (Phalcoboenus australis) over a period of 1 hour. Here the vector of dynamic body acceleration was used as a univariate summary of the three-dimensional raw acceleration data, and a gamma distribution was used for the state-dependent observation process. In this example, the HMM can be regarded as a clustering scheme which maps observed input data to unobserved underlying classes with biological interpretations roughly corresponding to ‘resting’, ‘minimal activity’ (e.g. preening), ‘moderate activity’ (e.g. walking, digging) and ‘flying’. Complete details of this analysis, including each step of the workflow and example R (R Core Team, 2019) code, can be found in the Supplementary Tutorial.

Spatial state

HMMs can also be used for inferences about the unobserved spatial location of an individual. For example, capture–recapture data can consist of sequences of observations arising from a set of discrete spatial states, where these often refer to ecologically important geographic areas, such as wintering and breeding sites for migratory birds (Brownie et al., 1993) or spawning sites for fish (Schwarz et al., 1993). For a -state HMM with two sites (A and B), where indicates ‘alive at site A’ and indicates ‘alive at site B’, we could for example have:

Clearly, this discrete-space HMM is structurally identical to the multi-state capture–recapture HMMs already described in the previous section; the only difference is the state transition probability parameters are now interpreted as site-specific survival and movement probabilities between the sites (e.g. fidelity or dispersal; Lagrange et al., 2014; Cayuela et al., 2020). Based on global state decoding, these HMMs can therefore also be used to infer the most likely spatial state for periods when an individual was alive but its location was not observed.

Another important application of HMMs is for geolocation based on indirect measurements that vary with space, such as light, pressure, temperature and tidal patterns (Thygesen et al., 2009; Rakhimberdiev et al., 2015). Although too technical to be described in detail here, geolocation HMMs can be particularly useful for inferring individual location from archival tag data (Basson et al., 2016). These HMMs have even been extended to include state-switching behaviours such as those described in the previous section (Pedersen et al., 2008, 2011b). Animal movement behaviour HMMs have also been extended to accommodate partially observed location data common to marine mammal satellite telemetry studies (Jonsen et al., 2005; McClintock et al., 2012).

Population level

We consider two ways that inference on the population level can arise: (1) an individual-level model, based on data from multiple individuals (e.g. capture–recapture), quantitatively connected to a population-level concept through an explicit model; or (2) a population-level model, based on population-level data (e.g. counts or presence–absence), with no explicit model for processes at the individual level.

Existential state

A fundamental existential state at the population level is abundance, the number of individuals alive in a population at a particular point in time. A common way to infer this using capture–recapture HMMs is to formally link abundance to the individual-level processes (e.g. survival, recruitment) that drive its dynamics. Intuitively, the abundance model specifies how many individuals go through the life history specified by the HMM. For the abundance component, the key pieces of information are the number of individuals in the population that were detected at least once and the probability of being detected at least once, given an individual was alive at any time during the study . The former is observed while the latter can be calculated as

using notation for the Jolly-Seber HMM presented in Individual level. This HMM formulation is equivalent to the original Jolly-Seber open population model (shown in Glennie et al., 2019), where population abundance at each time is derived from the individual-level process parameters.

Instead of inducing changes in abundance through individual-level HMMs, abundance itself can be modelled as the hidden state within an HMM (Schmidt et al., 2015; Cowen et al., 2017; Besbeas and Morgan, 2019). Here population dynamics are inferred from population-level surveys (Buckland et al., 2004), where the observation process can include counts or other quantities that are noisy measurements of the true abundance (the hidden state), and the state transition probability matrix is naturally formulated in terms of the well-known Leslie matrix for population growth (Caswell, 2001). For example, for imperfect count data that were collected from a population of true size (note the requirement to specify a maximum possible population size ), we could have:

and

Each state transition probability describes the population dynamics from time to time and can be parameterised in terms of survival, reproduction, emigration, the current population size and any additional population structure (e.g. sex or age classes; see Population level - Developmental state). The state-dependent distributions can take many different forms depending on the specific observation process, but common choices for count data are binomial or Poisson models (Schmidt et al., 2015; Besbeas and Morgan, 2019). Sometimes count data alone can be insufficient for describing complex population processes, and this has led to integrated population modelling (Schaub and Abadi, 2011) that uses auxiliary data such as capture–recapture, telemetry or productivity data (Schmidt et al., 2015; Besbeas and Morgan, 2019).

Spatial state

The spatial state of a population can be conceived as a surface (or map) quantifying density at each point in space, and population models for individual-level data can be extended to allow density to change over space (Borchers and Efford, 2008). Inferring density as a spatial population state, however, requires spatial information within the data. Spatial capture–recapture surveys (Royle et al., 2013), an extension of capture–recapture, collect precisely these data. Spatial capture–recapture HMMs can be formulated in terms of survival, recruitment, movement and population density (Royle et al., 2018; Glennie et al., 2019) and are readily extendable for relating environment and population distribution across space, including how distribution is affected by landscape connectivity, dispersal, resource selection or environmental impacts such as oil spills (McDonald et al., 2017; Royle et al., 2018).

A different viewpoint is to consider population-level data that are commonly collected over both space and time: presence–absence data. These data provide information on a population's spatial state that is not derived from abundance and arise from the monitoring of spatial units for the (apparent) presence or absence of a species. One of the most popular tools for analysing these data are patch (or site) occupancy models, which can be used to infer patterns and dynamics of species occurrence while accounting for imperfect detection (MacKenzie et al., 2018). As with capture–recapture models, patch occupancy models are also HMMs (Royle and Kéry, 2007; Gimenez et al., 2014) where, instead of the state dynamics of individual organisms, the hidden process describes the state dynamics of sites. Let indicate ‘occupied’ and indicate ‘unoccupied’, where the species can be detected or not during multiple visits to each site, with the following representation:

and

where is the initial patch occupancy probability at time , is the species detection probability at each occupied patch and is composed of the local colonisation and extinction probabilities. Single-season (or static) occupancy models (MacKenzie et al., 2002) are obtained as a special case with or (Gimenez et al., 2014). This HMM can not only be used to estimate patch occupancy, extinction and colonisation probabilities, but also the most likely state and times of any colonisation or extinction events within a patch. The flexibility of the HMM formulation allows patch occupancy to be conveniently extended to cope with site-level heterogeneity in detection using finite mixtures (Louvrier et al., 2018) or a discrete measure of population density (Gimenez et al., 2014; Veran et al., 2015) and even false positives due to species misidentification (Miller et al., 2011; Louvrier et al., 2019). Just as with multi-state capture–recapture HMMs (see Population level - Developmental state), species occurrence HMMs can be readily extended to multiple ‘occupied’ states accommodating reproduction (MacKenzie et al., 2009; Martin et al., 2009), disease (McClintock et al., 2010) and other (meta-)population dynamics (Lamy et al., 2013).

Inferences from HMMs for presence–absence data are not limited to occupancy models that account for imperfect species detection. For example, Pluntz et al., (2018) developed an HMM characterising seed dormancy, colonisation and germination in annual plant metapopulations based entirely on presence–absence observations of standing flora. In their study, the presence of a completely unobservable soil seed bank was the hidden state of interest, and they modified the dependence structure of a basic HMM such that the seed bank state dynamics at time depended not only on the seed bank state at time , but also on the presence or absence of standing flora at time . Let indicate ‘seed bank absent at time , flora absent at time ’, indicate ‘seed bank present at time , flora absent at time ’ and indicate ‘seed bank present at time , flora present at time ’, where standing flora is present or not during visit to each site and is assumed to be detected without error. We could for example have:

where is the probability that a seed bank was present the year before the first observation, is the probability of germination and survival to reproduction, is the probability of seed bank survival, is the probability of external colonisation and is a diagonal matrix of ones. Similar formulations could be applied to other organisms with dormant life stages (e.g. fungi, crustaceans).

Community level

Community-level studies often focus on a subset of species based on taxonomy, trophic position or particular interactions of interest, and the diversity of topics addressed in community ecology reflects its large scope (Vellend, 2010, 2016). Here we will only scratch the surface of two study systems that can be formulated as HMMs for multi-species presence–absence data commonly collected from field surveys or (e)DNA samples: (1) patch systems composed of (potentially) many species; and (2) patch systems composed of a few (possibly interacting) species.

Spatial state

Understanding geographic variation in the size and structure of communities is one of the major goals in ecology. While we have so far focused on some of the more ‘non-spatial’ aspects of community-level inference, all multi-species presence–absence HMMs are of course inherently spatial and also describe community distribution. Dynamic multi-species occupancy models provide inferences about changes in community distributions (Russell et al., 2009; Dorazio et al., 2010), and, when spatio-temporal interactions between species are of primary interest, dynamic co-existence HMMs can incorporate local species extinction and colonisation to investigate interspecific drivers of co-occurrence dynamics and community distribution (Fidino et al., 2019; Marescot et al., 2020). As a final illustrative example, suppose we have the states (respectively and ) for ‘site occupied by species A’ (respectively by species B and by both species) and indicates ‘unoccupied site’. Define , where indicates neither species was detected, indicates only species A was detected, indicates only species B was detected and indicates both species were detected on the th visit at time . We could for example have:

This model is more complex than previous examples, but it can still be readily expressed in terms of , and for inferring patterns and drivers of species co-existence distribution dynamics (see Appendix A in Supplementary Material).

Ecosystem level

Despite the well-recognised need for reliable inferences about broad-scale ecological dynamics in the face of climate change and other challenges (Turner et al., 1995), HMMs have thus far seldom been applied at the ecosystem level. This is likely attributable to many factors, including the difficulty of obtaining and integrating observational data at the large spatio-temporal scales required (Jones et al., 2006; Bohmann et al., 2014; Dietze et al., 2018; Estes et al., 2018; Compagnoni et al., 2019). Although there are fewer examples in the literature, HMMs have been used to make ecosystem-level inferences about stability and regime shifts (Gal and Anderson, 2010; Gennaretti et al., 2014; Economou and Menary, 2019), climate-driven community and disease dynamics (Moritz et al., 2008; Martinez et al., 2016; Miller et al., 2018a), the effects of management action on habitat dynamics (Breininger et al., 2010), climatic niches (Tingley et al., 2009) and ecosystem health (Xiao et al., 2019). HMMs are also frequently used by atmospheric scientists, hydrologists and landscape ecologists to describe regional- to global-scale ecosystem processes such as precipitation (Zucchini and Guttorp, 1991; Srikanthan and McMahon, 2001), streamflow (Jackson, 1975; Bracken et al., 2014), wetland dynamics (Siachalou et al., 2014) and land cover dynamics (Aurdal et al., 2005; Lazrak et al., 2010; Trier and Salberg, 2011; Abercrombie and Friedl, 2015; Siachalou et al., 2015). While many of these examples tend to focus on a few specific biotic and/or abiotic components in which to frame ecosystem state dynamics, we can envision future applications adopting a more holistic approach that integrates increasingly more complex ecosystem-level processes with observational data arising from a variety of sources and spatio-temporal scales (see FUTURE DIRECTIONS).

Software

Recent advances in computing power and user-friendly software have made the implementation of HMMs much more feasible for practitioners. However, the features and capabilities of the software are varied, and it can be challenging to determine which software may be most appropriate for a specific objective. We briefly describe some of the HMM software currently available, limiting our treatment to freely available R (R Core Team, 2019) packages and stand-alone programs that we believe are most accessible to ecologists and non-statisticians. While most HMM packages in R include data simulation, parameter estimation and state decoding for an arbitrary number of system states, they differ in many key respects (Table 2). Some of the more general packages provide greater flexibility for specifying state-dependent probability distributions (Visser and Speenkenbrink, 2010; Jackson, 2011; Harte, 2017; McClintock and Michelot, 2018). One of the earliest and most flexible HMM packages, depmixS4 (Visser and Speenkenbrink, 2010), can accommodate multivariate HMMs, multiple observation sequences, parameter covariates, parameter constraints and missing observations. Similar to depmixS4 in terms of features and flexibility, momentuHMM (McClintock and Michelot, 2018) can also be used to implement mixed HMMs (DeRuiter et al., 2017), hierarchical HMMs (Leos-Barajas et al., 2017a; Adam et al., 2019a), zero-inflated probability distributions (Martin et al., 2005) and partially observed state sequences. In addition to the R packages presented in Table 2, there are numerous R and stand-alone software packages that are less general and specialise on particular HMM applications in ecology, as well as general statistical programs with which these types of models can be relatively easily implemented (see Appendix B in Supplementary Material).

Table 2. Features of HMM packages available in the R environment for statistical computing, including capabilities for multiple observation sequences (‘Multiple sequences’), multivariate HMMs (‘Multivariate’), mixed HMMs (‘Mixed’), hierarchical HMMs (‘Hierarchical’), hidden semi-Markov models (‘Semi-Markov’), parameter covariate modelling (‘Covariates’), parameter constraints (‘Constraints’), missing observations (‘Missing data’) and state-dependent probability distributions

Package	Multiple sequences	Multivariate	Mixed	Hierarchical	Semi-Markov	Covariates	Constraints	Missing data	Reference
aphid	✓								Wilkinson (2019)
depmixS4	✓	✓						✓	Visser and Speenkenbrink (2010)
HiddenMarkov									Harte (2017)
HMM									Himmelmann (2010)
hsmm					✓				Bulla and Bulla (2013)
LMest	✓	✓	✓			or		✓	Bartolucci et al., (2017)
mhsmm	✓				✓			✓	O'Connell and Hojsgaard (2011)
momentuHMM	✓	✓	✓	✓				✓	McClintock and Michelot (2018)
msm	✓	✓						✓	Jackson (2011)
RcppHMM									Cardenas-Ovando et al., (2017)
seqHMM	✓	✓	✓					✓	Helske and Helske (2019)

	State-dependent probability distributions
	Bernoulli	Beta	Binomial	Categorical	Custom	Exponential	Gamma	Lognormal	Logistic	Negative binomial	Normal	Multivariate normal	Truncated normal	Poisson	Student's	Von Mises	Weibull	Wrapped Cauchy
aphid				✓
depmixS4			✓	✓	✓		✓				✓			✓
HiddenMarkov		✓	✓		✓	✓	✓	✓	✓		✓			✓
HMM				✓
hsmm	✓										✓			✓	✓
LMest				✓							✓	✓
mhsmm					✓						✓	✓		✓
momentuHMM	✓	✓		✓		✓	✓	✓	✓	✓	✓	✓		✓	✓	✓	✓	✓
msm	✓	✓	✓	✓		✓	✓	✓		✓	✓		✓	✓	✓		✓
RcppHMM				✓							✓	✓		✓
seqHMM				✓

• ‘Covariates’ and ‘Constraints’ can pertain to initial distribution , state-dependent probability distribution , state transition probability and/or mixture probability parameters. Several packages facilitate extensions for user-specified state-dependent probability distributions that require no modifications to the package source code (‘custom’).
• * Covariates are only permitted on state-dependent distribution location parameters for the binomial, gamma, normal and Poisson distributions.
• ^† Covariates are only permitted on state-dependent categorical distribution parameters.
• ^‡ Covariates are only permitted on state-dependent distribution location parameters.
• [Corrections added on 10 November 2020, after first online publication: Table 2 has been updated.]

Challenges and pitfalls

HMMs are natural candidates for conducting inference related to a wide range of ecological phenomena, but they are not a panacea (see Box 2). There are many ecological processes that cannot be faithfully characterised under the simplifying assumptions of HMMs, in which case other latent variable models may be more appropriate (see Box 1). When HMMs are appropriate, it can be challenging to tailor HMMs to real data, even when using user-friendly software packages. Here we briefly highlight those issues that, based on our experience, constitute the key challenges when using HMMs to analyse ecological data. Other important aspects of statistical practice that are not unique to HMMs, such as model checking and selection (e.g. Zucchini et al., 2016, Chapter 6), are covered in more detail in the Supplementary Tutorial.

Depending on the complexity of the state and observation processes, various modelling decisions may need to be made. Among these are the number of states to include, whether to incorporate covariates for the model parameters and whether the basic dependence structure is sufficient. These decisions tend to be case-dependent and require expert knowledge of the system of interest, so we make no attempt to provide general guidance in this respect. However, in some cases the model structure may be a direct consequence of the ecological process. For example, in the CJS model, the two states (alive or dead) and also the state-dependent (Bernoulli) distributions follow immediately from the capture–recapture process. In situations with more complex data, such as multivariate time series related to animal behaviour (DeRuiter et al., 2017; Ngô et al., 2019; van Beest et al., 2019), it takes experience and a good intuition both for the data and for the HMM framework to identify an adequate model formulation (Pohle et al., 2017).

Unlike other statistical models such as linear regression, there is no analytical solution for HMM parameter estimation. One must therefore resort to numerical procedures, all of which involve technical challenges: local maxima for maximum likelihood estimation (Myung, 2003), or label switching (Jasra et al., 2005) and poor mixing (Brooks et al., 2011) for MCMC sampling. Any increase in model complexity with respect to the number of states or the parameters tends to rapidly exacerbate these problems. When working with HMMs, it is thus important to develop an appreciation for these challenges and the associated risks. For maximum likelihood estimation, the risk of false convergence to a local rather than the global maximum of the likelihood must not be underestimated. In addition to the general advice to avoid overly complex models (Lavine, 2010; Cole, 2019), the main strategy to reduce this risk is to try many initial parameter vectors within the maximisation.

While it is tempting to interpret the states of an HMM fitted to ecological data as biologically meaningful entities, this is not always justifiable. Outside the standard capture–recapture or species occurrence applications, HMMs are often applied in an unsupervised learning context (see Figs 3 and 5, Supplementary Tutorial), such that the state characteristics are completely data driven rather than pre-defined (Leos-Barajas et al., 2017b). The model then picks up the statistically most relevant modal patterns in the data, and these may or may not correspond closely to ecologically meaningful states. It is thus important not to over-interpret the model states, as in some cases they may only be crude proxies for the ecological system states of interest. A classic example is the simple state HMM for animal movement behaviour based on step lengths and turning angles (Morales et al., 2004), where evidence of an area-restricted search-type state is often labelled as ‘foraging’. Although for many animals area-restricted search is commonly associated with foraging, one usually cannot definitively conclude when an individual was actually foraging based solely on location data. Furthermore, while it can be useful to refer to these modalities using descriptive terms such as ‘foraging’ (or ‘resident’) and ‘searching’ (or ‘transient’), this does not mean that an animal has only two modes of behaviour.