Hamilton's rule, inclusive fitness maximization, and the goal of individual behaviour in symmetric two-player games

Additive Prisoner's dilemma

Consider a simple model of the evolution of social behaviour of the sort used in evolutionary game theory. An infinite population of haploid asexual organisms engage in pairwise social interactions in every generation. Organisms are of two types, altruists (A) and selfish (S). A types perform an action that is costly for themselves but benefits their partner; S types do not perform the action. Type is hard-wired genetically and perfectly inherited.

An organism's pay-off from the social interaction depends on its own type and its partner's type. Pay-offs are interpreted as increases in lifetime reproductive fitness over a unit baseline. The social action is assumed to affect only the actor and their partner, and thus local interaction is assumed absent. An A type incurs a cost of −c as a result of its action and confers a benefit of b on its partner, where c > 0 and b > 0; thus, the game is a Prisoner's dilemma.

Pay-offs to the actor, referred to as ‘personal pay-offs’, are shown in Table 1. We let V(i, j) denote the pay-off to an actor from playing i when her opponent plays j, where i, j ∈ {A, S}. Note that pay-offs are additive: an altruist alters their own pay-off by −c and their partners’ pay-off by b, irrespective of the type of their partner.

There are three pair-types in the population, AA, AS and SS, whose relative frequencies in the initial generation are f_AA, f_AS and f_SS, respectively, where f_AA + f_AS + f_SS = 1. The overall frequency of the A type in the initial generation is denoted p, where . The change in p over one generation is denoted Δp.

The sign and magnitude of Δp depend on the rules by which the pairs are formed. If pairing is random, then the S type must be fitter overall, so Δp will be negative. However, if pairing is assortative, then the A type may be fitter overall, for the benefits of altruistic actions then fall disproportionately on other altruists. Random pairing means that the frequency distribution of the pair-types will be binomial, that is f_AA = p², f_AS = 2p(1 − p) and f_SS = (1 − p)².

Where pairing is nonrandom, a simple regression analysis yields a measure of the statistical correlation between social partners. We use the variable p_i to indicate an organism's own type and to indicate its partner's type; thus p_i = 1 if the ith organism is an A, p_i = 0 otherwise; and if the ith organism is paired with an A, otherwise. We then compute the linear regression of on p_i, given by , which is a standard way of defining the r term of Hamilton's rule. Henceforth, we refer to as r.

In the early kin selection literature, r was often defined in genealogical terms, for example as the probability that actor and partner share an allele that is identical by descent, yielding the familiar values of for full sibs, for offspring and for grandoffspring (see Michod & Hamilton, 1980). In some ways, this definition of r is the more natural one for expressing the idea that organisms value their relatives’ reproduction in proportion to how closely related they are. However, the statistical definition of r, above, yields a version of Hamilton's rule that is more generally applicable.

In the context of pairwise interactions, r can be conveniently expressed as a difference in conditional probabilities:

It follows that r ranges from −1 (perfect disassortment) to 1 (perfect assortment); when pairing is random, r = 0.

Evolutionary analysis

In the Appendix, we show that the change in p over one generation is given by:

(1)

where is average population fitness. As Var(p) is non-negative, this tells us that so long as 0 < p < 1, the A type will increase in frequency in the population whenever rb > c, which is of course Hamilton's rule. As b and c are fixed parameters of the pay-off matrix, this condition for the spread of A is frequency-independent so long as r itself does not change as the population evolves. Constancy of r across generations will sometimes be a reasonable assumption, for the pattern of assortment in the population, which is what r measures, may be determined by biological factors, for example dispersal, which are independent of the social trait that is evolving. This assumption is considered further in the Discussion section.

With the constant r assumption, the outcome of the evolutionary process is easily determined. If rb > c the A type will spread to fixation; if rb < c the S type will spread to fixation; if rb = c there will be no evolutionary change.

Rational actor analysis: preliminaries

To apply a rational actor analysis, we transpose our evolutionary model to a rational choice context. We consider two players playing a symmetric game. Each player has two pure strategies, A and S. If a player plays a mixed strategy, this means that they randomize over their pure strategies; thus, π_A denotes the mixed strategy in which A is played with probability π_A and S with probability 1 − π_A. The pay-off to a mixed strategy is then simply its expected pay-off.

Each player has a utility function which measures how desirable they find the possible outcomes of the game; we assume that both players have the same utility function. Each player's goal is to maximize their utility function. One possibility is that the utility function is given by the personal pay-offs in Table 1 above, in which case we write U(i, j) = V(i, j), where U(i, j) is the utility a player gets from playing i when their partner plays j; i, j ∈ {A, S}. There are other possibilities too; see below.

Once the players’ utility function has been specified, the next step is to seek the Nash equilibrium (or equilibria) of the game. (A Nash equilibrium is a pair of strategies, possibly mixed, one for each player, each of which is a best response to the other.) Game theory predicts that if the players are rational, they will end up at a Nash equilibrium of the game (see for example Binmore, 2007). We can then ask whether the Nash equilibria of the game correspond to the outcomes of the evolutionary process described above. If so, we can conclude that evolution will lead organisms to behave as if trying to maximize the utility function in question.

This is a natural way of formalizing the rational actor heuristic in a game-theoretic context. It differs somewhat from Grafen's (2006) formalization of the same idea, which posits ‘links’ between gene-frequency change and individual optimization. Our approach allows recovery of Grafen's main result, and by taking optimization to include best-response, that is optimal choice conditional on the other player's choice, extends easily to the nonadditive case. A similar approach is found in Alger & Weibull (2012) and Lehmann et al. (2015).

Utility as inclusive fitness

One possibility to explore is that a player's utility function depends on their partner's pay-off as well as their own. For example, suppose that a player's utility for any outcome is given by the quantity: personal pay-off plus r times partner's pay-off, that is U(i, j) = V(i, j) + rV(j, i). Applying this transformation to the personal pay-offs yields Table 2 below, which we refer to as the ‘inclusive fitness pay-off matrix’.

This transformation was first suggested by Hamilton (1971) and has been discussed by Grafen (1979), Bergstrom (1995), Wade & Breden (1980), Day & Taylor (1998), Taylor & Nowak (2007) and Martens (2015). It is a natural formalization of the idea that an actor, in their social behaviour, will care about their partner's pay-off, discounted by relatedness, as well as their own pay-off. Clearly, other transformations of the personal pay-off matrix are also conceivable.

The pay-offs in Table 2 do not correspond exactly to the verbal definition of inclusive fitness in Hamilton (1964), which was ‘the personal fitness which an individual actually expresses…once it is stripped of all components which can be considered as due to the individual's social environment…then augmented by certain fractions of the quantities of harm and benefit which the individual himself causes to the fitnesses of his neighbours…The fractions in question are simply the coefficients of relationship’ (p. 8). This definition is sometimes but not always adhered in the literature.

The discrepancy between Table 2 and Hamilton's definition arises because in the left column, the actor's pay-off has not been stripped of the component that is due to the partner's altruistic action (b) and has been augmented by r times the partner's entire pay-off, rather than the portion of that pay-off that is caused by the actor (b). Applying Hamilton's definition exactly would lead to the pay-off matrix in Table 3 below.

Note that Table 3 derives from Table 2 by subtraction of the quantity b − rc from the left column. In game-theoretic terms, Table 3 is thus a ‘local shift’ of Table 2 (and vice versa), which means that their Nash equilibria are necessarily identical (Weibull, 1995). Therefore, if the players’ utility function is given by Table 3, game theory predicts exactly the same outcome(s) as if it were given by Table 2. So although taking Table 2 as the definition of inclusive fitness involves an element of ‘double counting’ – which Hamilton's definition was designed to avoid – it is harmless.

In fact, there is a positive reason to prefer Table 2 as the definition of inclusive fitness, in a game-theoretic context, for Hamilton's definition does not generalize easily to nonadditive pay-offs. With nonadditivity, it is unclear how to decide which component of the actor's pay-off is ‘caused’ by its partner's action and vice versa (cf. Allen et al., 2013). By contrast, the definition used in Table 2 – actor pay-off plus r times partner pay-off – applies just as well to the nonadditive case. In order not to prejudge the issue of whether inclusive fitness maximization, or a similar result, obtains under nonadditivity, this is the definition preferred here.

Rational actor analysis: results

Suppose firstly that the utility function is personal pay-off (Table 1). It is easy to see that (S, S) is the only Nash equilibrium of the game, as S strongly dominates A, that is each player does strictly better by playing S irrespective of their partner's choice. This familiar result shows that the rational actor heuristic fails for this choice of utility function, as it would have us conclude that altruism can never evolve, which we know to be false.

What if the utility function is inclusive fitness pay-off (Table 2)? In that case, we can show the following. If rb > c, then (A, A) is the unique Nash equilibrium; if rb < c, then (S, S) is the unique Nash equilibrium; if rb = c, then (A, A) and (S, S) are both Nash equilibria, as is every pair of mixed strategies, so game theory makes no prediction about the players’ choices (See Appendix for proof).

It follows that with additive pay-offs, defining utility as inclusive fitness makes the rational actor heuristic valid. The condition for the A type to evolve, rb > c, is identical to the condition for (A, A) to be the unique Nash equilibrium of the rational game, and similarly for S (Table 4). This supports the idea that evolution will lead organisms to appear as if trying to maximize their inclusive fitness, just as Hamilton originally argued.

An equivalent perspective on the situation is this. The quantity (rb − c) equals the difference in a player's inclusive fitness pay-off between playing A and S, irrespective of what its partner does (see Table 2). Thus, we can determine whether the A type will evolve by asking whether a rational agent, who wants to maximize their inclusive fitness, would choose A over S. In short, equating utility with inclusive fitness ensures that the rational agent's choice coincides with the ‘choice’ made by natural selection.

A caveat: uniqueness

One important caveat is needed. In the above model, the inclusive fitness pay-off matrix (Table 2) is not the unique utility function that yields the rb > c condition for action A to be chosen over S. In game theory, the utility function is only ever unique up to choice of origin and unit; so any affine transformation (of the form U^′ = aU + b, where , a > 0) will leave all Nash equilibria of the game unchanged. Furthermore, a ‘local shift’ of the utility function, which involves adding a constant to any column of the utility matrix, will also leave unchanged the Nash equilbiria, as noted above.

One local shift of the inclusive fitness pay-off matrix (Table 2) is of particular interest. If we add the quantity (rc − rb) to the left-hand column of Table 2, we get the matrix in Table 5 below.

The pay-offs in Table 5 are related to the personal pay-offs (Table 1) by the transformation U(i, j) = rV(i, i) + (1 − r)V(i, j). This transformation was first suggested by Grafen (1979), hence the label ‘Grafen 1979 pay-off’; see Bergstrom (1995), Day & Taylor (1998), Alger & Weibull (2012) for discussion. By contrast with the inclusive fitness pay-offs (Table 2), which involve adding r times partner's pay-off to the actor's personal pay-off, the Grafen 1979 pay-offs involve taking an (r, 1 − r) weighted average of the personal pay-off that would accrue to the actor if their partner had chosen the same as the actor and if their partner made the choice that they actually did.

As the Grafen 1979 pay-off matrix (Table 5) is a local shift of the inclusive fitness pay-off matrix (Table 2), the Nash equilibria of the resulting games are identical; thus, the rational actor heuristic works equally well with either. [This is because in both cases, (rb − c) is the pay-off difference between playing A and S.] Therefore, although our simple model has vindicated Hamilton's claim that evolution will lead organisms to behave as if trying to maximize their inclusive fitness, it is important to see that inclusive fitness [whether defined our way (Table 2) or in Hamilton's original way (Table 3)] is not the unique quantity of which this maximization claim is true.

Evolutionary analysis

As before, Δp denotes the change in frequency of the A type over a generation. Unsurprisingly, rb > c is no longer the condition for Δp to be positive. However, an exact version of Hamilton's rule can be recovered by suitably defining the cost and benefit terms, as emphasized by Gardner et al. (2011), whose approach we follow here. (A different approach, not discussed here, incorporates nonadditive pay-offs into Hamilton's rule by a weak selection approximation, see for example Lehmann & Rousset, 2014b).

For each individual i, we let w_i denote its actual reproductive fitness (number of offspring). We then write w_i as a linear regression on p_i and :

(2)

where α is baseline fitness; is the partial regression of an individual's fitness on their own type, controlling for their partner's type; is the partial regression of an individual's fitness on their partner's type, controlling for their own type; and e_i is the residual. These partial regression coefficients quantify the average effect (sensu Fisher, 1930) of the actor's action, and their partner's action's, on the actor's fitness.

Following Hamilton (1964), instead of considering the effect on the actor's fitness of their partner's action , we can consider the effect on their partner's fitness of the actor's action, denoted . These two partial regression coefficients are numerically dentical (Taylor et al., 2007). (This is the well-known switch from ‘neighbour-modulated’ to ‘inclusive’ fitness.) Following Gardner et al. (2011), we denote the and coefficients as −C and B, respectively.

Importantly, eqn 2 can be fitted whether or not the true relation between w, p_i and is linear. In the nonadditive case under consideration that relation is nonlinear (since d > 0), which implies that the partial regression coefficients −C and B will be functions of population-wide gene frequencies, and liable to change as the population evolves. Therefore, unlike c and b, which are fixed pay-off parameters, −C and B are population variables.

Following Gardner et al. (2007, p. 219), we can write explicit expressions for −C and B in terms of r, p, and the parameters of the pay-off matrix b, c and d. This yields:

(3)

(4)

We can then derive the following expression for evolutionary change:

(5)

where is average population fitness (see Appendix). Equation 5 tells us that when 0 < p < 1, the A type will increase in frequency if and only if rB > C. This is a generalized version of Hamilton's rule, applicable whether pay-offs are additive or not.

The quantity (rB − C), whose sign determines whether altruism spreads, can be computed by adding eqn 3 to r times eqn 4. After simplifying, this yields:

(6)

Note that (rB − C) is a function of p, so satisfaction of rB > C in generation t does not imply its satisfaction in generation t + 1. Selection is thus frequency-dependent, and neither type will necessarily spread to fixation. A polymorphic equilibrium will obtain when p = [c − r(b + d)]/d[1 − r]; the stability of this equilibrium depends on the sign of d. The full evolutionary dynamics are summarized in Table 7 below, see Appendix for proof.

The general version of Hamilton's rule embodied in eqn 5 raises interesting interpretive questions. Some have argued that the rule in this form has little explanatory value (Nowak et al., 2011; Allen et al., 2013), whereas others have seen the generality of the rule as an advantage, a proof that inclusive fitness theory does not rely on restrictive assumptions (Gardner et al., 2011). This debate has been analysed elsewhere (Birch, 2014; Birch & Okasha, 2015) and is not the focus here.

Instead, our question is this. Given that eqn 5 is true, and given the resulting evolutionary dynamics, can the rational actor heuristic be applied? Will evolution lead organisms to behave as if maximizing a utility function, and if so what is it?

Importantly, the answer to this question cannot simply be read off eqn 5. In the additive case, there was a simple link between Hamilton's rule and a utility function with the desired property: rb − c > 0 was the condition for the A type to spread, and (rb − c) the utility difference between playing A and S. One might hope to extrapolate this to the nonadditive case by simply replacing (rb − c) with (rB − C) in Table 2. However, as B and C are functions of p, they cannot meaningfully feature in the utility function.

The reason is as follows. The point of the rational actor heuristic is to find a link between gene-frequency dynamics and a ‘goal’ that organisms behave as if they are trying to achieve. Such a link would be trivial if the ‘goal’ were allowed to change as gene frequencies change. For the heuristic to have any value, the goal must remain fixed. So our task is to find a utility function whose arguments are restricted to the pay-off parameters (b, c and d), and the relatedness coefficient r, which makes the rational actor heuristic work.

Rational actor analysis

To address this question, we again transpose the evolutionary model to a rational choice context and study the Nash equilibria of the resulting game. Suppose firstly that the utility function is given by the inclusive fitness pay-off transformation, that is personal pay-off plus r times partner pay-off. This yields the pay-offs in Table 8 below.

The Nash equilibria are then as follows:

It follows that, unlike in the additive case, the rational actor heuristic does not work when utility is defined as inclusive fitness. The condition for (A, A) to be a Nash equilibrium is not identical to the condition for A to evolve to fixation, similarly for S. Furthermore, the condition for there to be a mixed-strategy Nash equilibrium is not the same as the condition for there to be a polymorphism. So it is not true that at evolutionary equilibrium, organisms will behave as if trying to maximize their inclusive fitness.

Can we find a utility function modulo which the rational actor heuristic works? The answer is yes. The Grafen 1979 pay-off matrix, which to recall is derived from the personal pay-off matrix by the transformation U(i, j) = rV(i, i) + (1 − r)V(i, j), does the trick. This yields the pay-offs in Table 9 below.

The Nash equilibria are then as follows:

This restores the rational actor heuristic. In particular, if (A, A) is the only pure-strategy Nash equilibrium, then A evolves to fixation; if (S, S) is the only pure-strategy equilibrium, then S evolves to fixation. If there is a mixed-strategy Nash equilibrium but no pure-strategy equilibria, the population evolves to a stable polymorphism; if there is a mixed-strategy Nash equilibrium and both (A, A) and (S, S) are pure-strategy equilibria, then there is an unstable polymorphism; in both cases, the weights on A and S in the mixed-strategy Nash equilibrium equal the proportions of A and S in the polymorphism. Thus, there is a tight correspondence between the Nash equilibria and the evolutionary dynamics, summarized in Table 10 (see Appendix for proof).

The upshot is that with nonadditive pay-offs, the rational actor heuristic will work so long as the utility function is defined as Grafen 1979 pay-off, rather than inclusive fitness pay-off. Again any affine transformation of the Grafen 1979 pay-off matrix, or any local shift, will also preserve the correspondences above. Note that, unlike in the additive case, the Grafen 1979 pay-off matrix (Table 9) is not a local shift of the inclusive fitness pay-off matrix (Table 8). This is why the rational actor heuristic fails if utility is defined as inclusive fitness in the nonadditive case.