Observability and Subjective Performance Measurement

Performance Measurement

Management accounting research has investigated how performance measurement properties, such as informativeness, influence their use in rewarding performance, as predicted by economic theory (Ittner and Larcker, 1998a, 2009). Informative performance measures are sensitive to a manager's actions, precise (low noise), and provide incremental information about a manager's actions (Ittner et al., 2003). For a measure to be informative, it must be relevant and faithfully represented. To satisfy the requirement of faithful representation, a measure needs to be verifiable by an independent party (e.g., an auditor) (Moers, 2006; Rajan and Reichelstein, 2009). Verifiability in agency literature has been used to describe whether a signal can be contracted upon (i.e., observable by a third party, e.g., the courts). Agency theory models of aggregation suggest that the informativeness of one measure is dependent on the level of informativeness of another measure (Banker and Datar, 1989). That is, performance measures should receive more weight in an incentive contract when it is more sensitive and/or less noisy about managerial actions. Studies such as Lambert and Larcker (1987), Clinch (1991), and Sloan (1993) provide evidence consistent with this.

Empirical research to date has examined incentive schemes where the weighting of the measures in a formula-based incentive plan can vary, and superiors in organizations have discretion and incentives to bias the performance evaluation (Prendergast and Topel, 1993; Moers, 2005). For example, Moers (2005) examined an incentive plan in which the weights of the constituent measures were allowed to vary and found that the use of multiple objective performance measures and the use of subjective performance measures are related to more compressed and lenient performance ratings that determine annual bonuses. While the setting allows for discretion with respect to the weights of the performance measures, most analytical studies assume that formula-based contracts have zero discretion (Baiman and Rajan, 1995) or that the principal with discretion will not renege because of concerns about reputation (Baker et al., 1994). We consider the case in which the formula-based contract has zero discretion (with respect to the weights of the financial and nonfinancial measures) and examine how the subjective measure is influenced by the informativeness of the other measures in the contract.

Observability

Studies have shown that geographical proximity has economic benefits for monitoring subunits' or subordinates' performance (Lerner, 1995). Closer proximity allows one to acquire superior information about an object that is of interest (e.g., Coval and Moskowitz, 1999, 2001; Ivkovic and Weisbenner, 2005, Bernile et al., 2010) and lowers the monitoring costs (Chhaochharia et al., 2012). There is also a growing body of research in home-bias finance literature that predicts and finds that geographical distance influences investors' decisions and judgements about investees, primarily because of information advantages associated with shorter geographical distances (e.g., visits to the investment). Holding the quality of investments constant, equity investors are more likely to decide to invest in local stocks versus overseas stocks (Kang and Stulz, 1997), and fund managers and individual investors are more likely to decide to invest in firms with headquarters closer to them (Coval and Moskowitz, 1999; Ivkovic and Weisbenner, 2005). The accuracy of forecasts about the earnings of financial analysts is also higher for geographically proximate firms (Malloy, 2005).

Relating to loan officers' loan decisions, Petersen and Rajan (2002) find that loan officers make greater use of objective information when assessing the loans for small business that are more geographically distant. Liberti and Mian (2009) find evidence that is consistent with the theory that greater hierarchical distance between a subordinate and his boss makes it more difficult to incorporate abstract and subjective information in decision-making. They find that higher frequency of interactions between a loan officer and his boss, both over time and through geographical proximity, helps to mitigate the effects of hierarchical distance on information use.

Based on psychometric theory, Wherry and Bartlett (1982) predict that the greater the geographic distance between the evaluator and the evaluatee's workplaces, the lower the probability of multiple and/or long direct experiences. This reduces the evaluator's opportunity to learn about the evaluatee's behaviour by observing or interacting with the evaluatee and thus lowers the probability of deriving an accurate rating. As the number or length of informative experiences decreases, evaluators increasingly rely on other cognitively accessible information to derive their ratings. Expressed in the psychology of decision-making terms (Tversky and Kahneman, 1974), observation enables the development of cognitive accessibility (perceived reliability) of an anchor, which in turn affects the rating decision (Mussweiler and Strack, 2000).

Our review of the performance measurement literature and the effects of direct observation on performance evaluation indicate the following: the effectiveness of the performance evaluation decision depends on the informativeness of the performance measures. The information being assessed during the performance evaluation decision can be about a person or a unit's performance as indicated by the performance measures, or the perceived reliability of information being produced by the performance measures. As with financial statement audits, direct observation enables the evaluator to assess the reliability of nonfinancial performance measures through observing new sources, which in turn increases the perceived reliability of the measure. In the next subsection we develop the arguments for how direct observation of the subordinate moderates the association between objective and subjective performance measures.

Hypotheses Development

Based on the evidence from prior research, we consider a model in which the objective function is the subjective measurement of a branch office manager's integration behaviour (integration). Following Banker and Datar (1989) we predict that the subjective measurement will be a function of three signals: (i) observation of the branch offices' integration behaviour (observation), (ii) financial performance measures (financial), and (iii) customer satisfaction (nonfinancial), and their interaction: (i) the interaction of observation and financial, and (ii) the interaction of observation and nonfinancial performance measures (see Model 1).

(1)

Since the integration measure was included in the incentive system for the purposes of capturing managerial behaviour that is not directly captured by the financial or nonfinancial measurements, both the financial and nonfinancial measures are not expected to be associated with the level of the subjective performance evaluation (integration). To the extent that such a relationship exists, given the bounded rationality of the home office manager, we argue that such relationships will depend on the level of the evaluating manager's direct observation of the branch office manager.

Financial Measures

The influence of performance measures on the performance evaluation decision depends on the informativeness of the performance measures, which can depend partly on the amount of direct observation between the evaluator and the evaluatee (branch office manager). While financial measures are more reliable, they are less likely to be as informative as the nonfinancial measure due to the latter's lead indicator properties (Banker et al., 2000) or because they are less value-relevant (Amir and Lev, 1996). This is particularly likely to be the case in the subjective evaluation context when the evaluating manager's direct observation of the branch manager is high. At high levels of observability, the relative weight of the financial measure signal will be a weaker cue for the subjective evaluation of branch manager performance because of the information gained through direct observation of the branch manager's integration behaviour. That is, compared with the information gained from direct observation, financial measures have less value relevance (Amir and Lev, 1996).

However, as the level of observability of integration behaviour by the home office manager decreases, the associated signal strength of the direct observation decreases and the relative weight of the financial measure signal is expected to increase. Support for this argument comes from two literature streams that have considered observability in the decision-making context. First, in the finance literature, Petersen and Rajan (2002) find that loan officers make greater use of objective information when assessing loans for small business that are more geographically distant. Second, from a psychometrics perspective, Wherry and Bartlett (1982) argue that evaluators increasingly rely on other cognitively accessible information (e.g., objective information) to derive their ratings, as the number or length of informative experiences decreases. In such situations, the other information is likely to include information about the evaluatee that is impounded in cognitively accessible financial performance measures, where the accounting system-based verifiability is independent of direct observation.

Nonfinancial Measures (Customer Satisfaction)

Compared to financial measures, nonfinancial measures are more likely to have lead indicator properties (Banker et al., 2000) and be value-relevant (Amir and Lev, 1996). However, they are less reliable, and they are not verifiable by a system such as accounting and auditors. Therefore, the salience of nonfinancial measures depends in part on direct observation. For example, contact with the object of the nonfinancial measure (e.g., customers) is important because they provide a verifiable basis for the evaluator to assess the reliability of the nonfinancial measure.

Prior studies have found that customers are unlikely to be able to express their firm's level of satisfaction accurately because satisfaction is multidimensional and their response modes in customer satisfaction survey may not capture all aspects of their firm's satisfaction (Hemmer, 1996). Similarly, in the loan officer decision-making setting, Petersen and Rajan (2002) demonstrate that being close to one's customers is likely to facilitate a loan officer's collection of soft information, such as that gathered through face-to-face interaction with a customer. Therefore, as customer visits increase, the principal has greater opportunity to learn about customer satisfaction from an independent source, such that the reliability of the nonfinancial performance measure can be verified. For example, the home office manager can verify the measure by his direct observation of interactions with customers by comparing it to what customers say about their satisfaction levels.

While the economic informativeness of the measure remains constant (i.e., the measure would be expected to rate the same customer satisfaction for both close and distant branch offices), the perceived reliability of the measure is expected to be higher for more observed branch offices, that is, the customer satisfaction measure is perceived to be unreliable until the home office manager assesses it to be reliable, in which case its perceived reliability now depends on its cognitive accessibility through direct observation. Therefore, we predict that the perceived reliability of the nonfinancial measure will be influenced by the measure's cognitive accessibility (the economic informativeness remains constant) as the home office manager's direct observation of customers increases.

The Incentive Plan

TTE has a formula-based performance evaluation plan that provides annual bonuses to branch managers based on the three branch financial performance measures (sales, expenses, accounts receivable) and two branch nonfinancial measures (customer satisfaction and the branch manager's integration behaviour). The plan comprises a formula in which the Yuan bonus is based on (1) a bonus for sales and (2) a bonus of 4000 Yuan + ((points – 60) * 100 Yuan) with a total of 100 points possible, distributed as 40 for accounts receivable, 5 for expenses, 30 for customer satisfaction, and 25 for integration behaviour. The annual bonus is given to each branch manager and the bonuses range from 60% to 90% of their total annual compensation.5

The financial measures are not adjusted for the effects of uncontrollable variables, but the integration behaviour measure can be adjusted. The weights on the financial and nonfinancial measures used to determine the annual bonuses are fixed. The incentive weights on the performance measures are fixed before the start of each year. The home office manager can neither change the weights, nor directly influence any performance measure except the integration behaviour measure, because the financial measures are from TTE's accounting system and branch customers measure their own satisfaction and send these measurements directly to TTE's home office.

When the home office management designed the performance measurement plan, it considered each of the five branch performance measures to measure different aspects of a branch and the manager's behaviour. In other words, the other performance measures were not considered informative about the branch managers' integration behaviour (see Appendix A).6

The Subjective Performance Measurement Decision

Interviews with home office managers (CEO, CFO-VP finance, VP marketing, and home office manager) indicate that their intention was for the home office manager to make accurate measurements of integration behaviour. These home office managers told us that the branch managers' integration behaviour is important for achieving TTE's strategy and financial targets; they consider it to be a leading indicator of financial results. Thus, they want the home office manager's measurements to be accurate, in order to help them assess how well they are achieving TTE's strategy. They do not want the home office manager to use his subjective measurements to game the performance evaluation plan (e.g., by assigning high levels of integration behaviour to branch managers he likes). The CFO-VP finance told us that he was not aware of any gaming of the subjective measurement of integration behaviour. Indeed, the home office manager did not have personal relations with any of the branch managers, in part because he was new to his position and also because of the geographical distance separating him from most branch managers, whom he visits only twice (on average) per year.

The interviews also indicate that the home office manager uses a subjective approach when measuring integration behaviour. He uses his memory and refers to performance measures at the end of the evaluation period to make his subjective measurements: for example, in an interview he said that, ‘I will not deliberately go to find the sales numbers for reference. Rather, I use my memory. For branches that have a good sales record, I remember much clearer. I won't clearly categorize them by numbers, but I get a sense of approximately which category each manager should belong to’ (Appendix B).

Observability—The Branch Visits

The home office manager visits the branch offices to inspect records, interact with the branch manager and branch employees, and meet with customers. ‘I visit every branch office, conduct face-to-face interviews, and talk with the branch employees and branch customers. This is very important in the measurement’ (Appendix B). The visits usually last one or two days and are preferably unannounced. Additional opportunities to gather information about the branch managers' integration behaviour can occur when a branch manager visits the home office to attend quarterly meetings.

The majority of initial visits occurred at the beginning of the year (i.e., right after Chinese New Year). The frequency of visits to branches depends on the distance between a branch and the home office, the performance of the branch, and the importance of the branch's customers to TTE. Additional visits were made to secure large sales transactions with specific customers, which necessitated the presence of the home office manager. Poor performance necessitated visits on only a few occasions and visits were not made for the express reason of acting on complaints received from customers.

During the branch visits, the home office manager is able to meet and directly observe the branch's customers. Direct observation of customers not only allows the home office manager to verify the customer satisfaction measure, it also provides him with concrete and proximate information about customers and their satisfaction. For example, in an interview he stated that the way in which customers talk about the branch manager they deal with is informative about the manager's integration behaviour. When customers are not very satisfied (according to the customer satisfaction measure), they tend to say very little about a branch manager, leading the home office manager to develop the belief that the manager has a poor relationship with the customers and is not providing good customer service. In contrast, more satisfied customers talk more about the branch manager and how he helps their business to be successful. Consistently, the home office manager stated in an interview, ‘In terms of the customer satisfaction measure, I have a higher confidence level for those results which came from customers whom I visited before. The customer satisfaction results influence my subjective measurement of the branch managers’ (Appendix B). Overall the branch visits and customer interactions are potentially informative about the integration behaviour performance of the branch manager.

Measurement of Variables

The data consist of annual data for five years (1998–2000 and 2002–2003) for nine (1998), 10 (1999), 17 (2000), and 26 (2002 and 2003) TTE branch offices. The data came from TTE's accounting and performance management system. From 1998 to 2003 data on direct observation (i.e., distance) were available, and in 2002 and 2003 data on direct observation (i.e., visits) were available. We tested the hypotheses with both samples, substituting distance and visits as proxies for direct observation.

Empirical Model

We used a pooled OLS regression model with year fixed effects to test the hypotheses.12 As the data cover several years with repeated observations that are associated with the same branch (in most cases the same branch manager), we included fixed effects for year (YEARDUM). That is, for the 1998 to 2003 data, four YEARDUM control variables are used (1998, 1999, 2000, and 2002). For the 2002 to 2003 data only one YEARDUM control variable is used (i.e., 0 for 2002 and 1 for 2003). For the 1998 to 2003 data model we included a manager fixed effect (MANAGER) as there was a change of sales manager during 2001. The least squares dummy variable (or fixed effect) estimate controls for the unobserved (time-invariant) heterogeneity, but it also yields biased coefficient estimates.

We tested the hypotheses with the following regression model:

where

INTEGRATION_it

= branch manager i's integration behaviour in period t;

SALES_it (S)

= log of branch i's sales in millions of Yuan in period t;

EXPENSE_it (E)

= log of branch i's total direct operating expenses in millions of Yuan in period t;

ACCREC_it (A)

 = branch end-of-period total accounts receivable divided by mean credit sales per day in period t;

CUSSAT_it (C)

 = branch customer satisfaction in period t;

OBSERV-DIST_it (data available 1998–2003) 

= negative distance between branch office i and home offices in average commercial flight time in hours. Raw values are used for additive effects and 0–1 indicator values are used for interaction effects with 0(1) for branch offices equal to or greater (less) than 1.55 hours;

OBSERV-VISIT_it (data available 2002–2003)

 = number of visits to each branch office with raw values for additive effects, and 0–1 indicator values for interaction effects with 0 for branch offices with less than two visits (low direct observation) and 1 for branch offices with two or more visits (high direct observation) in period t;

INTEGRATION_it−1

 = branch manager i's integration behaviour in period t − 1;

TENURE_it

 = months that branch manager i has been employed by TTE in period t;

TURNOVER_it

 = indicator variable for turnover of branch manager i, with 0 for a year in which branch i has no turnover and 1 for the year in which branch i has turnover of its manager and for all subsequent years for branch i;

SIZE_it

 = number of employees working at branch i at the beginning of period t in period t;

PROVGWTH_it

 = annual percentage growth in sales of telecommunications terminal equipment in the province where branch i is located in period t; and

MANAGER

 = indicator variable for the change of home office manager, with 0 for 1998–2000, and 1 for 2002–2003.

Given the multiple period and branch nature of the panel data, Huber-White robust standard errors with the cluster command were used to control for sample dependence for each branch. Whilst there were no influential observations for the 1998–2003 sample (n = 87), for the 2002–2003 sample (n = 52, 26 branch offices x two years), six influential observations were excluded that were identified based on DFITS and Cook's D-statistic, leaving a sample size of 46.13

Descriptive Statistics

Table 1 presents the variables' descriptive statistics. The variables have considerable realized variance, as assessed by their standard deviations, and ranges relative to their means. For example, INTEGRATION_t ranges from 5.95 to 21.19 and CUSSAT ranges from 28.85 to 59.60, both more than two-thirds of their respective theoretical ranges. The financial measures also have large realized ranges: SALES ranges from 0.01 to 66.86 million Yuan, EXPENSE ranges from 0.01 to 3.39 million Yuan, ACCREC ranges from 17.61 to 331.67, and OBSERV-VISIT ranges from 0 to 9 visits. All variables are normally distributed.

Table 2 presents Pearson coefficients for the variables. INTEGRATION_t has significant (p < .10) positive correlations with ACCREC, CUSSAT, TURNOVER (p < 0.05), and SIZE. Six correlations between independent variables are significant at p < .10. In particular, SALES is significantly correlated with EXPENSE, ACCREC, and CUSTOMER. EXPENSE is correlated with OBSERV-DIST and OBSERV-VISIT (p < .05). The correlation between OBSERV-DIST and OBSERV-VISIT is –0.39 (p < .01) indicating that the OBSERV-DIST variable is a reasonable proxy for direct observation in terms of the number of visits to a branch by the home office manager. Among the control variables, the correlation between SIZE and each of TURNOVER (p < .01) and MANAGER (p < .05) is significant. The correlation between MANAGER and each of TENURE (p < .05) and TURNOVER (p < .05) is significant. The correlation between SIZE and INTEGRATION_t−1 (p < .05) is significant.14 The lack of highly significant correlations reduces the probability of a significant multicollinearity problem.

Hypothesis Tests

The two regression models in Table 3 include the independent and control variables as well as the interaction terms. The adjusted R²s range from .12 (p < .05) and .33 (p < .01) and are significantly (p < .05) higher than the adjusted R′²s of the main-effects models, which includes the independent and control variables, but not the interaction terms. Multicollinearity is not a problem, as evidenced by the largest VIF of 6.15 and the largest condition index value of 33.47, which are within the accepted limits (Belsley, 1991). The residuals of the model are normally distributed.

Main Effects Hypotheses—Financial and Nonfinancial Performance Measures

H1 predicts that the level of the integration behaviour performance measure will be positively related to the level of the sales measure and negatively related to the levels of the expense and accounts receivable measures, and that the absolute magnitudes of these bivariate relations will decrease with the home office manager's direct observation of a branch manager.

For the sales measure, the significant (p < .05) negative betas for the SALES by OBSERV-DIST (p < .01; Model 1) and SALES by OBSERV-VISIT (p < .05; Model 2) interactions provides support for H1. For the expense measure, the significant (p < .01) positive sign of the beta for the EXPENSE by OBSREV-DIST interaction (Model 1) provides support for H1. However, the beta for the EXPENSE by OBSERV-VISIT is not significant (p > .10), thus providing mixed support for H1. Finally, for the accounts receivable measure, the beta for ACCREC by OBSREV-DIST (Model 1) and ACCREC by OBSREV-VISIT (Model 2) interactions are not significant (p = 0.50 and .34) for each of the interaction models, which does not provide support for H1.

H2 predicts that the positive influence of the level of the customer satisfaction measure on the level of the integration behaviour measure will increase as direct observation increases. The result of significant positive betas for the CUSSAT by OBSERV-DIST (p < .05; Model 1) and CUSSAT by OBSERV-VISIT (p < 0.01; Model 2) interactions provide support for H2.

Sensitivity Tests

The sensitivity of the results was tested in several ways. First, the results were qualitatively the same whether none, all, or any one of the six control variables was included. Second, as observability was expected to be related to both the reliability of the customer satisfaction measure and the level of integration behaviour, this raised the potential for a correlated omitted variable problem, which will bias the results in favour of a larger coefficient on the customer satisfaction measure for closer branch offices. When we re-ran the models with a measure of customer satisfaction that was orthogonal to the level of observability (e.g., the residual from a model in which customer satisfaction was regressed separately on each of the three observability proxies in their continuous form), the significance of all of the hypothesized estimates was qualitatively unchanged. Third, since the data are comprised of the same branches for multiple years, two observations have the median value (1.55 hours). The result of testing the hypothesis is qualitatively unchanged when these two observations are excluded, included in the low OBSERV level (as reported in Table 3), or included in the high OBSERV level.

Sensitivity Tests for the Effect of Branch Manager Experience and Home Office Manager Knowledge

We controlled for the tenure of the branch manager as this could affect the level of experience and/or knowledge that the home office manager had of him. It is possible that the level of experience of the branch manager might affect the subjective performance measurement rating. For example, less experienced managers might be subject to more frequent visits or be located closer to the head office. The home office manager might want to locate less experienced managers at branches nearer to the home office to observe and monitor their integration behaviour. We found that the correlation between TENURE and DISTANCE was –.09 and not significant (p = 0.42).