Computer-automated bird detection and counts in high-resolution aerial images: a review

Automated techniques for detecting and counting birds

Examples of aerial images collected using manned and unmanned aircraft containing birds detected using computer-automated approaches are presented in Figure 1. The most straightforward way to analyze digital images is by the spectral values of their individual pixels. For automated bird detection and counting, this works best if the birds contrast strongly with the image background. If so, they can be readily isolated in the image using spectral thresholding, i.e., creating a binary image of pixels above and below a specified spectral threshold. Image-analysis software can then provide an instant tally of discrete segments (single isolated pixels or clusters of pixels) above or below the threshold. The first demonstration of this technique was with images of white Snow and Ross's (Chen rossii) geese in their southern wintering grounds (Gilmer et al. 1988, Table 1), although it can also be applied to dark birds on pale backgrounds. A variation of thresholding, sometimes called level slicing, involves setting an interval of spectral values within which pixels are isolated that can be manipulated in software using a sliding bar that allows users to set minimum and maximum values (Cunningham et al. 1996, Laliberte and Ripple 2003, Table 1). Thresholding can be performed on a grayscale version of an image, on any of the bands of a color image (e.g., red, blue, or green for standard true-color images), or on the combined color bands (based on hue, saturation, and brightness). Bird counts using spectral thresholding alone can be remarkably accurate in situations with high bird contrast where few to no other image elements share the same spectral range as the birds, and the birds are spatially separated (Chabot and Bird 2012) (Table 1, Fig. 1A). Standard image processing software such as Photoshop (Adobe Systems, San Jose, CA) or PaintShop Pro (Corel Corporation, Ottawa, Canada) is sufficient for this type of analysis, although more specialized software may offer greater flexibility (see below).

A common challenge arises, however, when other elements are in the same spectral range as birds. In the case of bright birds, for example, these might be pale rocks on the ground, or froth, wave crests, or glint on water. Short of manually cropping these elements out of the images, which may in some cases be accomplished reasonably fast if concentrations of birds can readily be carved out with a polygonal selection tool, a notion of “objects” beyond clusters of individual pixels must be introduced into the analysis to reduce errors. The most basic object-oriented attribute to characterize a cluster of pixels is its size in number of pixels. Bajzak and Piatt (1990) were the first to supplement spectral level slicing with a size-range filtering operation to better isolate clusters of bright pixels representing Snow Geese staging in farm fields (Table 1). As with spectral thresholding operations, automated size filtering and sorting subsequently became standard tools in basic image-analysis software applications used to count birds (Cunningham et al. 1996, Laliberte and Ripple 2003), including the ability to appropriately tally large pixel clusters composed of multiple birds bunched together (Fig. 2). In some cases, applying a smoothing (low-pass) or sharpening (high-pass) filter to the images prior to size filtering may help accentuate, or normalize, the size and/or coherence of pixel clusters representing birds that either contrast more weakly with the background or display internal spectral variation (Laliberte and Ripple 2003, Trathan 2004, Table 1).

Other basic object attributes can help to identify birds, including the shape of pixel clusters, particularly their roundness/compactness, which is a function of the ratio of perimeter to area. For example, filtering pixel clusters according to roundness has been used to isolate waterfowl on water surfaces (Cunningham et al. 1996) and Adélie Penguins (Pygoscelis adeliae) in breeding colonies (McNeill et al. 2011) from more elongate image elements of similar color and size (Table 1). For basic spectral thresholding, object size and shape filtering, automatic object tallying, and other common image processing operations such as smoothing and sharpening, one current software solution is ImageJ (National Institutes of Health, Bethesda, MD; https://imagej.nih.gov/ij/). This multiplatform freeware application was originally developed for medical purposes (e.g., microscope and radiological images), but has spawned a large community of users spanning a variety of fields, as well as numerous third-party plugins for expanded analysis options.

Filtering and sorting of pixel clusters based on object-oriented characteristics requires images of sufficiently fine resolution, with each bird comprised of multiple pixels, to be able to distinguish these characteristics. This is why automated analyses have typically been carried out with very high resolution images of ~10 cm/pixel or finer, depending on the size of the birds being analyzed (Table 1). It is also generally desirable for the angle of aerial images to be as vertical as possible so that subject size and shape remain consistent. The continuously increasing resolution of digital cameras and the ability to perform very low altitude surveys using small UAS are enabling detection of ever-smaller species, such as Dunlins (Calidris alpina) (Drever et al. 2015).

A variety of alternative computing approaches have also been developed for analyzing image spectral and object-oriented characteristics to automatically detect birds. Abd-Elrahman et al. (2005) developed a template-matching approach (Table 1) where customized algorithms were used to calculate a correlation coefficient (i.e., the degree of similarity based on overall color and size) between image elements and sample images of white wading birds (Fig. 1B), classifying all elements above a certain correlation threshold as birds. Grenzdorffer (2013) employed a traditional supervised pixel-based image classification routine (Table 1) in ArcGIS (ESRI, Redlands, CA), commonly used for land use/cover classification in remotely sensed images, to automatically detect and count Common Gulls (Larus canus) in a breeding colony in images acquired with a UAS (Fig. 1F). “Spectral signatures” of seven image classes including “bird” were first established by manually tracing representative “training samples” in the images, then all image pixels were automatically assigned to the class to which they were spectrally most similar. This approach typically involves some spectral overlap among the signatures of different classes and tends to produce results that are not as clear-cut as straightforward spectral thresholding, especially in very high resolution images (in this case, ~1.6 cm/pixel) characterized by a high degree of fine-scale spectral heterogeneity. Therefore, a series of post-classification operations was needed to refine results, including filtering out bird-classified objects that were too small or large to be part of gulls, then using a distance rule to merge close-together bird objects representing single birds that were split into multiple objects, and again filtering out remaining objects that were too small to be gulls.

Descamps et al. (2011) developed an entirely different “unsupervised” method to automatically count aggregations of birds based on their shape and spectral discreteness, using dense breeding colonies of Greater Flamingos (Phoenicopterus roseus) as test subjects (Table 1). Based on the underlying assumption that birds are ellipse-shaped in the images, they developed an algorithm that attempts to “fit” multitudes of ellipses onto the inputted image in a stochastic, iterative manner to achieve the arrangement that minimizes total “data energy.” Energy is reduced when spectral variation within individual ellipses is reduced, which occurs when they are correctly fitted onto actual birds represented by objects of local brightness maximum or minimum. The algorithm is therefore flexible to the exact size and spectral range of birds as long as they are ellipse-shaped and have sufficient contrast with the background. However, Descamps et al. (2011) suggested that this method was mainly effective for dense, localized aggregations of birds that can be manually cropped to the aggregation area prior to running the algorithm. Including areas outside an aggregation may result in erroneously counting other ellipse-shaped and spectrally discrete image elements that are not birds. The software application developed based on this algorithm, FLAMINGO, is available online for free (http://www.flamingoatlas.org/dwld_flamingo.php).

Most previous analysis methods will not work well in more complex situations involving sparsely distributed single birds, isolated flocks throughout large numbers of images, or multispecies detection. Large image sets tend to collectively contain a greater variety of background elements that could be mistaken for birds when using relatively simple detection attributes. In addition, variation in camera exposure and lighting conditions create spectral inconsistency among photos in both birds and background, even in a single aerial-survey flight. This is especially true with consumer-grade cameras, such as those usually used in UAS, that tend to have less consistent performance and spectral response than professional-grade aerial photogrammetric cameras. The methods described in many of the above-cited studies involve adjusting the spectral threshold value(s) on an image-by-image basis to account for this sort of inconsistency. This can take a few minutes per photo to determine the appropriate settings based on visual assessment of a live-updating preview image while adjusting the threshold range (Cunningham et al. 1996, Laliberte and Ripple 2003). Obviously, image-by-image threshold adjustments or other manual preprocessing operations such as cropping become increasingly inefficient as image sets grow larger.

A valuable approach for addressing the aforementioned challenges presented by large images sets is true “object-based image analysis” (OBIA). True OBIA consists of first “segmenting” images into a mosaic of adjoining objects, then classifying the objects based on sometimes elaborate rule sets. Segmentation clusters spectrally similar adjacent pixels into discrete image objects using algorithms that detect boundaries between image features and analyze the extended neighborhood of each pixel rather than just the pixel itself or its immediate surroundings. The sensitivity of the segmentation process can be controlled by adjusting the scale and merge parameters, with smaller values resulting in more total objects created; this may result in millions of objects in a single large image. Various geometric rules can be applied to segmentation as well. For example, the loosest form of segmentation, “multiresolution”, generates objects of irregular sizes and shapes, whereas “quadtree” segmentation only creates square objects of 1, 4, 16, 64, 256 (and so on) pixels. Importantly, segmentation has a recognized propensity for delineating “real” objects of interest in images, including individual birds (Fig. 3).

OBIA has been used, for example, to analyze images of birds at sea (Milton et al. 2006, Groom et al. 2007, 2013, Table 1), which typically involves large photo sets where birds are sparsely distributed over the spectrally variable water surface. After using segmentation to separate birds from swaths of water and other conspicuous surface features, rule sets based on similar attributes to those covered above, including spectral range, size, shape, and additionally spectral “texture” and object neighborhood, can be used to automatically classify birds. Rule sets can be developed manually by selecting classification attributes and directly setting their values, via machine learning by inputting training samples, or through a combination of both (Groom et al. 2007).

Another valuable aspect of OBIA is its high degree of programmability and scriptability, which has led to the development of creative and versatile image-processing workflows. For example, Groom et al. (2011) developed a routine that involves multiple “nested” segmentations followed by an iterative spectral thresholding and object classification loop (Fig. 4) to ultimately count 81,664 non-breeding Lesser Flamingos (Phoeniconaias minor) spread across water and land in a 5.25-km² area captured in 31 aerial photos (Table 1). The precise mapping of each bird's location enabled subsequent analyses to characterize flock distribution patterns in relation to ecological variables such as food abundance and predation risk (Henriksen et al. 2015). Groom et al. (2013) developed a multispecies detection routine (Table 1) that can batch-process up to thousands of overlapping images acquired at sea, first performing a series of systematic cropping operations, then segmentations and classifications aimed at extracting objects of “local brightness [or darkness] contrast” that represent birds (either flying or on the water surface; e.g., Fig. 1D, E, G, and H), filtering out numerous confounding water surface features in the process. Liu et al. (2015) used OBIA successfully with very high-resolution (down to 1.3 cm/pixel) UAS images of wintering Black-faced Spoonbills (Platalea minor) (Fig. 1I).

The original and still widely used commercial OBIA software package is eCognition (Trimble Navigation, Sunnyvale, CA; formerly Definiens eCognition), but other commercial packages such as ENVI (Exelis Visual Information Solutions, Boulder, CO) and freeware applications such as InterImage (Laboratório de Visão Computacional, Rio de Janeiro, Brazil) have also emerged. Even traditional image-analysis platforms such as ArcGIS now possess tools to perform segmentation, and host third-party object-based classification extensions such as Feature Analyst (Textron Systems, Providence, RI).

Automated techniques for detecting and counting mammals

Automated analysis of images to detect mammals has involved a number of additional approaches, in part because the spectral and size/shape characteristics of mammals are not always as distinctive as those of birds. For example, the inferior performance of the automated detection and counting approach used by Laliberte and Ripple (2003) on images of caribou (Rangifer tarandus) than of Snow and Canada (Branta canadensis) geese was attributed to the more variable size of caribou and their inconsistent contrast with the background.

Terletzky and Ramsey (2014) tested a “change detection” approach (commonly used to detect landscape-scale changes in GIS applications) to inventory cattle and horses by comparing aerial images of the same pastures taken a day apart, during which time the animals had changed locations. Their technique correctly identified 82% of the animals, but was marked by a high commission error rate of 53%. Although we are not aware of this technique having been used for aerial photographic surveys of birds, Merkel et al. (2016) used an automated change-detection approach to estimate breeding success of cliff-nesting Thick-billed Murres (Uria lomvia) based on time-lapse photography captured by a stationary camera on the ground.

Other efforts have focused on marine mammals that, like offshore birds, tend to be sparsely distributed over large expanses that render manual image review very labor-intensive. Podobna et al. (2010), Schoonmaker et al. (2010, 2011) and Selby et al. (2011) developed techniques to detect whales at or near the water surface based on spectral characteristics, and Maire et al. (2013, 2015) and Mejias et al. (2013) developed OBIA approaches similar to those described above for birds to detect dugongs (Dugong dugon) in high-resolution UAS images. Maussang et al. (2015) proposed general-purpose algorithms for detecting miscellaneous marine animals in ocean-surface images. Additional aerial image analysis techniques used to detect mammals have been reported by Sirmacek et al. (2012), van Gemert et al. (2015), Terletzky and Ramsey (2016), and Torney et al. (2016).

Aerial thermal-infrared images have been used for automated detection of mammals, including rabbits (Christiansen et al. 2014), seals (Conn et al. 2014), hippopotamuses (Hippopotamus amphibius) (Lhoest et al. 2015), koalas (Phascolarctos cinereus) (Gonzalez et al. 2016), and captive bison (Bison bison), elk (Cervus canadensis), fallow deer (Dama dama), white-tailed deer (Odocoileus virginianus), and gray wolves (Canis lupus) (Chrétien et al. 2015, 2016). However, a major obstacle to using aerial thermal images to detect birds is the low-resolution of the cameras that, at best, record 640 × 480-pixel frames, combined with the generally smaller size of birds compared to mammals. For example, the resolution of aerial thermal images was found to be inadequate for detecting roosting Rio Grande Wild Turkeys (Meleagris gallopavo intermedia) (Locke et al. 2006), although other birds such as Malleefowl (Leipoa ocellata), Sandhill Cranes (Grus canadensis), and grouse (Centrocercus and Tympanuchus spp.) have been detected (Gillette et al. 2013, 2015, McCafferty 2013). The automated-detection method used by Christiansen et al. (2014) was only effective for chickens in thermal images acquired from ≤20 m in altitude. Very low altitude aerial surveys will generally be hindered by a correspondingly narrow field of view on the ground, a resulting increased probability of missing or double-counting subjects that have moved short distances between transects, and a greater chance of disturbing birds.

Automated analysis of satellite images to survey birds

In contrast to aerial images acquired using manned or unmanned aircraft, satellite images generally have a much lower ground resolution. Until recently, the finest resolution available was 60 cm/pixel (e.g., QuickBird images), but satellite image resolutions down to 25 cm/pixel are now available (e.g., WorldView-3), and this may further improve in the future. Following the demonstration of an image-analysis method to estimate the size of Emperor Penguin (Aptenodytes forsteri) colonies from QuickBird images by Barber-Meyer et al. (2007), several investigators have examined approaches for surveying penguins and other waterbirds using satellite images. Although individuals of some large species such as Emperor Penguins can be detected in higher resolution satellite images (Barber-Meyer et al. 2007, Fretwell et al. 2012), the main focus with smaller species and coarser images has been on detecting and estimating the size of colonies based on the extent of fecal staining (i.e., guano), which has a distinct spectral signature (Fretwell and Trathan 2009, LaRue et al. 2014). McNeill et al. (2011) also incorporated automated colony delineation based on fecal staining into their analysis routine for higher-resolution images acquired by aircraft (Table 1). Computer-automated detection of guano in satellite images has also been used to detect colonies of seabirds other than penguins (Fretwell et al. 2015). The overall coarser resolution of satellite images compared to images from aircraft lends itself better to traditional pixel-based spectral analysis, which these studies have employed almost exclusively, in most cases using the supervised Maximum Likelihood Classification approach with ArcGIS's Spatial Analyst extension. Other species that have been manually detected in satellite images include flamingos (Sasamal et al. 2008) and nests of Masked Boobies (Sula dactylatra) (Hughes et al. 2011).

Time and efficiency considerations

All of the computer-automated image-analysis techniques described above require a certain amount of time to set up (e.g., perform any cropping, determine the appropriate spectral threshold value(s) for each image, or develop segmentation and classification workflows) and execute (i.e., computer-processing time). An automated approach may not be worth using if it will require more total time (particularly in setup) than manual analysis of the images, although use of automated analysis may help avert repetitive motion syndrome resulting from tedious manual counts (Milton et al. 2006), and may reduce variation due to observer fatigue. Gilmer et al. (1988) estimated that their rudimentary spectral-thresholding technique using now long-outdated computer hardware and software was only worthwhile if individual images contained >2000 geese. The more streamlined method described by Laliberte and Ripple (2003) was faster than manual counts if >230 subjects were present, and Trathan (2004) reported that a computer-automated approach took half as much time as manual analysis to count a colony of ~11,000 penguins. Milton et al. (2006) similarly found that total analysis time for a sample of 35 photos (some containing several thousand birds) was reduced by half by using an automated approach to count molting Common Eiders (Somateria mollissima) at sea, although they only factored in setup time, arguing that computer processing time did not consume “person time.” The highly autonomous method introduced by Descamps et al. (2011), requiring virtually no setup beyond quick image cropping, took a total of 1:38 h (processing time) to count just under 40,000 flamingoes in five colonies, compared to 24:15 h for manual counts. Elaborate OBIA workflows such as those described by Groom et al. (2011, 2013) may be time-consuming to develop, but, if sufficiently versatile, can subsequently be applied to countless batches of images, year after year, thus making the initial time investment worthwhile. This may be especially useful for maintaining consistency in long-term monitoring that would alternatively involve a turnover of airborne observers.