Identification of non-glandular trichome hairs in cannabis using vision-based deep learning methods

3.1 Data collection

The plants examined were those obtained from various cases under investigation and indictment. The plants were collected from different apartments or greenhouses where they were suspected of being cannabis plants. In some cases, the plant material was already packaged as drug parcels intended for distribution. For plants that were determined not to be cannabis through laboratory testing using various solvents and microscopic analysis, which showed no evidence of cannabis trichomes, there was suspicion that these plants had been sprayed with synthetic cannabinoids. Synthetic cannabinoids typically consist of various chemical compounds designed to mimic the effects of THC, the active compound in cannabis. Criminals attempt to spray different substances that vary from case to case and plant to plant.

In this study, we did not analyze the chemical composition of the plants in the numerous cases where laboratory results were negative for cannabis (i.e., they did not contain THC) or showed hairs/trichomes that did not resemble those of cannabis. In each case, the plants were examined by an expert using the two mentioned chemical tests and a microscopic visual analysis to provide expert opinions for investigative teams and courts regarding whether the plant material was cannabis (defined as a dangerous drug under the Dangerous Drugs Ordinance) or not.

The plant material's texture is typically in the form of broken leaves or flowers, some of which are evenly ground up. A small amount, ~26 mg (a minimal quantity is sufficient for qualitative rather than quantitative tests), is taken and transferred to a Petri dish. The material is then examined under a microscope with a zoom magnification level between 20× and 128× to identify the characteristic cannabis trichomes. This process involves a visual identification of cannabis trichomes, which are unique to the cannabis plant, using a microscope for enhanced resolution. This method is commonly used for qualitative testing, ensuring that the sample contains cannabis material based on the presence of its distinctive features, such as glandular trichomes.

3.2 Data processing

The dataset consists of 2826 microscopic images, 1286 of genuine cannabis and 1540 of non-cannabis material. These were collected from more than 2000 different samples. In most cases, each sample was photographed once, but in some cases twice to achieve better focus on different parts of the material. The exact number of sampled plants is unknown, since the material was collected by the police as part of law enforcement activity against illicit drug trafficking. As such, most of the plant material was collected in packages ready for distribution, already ground, crushed, and minced, and in some cases, pulverized and powdered. We used an OLYMPUS SZX16 microscope [41] and a Pixelink PL-D799CU camera with Pixelink software [42], with image resolution of 4096 × 1710 pixels and standard jpg RGB color format. To annotate the trichome hairs, we used MakeSense, a free open-source visual software tool [43]. Most of the images were annotated with rectangular regions of interest (ROI) surrounding the trichomes, whether “real” or “fake”, usually 1–2 bounding boxes per image. Bounding boxes are marked by two mouse clicks determining the object's upper-left and lower-right corners. The annotation software allows zooming in and dragging mode, thus enabling a tight enclosure of the objects. Each annotation contains five parameters; the first specifying the object's semantic label (“real” or “fake”), and four numerical values indicating the x,y location of the region's center, and width, height of the rectangular bounding box, normalized with respect to the image size. These ROIs vary in size, between 45 × 35 pixels and 1520 × 800 pixels; most are smaller than 512 × 512, the average size being 287 × 279 pixels for cannabis trichomes and 339 × 337 pixels for non-cannabis trichomes. The total number of annotated bounding boxes was 3158, 1461 of cannabis trichomes and 1697 of non-cannabis trichomes. Samples of original images and bounding box annotations are presented in Figure 3. It should be noted that this dataset contains a variety in terms of lens focus, illumination and reflectance, artifacts, and amount of relevant foreground material versus irrelevant or noisy background in the images. At this stage of data acquisition and trichome annotation, the raw images themselves were used, without any preprocessing.

3.3 CNN classifier models

CNN classifiers are deep learning algorithms famous for their ability to extract representative information from image data with minimal preprocessing. CNNs have demonstrated outstanding performance in image recognition and classification tasks, mainly due to their structure: A sequence of computational layers enables stratified representations of the data, from low-level characteristics to higher-order abstract features. A key aspect of network optimization is that parameter values are iteratively fine-tuned, or “learned,” as opposed to being engineered or “hand-crafted,” as in traditional image processing algorithms [44].

We used two different CNN models for binary classification of the images. The first is a basic CNN implementation, widely used in image classification. It consists of four blocks of layers, each block composed of a convolution layer, a batch normalization layer, a rectified linear unit (ReLU), and a max-pooling layer. These are followed by three fully connected linear layers with dropout and ReLU activation between them, and finally a softmax operation that outputs a probability score for each class. The class with the higher probability is selected as the network's prediction of the image label. Figure 4 depicts a schematic view of this particular CNN structure. Images inputted to this network are rescaled by a factor of 0.25 on each axis, into 1024 × 428 pixels, using bi-cubic interpolation. The first four (convolutional) blocks output 16, 32, 64, and 128 feature maps, respectively. All convolution kernels are 3 × 3, default batch normalization and ReLU are used, and all max-pooling layers use a 2 × 2 pixel kernel. The input and output number of nodes in the three fully connected layers are (128 × 26 × 64, 1024), (1024, 32), (2, 32), respectively. The two dropout layers apply a 0.25 probability.

The DLA implementation consists of similar layered blocks (convolution, batch normalization, and ReLU), but with a tree structure, as opposed to a completely sequential arrangement. This structure utilizes skip connections, residual operations, and hierarchical representations by adding and concatenating outputs from previous layers as inputs to next layers. The DLA network is more densely populated with blocked layers compared with the basic CNN, followed by a composition of fully connected, dropout, and softmax activation, ending with a probability score for each class. A schematic diagram of the DLA's architecture is presented in Figure 5. Images inputted to this network are rescaled by a factor of 0.125 on each axis, into 512 × 214 pixels, using bi-cubic interpolation. The first three blocks consist of convolution, batch normalization and ReLU layers, without any pooling. These block output 16, 32, and 32 feature maps, respectively. All convolution kernels are 3 × 3; default batch normalization and ReLU are used. These are followed by four tree blocks, composed of hierarchical structure of multiple nodes consisting of convolution, normalization and ReLU layers. These nodes are connected via skip connections with addition and/or concatenation operations linking them. The four tree nodes output 64, 128, 256, and 512 feature maps, respectively. These are followed by an average pooling layer with a 4 × 4 kernel, and two blocks of dropout and fully connected layers, dropouts with 0.25 probability and fully connected layers with (512 × 3 × 32, 32), (2, 32) input and output nodes, respectively. Finally a softmax operation outputs a probability score for each class, and the class with higher probability is selected as the DLA's prediction of the image label.

3.4 YOLO and DETR object detection models

YOLO is a single-shot model for detection and localization of objects in images that uses a robust CNN architecture [45]. While previous object detection models used a pipeline that went over the input image multiple times, YOLO was the first to provide real-time detection using a single pass on the input images [39]. The model divides the image into a fixed grid and calculates for each grid cell the probability score for objects' location and class label. Though YOLO predicts multiple bounding boxes per grid cell, it selects one object prediction per bounding box during training. The YOLO version chosen for this study is YOLO-V4, which provides faster and more accurate detections compared with other versions [46]. We used Bochkovskiy's yolov4-darknet implementation [47].

DEtection TRansformer (DETR) [40] is a type of vision transformer (ViT) [48] that adapts a self-attention mechanism [49] that replaces convolutional operations in image classification and recognition tasks. The basic attention mechanism consists of a sequence of linear embeddings of image patches and non-linear embeddings of the patches' positional encoding. DETR uses multiple attention blocks in an encoder-decoder structure. With a flexible architecture that is relatively simple to implement, DETR reaches high accuracy and impressive run-time performance on standard benchmarks. We used the roboflow-huggingface implementation [50].

Object detectors do not classify images; they output bounding boxes surrounding candidate sought-for objects, with a label and probability score attached to each such object. Therefore, a decision scheme for classifying the image is still required. Basically, if more objects of a certain type are detected, prediction of the image label is straightforward. In this binary classification task, if more objects are detected from one class than the other, the image is labeled as belonging to that class. If the number of detected objects is equal between the two classes, the label is determined by the object with the highest probability score. If no trichomes are detected, the output is labeled as “no detection,” which is treated as a false prediction. This decision scheme is depicted in Figure 6. The accuracy of these models was assessed according to the percentage of true detections out of all detections.

3.5 The composite method

Since the main objective is to distinguish between two categories, “real” and “fake” cannabis, and not necessarily to obtain details or features appearing in the images, a binary classifier is the first and most obvious choice. However, microscopic images contain much background and artifacts. These affect the models' algorithmic mechanism, and heavily bias the performance and outputs, especially in small datasets (thousands as opposed to millions) as used for this study. This leads us to identify specific object features that are assumed to differentiate cannabis hairs (non-glandular trichomes) from non-cannabis plant hairs, namely, the shape of the hairs. As these are not trivial to detect, and in some cases are not detected at all, we propose to combine the two approaches, taking advantage of the strengths of each approach. To this end, we propose two versions of a special purpose decision strategy, integrating a classification model and an object recognition model, implemented via a multi-stage algorithm. The basic idea is to apply an object detector, and in cases where it fails in confident object detections to apply an image classifier. A more sophisticated option is to use, in case of no detections, a similar object detector trained to a lower confidence threshold, thus outputting more candidate bounding box predictions. The second stage classifier, in this case, operates on the bounding boxes produced by the first stage detector, not on the entire image, thus having a better chance of correct identification due to its focus on image patches which are more informative. Below, we formulate these strategies as algorithmic pipelines. Flowcharts depicting these mechanisms are presented in Figure 7.

4.1 CNN classification models

Both classification models, the basic CNN and the DLA, were trained for 80 epochs. We used a custom binary cross entropy loss function with equal weights for each class, and the standard Adam optimizer [51] with a 10⁻⁴ learning rate. The batch size selected was the largest possible for the GPU's memory, 16 for the CNN model and 6 for the DLA model. Training augmentations included geometric transformations – rotations, translations, scaling and horizontal flip, and texture transformations – brightness, hue, saturation, gamma, and contrast. All hyper-parameters values are specified in the configuration files available in the authors' github repository. The model weights selected were those that yielded the highest prediction accuracy on the validation set. Validation was performed upon the end of each training epoch. Results were evaluated by the percentage of correct prediction of image labels on the test sets. Performance was measured by the average precision of the models' predictions on the two distinct data partitions. Table 2 presents results. Similar accuracies were achieved in the two distinct data splits. As expected, the DLA performed better than the basic CNN model, achieving 95.89% and 92.77% accuracies, respectively.

4.2 YOLO and DETR object detection models

In both object detectors, we used the default configurations provided by the software implementers, including network hyper-parameters, loss functions, and optimizers. The only exceptions were the number of training epochs and the confidence threshold. Training deep neural networks is an iterative process, executed until performance on the validation subset does not significantly improve. The YOLO training process spanned over 6000 epochs, and the model's weights were selected based on the maximal mean average precision. The DETR training process spanned over 500 epochs, and the model's weights were selected based on minimal loss value. The confidence threshold is a value that determines the model's certainty in assigning a label to a detected object. In general, higher thresholds result in fewer detected objects, but with a higher probability that their assigned labels are correct, whereas lower thresholds result in more detected objects having lower confidence levels. After experimenting with a range of confidence values, we found that YOLO provided optimal detections (i.e., bounding box coordinate locations and correct labels) with a low-confidence threshold, whereas DETR required a high threshold for high performance. For optimal results on the validation subsets, we used confidence threshold values of 0.05 and 0.3 for YOLO and DETR, respectively. An example of a good YOLO detection of a genuine cystolithic hair with a confidence of 0.18 can be seen in Figure 8. Figure 9 displays an example of a good DETR detection of a non-cannabis plant hair with a confidence of 0.97. These specified confidence levels are on examples of detected trichomes taken from the test subset, provided for each detected object along with its class label.

Performance of the object detectors is measured as the percentage of correct predictions after the decision scheme described in Section 4.3 and depicted in Figure 6. “No detection” predictions are considered erroneous. For fair comparison, the exact same training/validation/test data partitions were used. Moreover, this decision scheme, although relying on the object detectors' output, predicts a binary classification label for the whole image, as do the classification models. Results, on the test sets, are presented in Table 2. Overall, YOLO achieved higher accuracy (92.12%) than DETR (85.30%). This was expected, as ViT-based models are generally known to surpass CNN-based models only when using networks that were pre-trained on large datasets.

4.3 The composite method

The two versions of the composite method were tested with both YOLO and DETR as base detectors. Each base detector had the same configuration, hyper-parameters, and trained weights, as those used when applying the object detector as a standalone method. The second-stage image classifier is the same DLA network that was previously used as standalone classifier for whole images, with minor modifications. Since this classifier operates on detected objects from the first stage rather than whole images, the input size is smaller, and was further rescaled to 256 × 256 pixel patches. To optimize memory efficiency, the batch size was increased from 6 to 12, and the learning rate was reduced to 10⁻⁵. The smaller sized input affects the resolution, but not the structure, of the feature maps in the various layers, resulting in a reduced number of input nodes to the first fully connected layer, from 512 × 32 × 32 to 32 × 3 × 32 nodes. In the three-stage strategy (Figure 7, right), the bounding boxes provided to the classifier are generated by the YOLO/DETR models, which were trained with lower confidence thresholds. The reasoning here is the preference for “no detections”, at this stage, thus allowing an opportunity for using the whole-image classifiers, rather than erroneous detections that will eliminate the need for their usage, thus outputting incorrect labels. Results for all four composite methods are presented in Table 3. The training/validation/test data partitions used here are the same as in all previous experiments. The evaluation criteria are also the same, that is, the percentage of true classifications on the test subset. The composite methods that used YOLO had better accuracy compared with the methods using DETR. These results were to be expected, as YOLO also had better performance as a standalone object detector. An example of a good YOLO detection that DETR failed to detect correctly is shown in Figure 10. While YOLO detected mostly glandular hairs from the correct class (cannabis hairs), DETR wrongly detected a single hair as a non-cannabis hair. The best performing method was the one that used YOLO with the bounding box classifier, with an overall accuracy of 97.61%. This method was chosen as the optimal composite method, demonstrating a significantly better performance compared with other approaches (Table 2).