2.1. Feature Recognition

The past decades have witnessed the explosion of advances in feature recognition [12]. By collecting massive samples with big data technologies, deep neural networks (DNN) [13, 14] can use an artificial neural network to directly give inference results.

Metric learning [15] is an emerging feature recognition method that can verify whether two embedding samples belong to the same identity or not. Authors [16] use a DNN to transform a face image into a vector and then calculate and compare the Euclidean distance between two vectors. Directly using Euclidean distance is equivalent to considering the only intraclass distance. However, sometimes the intraclass distance may be larger than the interclass distance. To address the shortcomings of Euclidean distance, FaceNet [17] used a triplet-based loss function to embed face images. Minimizing the triplet loss function is both minimizing the distance of similar samples and maximizing the distance of nonsimilar samples at the same time.

To train a model enhancing the discrimination of learned features, Wen et al. [18] join supervision of center loss and softmax cross-entropy loss as the loss function of neural networks, thus significantly accelerating the convergence of training models.

2.2. Visual Cryptography

VC is a (k, n) threshold scheme for images. The simple decryption makes VC surpass other cryptographic schemes based on cryptography or watermark [19]. VC expands the application scenarios of secret sharing. It has been an emerging research field in the field of image cryptography since born in 1995 [9, 20, 21]. The conditional VCS encrypts a secret image into shares pixel-by-pixel, hence resulting in pixel expansion. It can eliminate pixel expansion by mapping a block in the secret image into an equal-sized block at the corresponding position of shares. Through vertical arrangements and elaborately processing the correspondence between the secret block and share blocks, we can keep the size and obtain a higher contrast than previous schemes. Another scheme keeping the size of recovery images is the probabilistic VCS which firstly randomly selects a column from a basis matrix and then distributes each pixel in the column to the corresponding position of shares.

The EVCS is also known as friendly VCS[22], which issues the management difficulty caused by noise-like shares in VCS. Ateni [23] proposed a general technique to implement EVCS for any access structure by hypergraph coloring. Due to the pixel-by-pixel encryption mechanism, restored images remain the problem of pixel expansion. Lee [24] proposed a novel algorithm of general access structures to cope with the pixel expansion problem. Instead of the traditional VC-based approach, this scheme first constructed meaningless shares using an optimization technique and then solved it by a simulated-annealing algorithm. Then, the method generates meaningful shares by adding cover images into shares with a stamping algorithm. No computational devices are needed for decryption in this scheme. However, it is only appliable to black-and-white images and needs a lot of time to generate shares.

To address the risk of leakage in biometric features, Jinu [25] proposed a multifactor authentication scheme based on VC and Siamese networks. However, the VCS adopted needs a key to encrypt, which destroys the printable characteristic of VC. Ross [26] preserved the privacy of face template data through a trusted third party and EVCS. In this scheme, shares for decoupling secret face data come from a group of general face images. To improve the quality of restored images, it may use up to 100 shares to encrypt a face image. How to transmit these shares will be a large challenge. Requiring huge storage blocks its application in many scenarios. The accuracy of face recognition decreases because of the interference of VC or EVC in these schemes.

3.1. Expansion-Free EVCS

In the traditional pixel-by-pixel encryption of VC, there is a corresponding matrix collection C₀ and C₁, for a white (w) or black (b) pixel in a secret image. The matrix in C₀ or C₁ consists of n × m Boolean values. When encrypting, we randomly select a matrix from C₀ or C₁ and assign  n rows of pixels of the matrix to the corresponding n shares. Each row contains m subpixels which are reinterpreted as recovery w or b pixel.

Conditions (i) and (ii) make images visible when overlaying. The condition (iii) implies that fewer than  k shares cannot gain any information about the secret pixel.

Taking the (2, 2)-VCS with two participants for example, if the pixel arrangement held by the other participant is complementary to itself, all black pixels will appear, while if pixels held by the other are the same as itself, it will generate half black and half white gray pixels after superimposing. The proportion of black-and-white pixels in each share is fixed, so it will not reveal any information about a secret image.

The disadvantage of VC is that generated shares are meaningless noisy-like images. Although such images will not reveal any information about secret images, they will increase the burden of management and pique the interest of attackers. If these images are tempered by malicious attackers, it is difficult to detect.

The difference between VCS and EVCS is that EVCS has to take into account the color of shares besides the secret image we want to get [27], hence we can see n + 1 meaningful images in an EVCS. Table 1 shows collections for traditional (2,2)-EVCS and recovered subpixels, where the gray-level 1(0) denotes a white (black) pixel (w, b). Each secret pixel is expanded to 4 subpixels in the EVCS (m = 4). Only when enough participants show held shares can the secret images be restored. For example, if the pixel is black in the secret image, black in Share_1, and white in Share₂, the share subpixels will be selected from the third row and second column of Table 1, that is, the [0 1 0 0] and [1 0 1 0] subpixel blocks. After superimposing the two rows, we get a subpixel with four black pixels. The subpixel blocks [0 1 0 0] and [1 0 1 0] may also appear in other basis matrices (e g., $$ and $$), hence the secret information is not exposed in the cover images. When encrypting a black secret pixel, regardless of which basis matrix is selected to generate the corresponding subpixels, we can obtain a whole black subpixel block ([0 0 0 0]), while for white secret pixel, we can only obtain a subpixel block with three black pixels and one white pixel ([1 0 0 0]), which is interpreted as a white pixel in the recovery image, hence the recovery pixel shows a contrast of 1/4 (three white pixels are lost). It can also be verified that for the (2,2)-EVCS, the contrast values of the covers image also are a = 1/4. No matter which basic matrix we choose, the white pixel in Share₁ or Share₂ is represented by a subpixel with two black pixels and two white pixels, while the black pixel is represented by a subpixel with three black white pixels and one black pixel, so the contrast in cover images is 2/4 − 1/4 = 1/4.

To eliminate the pixel expansion problem in the traditional EVCS model, we can use the block-wise operation instead of pixel-by-pixel encryption [28]. We first divide a gray-level image into n nonoverlapping black-and-white pixel blocks B_i,  B_i∩B_j = ∅,  for1 ≤ i ≠ j ≤ n. B_i and halftoned blocks B_h are the same in size. The number of black pixels in B_i and B_h must satisfy the following: b_Bi = s_l/(s_b + 1) × (s_l − ∑Bi/255), where s_l denotes the size of candidate black levels, b_Bi denotes the gray-level of cover blocks, and s_b denotes the block size. To meet the security requirement of VC, the contrast of black-and-white blocks cannot be any, which results in contrast loss after halftoning.

We reinterpret the image blocks as white and black pixels based on the original EVCS. For example, for the (2,2)-EVCS, we will dither a secret gray-level block into the closest equal-sized pixel block with 3 or 4 black pixels and dither a cover block into the closest equal-sized pixel block with 2 or 3 black pixels. If the encryption block has three black pixels (e.g., [1 0 0 0]) and cover blocks have two black (e.g., [0 1 1 0]) and three black pixels, e.g., [0 1 0 0], respectively, then we will set c_s to w and c_h1, c_h2 to w, b. Therefore, we will select $$ as the basis matrix and randomly permuting all columns of $$ to obtain the collections C. At last, we select a matrix C_p from C and dispatch each row of C_p to the corresponding blocks in share_i,  i ∈ [i ∈ 1, …, n].

After processing the secret image by the limited halftoning, we can reuse the underlying EVCS directly to encrypt the secret image into meaningful share images without pixel expansion. Ateni [23] et al. describe a general construction method for EVCS in detail.

3.2. The Recognition of Face Images Recovered from EVC

In this section, we will propose a new embedding method for VC-recovered images to maintain high accuracy in feature identification. Although the proposed EVCS solves the problems of size expansion and noise-like shares, it still cannot achieve perfect image restoration. For existing feature identification methods, the use of VC can alleviate the leakage of centralized storage of template data, but it will also mix noise into sample images, which will reduce the performance of feature recognition.

Witnessing the recent success of DNN, we hope to use these methods to solve the problem of noise interference caused by the introduction of VC in feature identification [6]. Figure 1 shows the flowchart of the proposed approach for distributing and matching face images. In a high-precision neural network model for face images, the face data is converted into the corresponding weights in the network. Using transfer learning [29], we can extract and transfer these weights to other neural networks (e.g., reduced images mixed with noisy signals). Transfer learning allows sharing the learned model parameters and structures to a new model in a specific way, thus speeding up and optimizing the learning efficiency of the model and avoiding learning from scratch. We first train a softmax classifier on the training data using a pretrained neural network model and then fine-tune the weights of the last layer or layers using a dataset recovered from EVCS. To avoid affecting learned weighted when training, we freeze the pretrained model and add multiple fully connected layers at the end of the network. After embedding reduced images, we select a loss function to efficiently obtain a high-precision model. When used as a classification task, the model selects the most similar image class ID in the training set to the test image as the output.

In deep learning, many methods use pairs of samples to compute gradient loss and update model parameters [15]. We use the triplet loss [17] gradient optimization in the face recognition system. Although center loss [18] may have a faster convergence rate, triplet loss is more robust for image quality degradation. The basic idea of triplet loss is to make the distance between negative sample pairs larger than that between positive sample pairs. In the training process, positive sample pairs and negative sample pairs are selected at the same time, and the anchor of the positive and negative sample pairs is the same. Choosing which triplets to train is important for achieving good performance. When the distance between the negative sample and the positive sample pairs is greater than a threshold, the model sets the loss to 0 and ignores the sample pairs. Because the distance between positive and negative samples and anchor points is considered at the same time, the performance using triplet loss is often better than that of contrastive loss [30].

The triplet loss minimizes L, then d(a, p)⟶0, d(a, n)⟶margin. The goal is to shorten the distance between a and p and lengthen the distance between a and n. For the case  L = 0, that is, d(a, p) + margin < d(a, n), this is an easily discernible triple and does not need to optimize, because a and p and a and n are far apart; for the case d(a, p) < d(a, n) < d(a, p) + margin, the triples are in the fuzzy region. This case is the focus of our training.

The same deep neural network is used to extract the features of these three images to obtain three embedding vectors, which are input parameters of triple loss. The model is updated by using a backpropagation algorithm according to loss and iterated until a stable neural network model is obtained. When inferring samples, we first generate triplet image pairs and then output the distances between anchor, positive, and negative embedding [31]. After transforming the distances into probability models, we can get desired results. During the registration process, we decompose the private face data into two or more meaningful images. After dispatching shares, the system discards the original data. The encrypted face data are stored in two or more database servers. Unless these servers collude, the private data will not be revealed to any server. In the authentication stage, the feature recognition system sends requests to database servers to transmit corresponding shares to it. Once finishing the classification recognition or similarity matching task, our system will discard the reconstructed secret image. In the whole registration and recognition process, the secret image will be only recovered in use. Because private biometrics cannot be extracted from any single database or server, the whole authentication does not reveal any feature information.