3.1.

CNN-based Watermarking

CNN is a deep-learning network that has convolution characteristics. It generally contains five layers: the input layer, the convolution + rectified linear unit (ReLU) layer, the pooling layer, the fully connected layer, and the output layer. CNN is inspired by the visual cortex of the human brain. A neuron is only activated in a receptive field in response to a stimulus. The CNN model takes an input (image) as a vector, and later, the convolution operation is applied to extract the essential features (kernels or filters are used for this). The ReLU activation function is used to make the negative values zero, and later, pooling is applied to reduce the number of parameters. The reduced pooled information is passed to the fully connected network by flattening the input for classification purposes. The CNN model is widely used in image watermarking and data hiding due to its lesser complexity. Though it has many applications, CNN is used primarily in watermarking for embedding the watermark, extracting the watermark, increasing the visual quality of the cover image, and identifying the best embedding location for the watermark. Ingaleshwar and Dharwadkar¹⁸ proposed an optimization-based watermarking technique using deep CNN to embed the watermark image into the cover image. Decomposition on the cover image was performed into the different grids to obtain the related features of gridlines. The interesting region was computed using the deep CNN, which was trained using the optimization techniques. Fitness measures are used for embedding the watermark, and for that, wavelet sub-bands are selected. For recovering the watermark, cover media and fitness function are required. Compared with the methods presented in Refs. 1920.21.22.–23, the suggested method performs better in peak signal-to-noise ratio (PSNR). Even though the technique is robust, there has been no detailed investigation of overall embedding and recovery costs. Also, further study is needed on the robustness of the approach by performing more attacks at varying noise levels.

A blind image watermarking scheme was suggested by Bagheri et al.¹⁹ in which Mask R-CNN was used to compute the embedding strength. The watermark was embedded in the cover by computing the block into the selected discrete cosine transform (DCT) and discrete wavelet transform (DWT) blocks using lower region-of-interest pixels. Even though the watermark is extracted well, the method provides limited embedding capacity and watermark security. A wavelet-based watermarking for digital media was developed by Zheng et al.²⁴ to investigate the imperceptibility and robustness of the watermark. Initially, the cover media was transformed into different bands using DWT, and then the watermark data singular value was inserted into the high bands of the cover media. Then, the transformation was applied to the low bands by wavelet transformation, in which the watermark sequence was embedded into the selected low bands. The singular vector of the watermark and scrambled sequence were used to obtain the singular and watermark sequence, respectively. Later, using the CNN established the connection between the watermark, cover media, and watermarked media for robust extraction of the watermark. Due to their low embedding cost for different types of watermarks, they extend the suitability for various practical applications. In Ref. 25, a CNN based watermarking technique was developed to embed the scrambled watermark into the DWT cover image. The technique uses a fast region-based CNN model to achieve robust and blind extraction of the embedded watermark. Compared with conventional techniques,^26–29 this method provided better classification accuracy with strong invisibility and lower execution time. However, proper investigation is needed into the robustness of performance against different image processing attacks. Plata and Syga³⁰ proposed a watermarking technique based on CNN to embed the spatial watermark data into the cover image. A loss function was designed by the authors for the neural network training to improve the robustness and other watermarking trade-offs. Compared with other existing work,^31–33 the scheme provides higher robustness with minimal distortion. The schemes need to be evaluated for different image processing attacks.

A deep CNN network-based blind watermarking technique for copyright protection was proposed by Mun et al.³⁴ In this technique, first the carrier and watermark media are divided into non-overlapping blocks. Later, the secret is computed using watermark data corresponding to the positions of the block. Subsequently, the watermark is embedded into the carrier using the deep-CNN model, and finally, the model is used to extract the watermark from the cover. The scheme is robust to different salt and pepper noise attacks with good imperceptibility. However, the cost of the method is relatively high. The encryption-compression-based watermarking technique was proposed in Ref. 35. The method uses transform domain watermarking using the lifting wavelet transform (LWT), randomized singular value decomposition (RSVD) and Heisenberg decomposition (HD). Further, a CNN-based denoising network was used to improve the extracted watermark. This technique outperformed the existing methods proposed in Refs. 3637.38.–39 with an improvement of 27.84% examined on different attacks.

A transform domain watermarking method was proposed by Hsu and Hu⁴⁰ based on quaternion discrete cosine transform (QDCT). The tradeoff between robustness and imperceptibility of the watermarked image was balanced by the grey wolf optimizer. The blind watermark extraction was used, and the denoising CNN model was used to improve the visual quality of the watermark. Based on the experimental evaluation, the watermarking technique performed better than the traditional techniques presented in Refs. 4142.43.–44. Another transform domain-based watermarking scheme was proposed by Kandi et al.⁴⁵ in using the CNN network to improve the tradeoffs of the watermark. The technique uses the auto-encoder learning capability of a CNN to improve its robustness. It uses the input–output information for embedding using the CNN weights, which are highly prone to attacks. To protect the ownership of the deep CNN network, a watermark-based scheme was proposed by Nagai et al.,⁴⁶ which uses the concepts of the technique proposed in Ref. 47. In their technique, the secret is inserted into the different groups of the convolution layer of the original network. The technique is adequate, but the technique is limited to dealing with attacks such as surrogate model attacks and watermark overwriting. Another technique for protection of the ownership of a CNN network was proposed by Guan et al.⁴⁸ using the watermarking techniques. The technique uses the model compression’s pruning theory to embed the hash using the secure hash algorithm (SHA-256) as watermark data into the convolution layer of the residual neural network (ResNet) 152 network. However, the technique limits the usability of the real-time applications due to their high computational costs.

Some of the recent works using the CNN model are mentioned below are also given in Table 3.

Table 3

Summary of CNN-based watermarking.

The challenges of CNN models for watermarking systems are as follows:

3.2.

Generative Adversarial Network-based Watermarking

This is a deep-learning-based model with generative properties. It has two networks: the generator network and the discriminator network. The GAN network is widely used in watermarking for watermark verification and generation due to its generative and discriminative ability. GAN is also used to enhance network security by generating a unique watermark for every input. The generator network $G$ takes random noise input $Z$ from distribution $P(Z)$ and generates sample $S$. The goal is to replicate a particular type of distribution $P(X)$. Later, the discriminator network $D$ discriminates between the generated (fake) samples and the actual sample. During training, the discriminator penalizes itself for misclassifying a real instance as fake or a fake instance as real. The combined generator and discriminator loss function ($L$) used by the GAN model is obtained by Eq. (3) as

Wu et al.⁴⁹ proposed a deep-learning model to verify the corrupted images caused by dense watermarks. The model consists of a generator and a discriminator for improving the recovered images’ quality and verification performance. The generator is an autoencoder that maps densely watermarked, corrupted images to a representation vector and decodes the vector to an red-green-blue (RGB) image. The discriminator controls the contents of the generated images and minimizes the feature loss. The ResNet-46 model is used to extract the recovered images feature and ground truth images. It achieves a verification accuracy of 96.36% at the false positive rate (FPR) of 1%. In Ref. 50, the authors proposed a robust data hiding scheme using the GAN for securing genuine documents. At first, using the geometric correction, the document is adjusted to the required form. After that, the document is generated by the adversarial network, and its security is enhanced by embedding the secret information into the document using the pseudo-random number to generate a watermarked document. The technique is robust and performs better than the work mentioned in Refs. 5152.–53. A robust watermarking system based on variational autoencoder networks is provided by Wei et al.⁵⁴ for copyright protection. Encoder, decoder and detector subnetworks make up the embedder and extractor network. A 1-bit watermark image is embedded in the host image during training, and the encoder and decoder subnetworks build a robust representation model of the cover image. The detector subnetwork acquires the ability to extract the 1-bit mark from the watermarked image. The approach improves the visual quality of the marked image, but further research is needed to determine the robustness of watermarked images. A blind watermarking scheme based on deep learning is proposed in Ref. 55. The technique consists of four components: an encoder, a decoder, two identical noise layers, and an adversarial discriminator. The two identical layers were used to embed and extract the watermark encoder and decoder, respectively, and to make the watermark robust against different attacks. Further, an adversarial discriminator was used to improve the robustness and concealment of the watermark. The technique is robust against various attacks with good imperceptibility. However, the model complexity is very high. To prevent malicious attacks during transmission or from illegal use of diffusion-weighted imaging (DWI) images, a multiscale robust watermarking technique was proposed by Fan et al.⁵⁶ The suggested technique includes multiscale characteristics and a generative adversarial. By integrating full-scale features, the DWI images are first rebuilt to mimic the original DWI images. Watermark is embedded into the multiscale reconstructed features. To enhance the visual quality of the reconstructed picture, an optimized boundary equilibrium generative adversarial network| discriminator is suggested. Finally, to learn the watermark distribution feature, pyramid filters and multiscale max-pooling are used. Fang et al.⁵⁷ introduced a unique triple-phase watermarking scheme to prevent image distortion in practice. A noise-free initial phase, a mask-guided frequency augmentation phase, and an adversarial-training phase comprise the approach. In the first phase, an encoder–decoder was trained end-to-end using a just-noticeable difference (JND) mask image loss. The encoded characteristics are then subjected to a mask-guided frequency augmentation method in the second step. Later in the final phase, it intends to train a decoder for dealing with non-differentiable distortion through adversarial training. The approach is more robust than the studies reported in Refs. 5859.60.–61. In Ref. 62, the authors proposed a semi-fragile watermarking scheme based on deep learning for media authentication. The technique consists of three modules: an encoder network, a decoder network, and an adversarial discriminator network. The encoder network encodes the input images and produces the watermarked images. After that, the marked image goes through two image transformation functions: one from a benign transformation set and the other from a malicious transformation set to produce a benign and malicious watermarked image, respectively. Later, the benign and malicious watermarked images are fed to the decoder network using an adversarial network that discriminates between the benign and malicious images. The technique is robust and provides tamper detection, but the technique needs to be investigated for additional types of image processing attacks.

Some works based on the GAN model are mentioned below and are summarized in Table 4. The challenges of the GAN model for watermarking systems include the following:

Table 4

Summary of GAN-based watermarking.

3.3.

DNN-based Watermarking

The DNN is the most commonly-used network; in it, multiple hidden layers are present between two layers: the input layer and the output layer. DNNs model resembles the human brain; to be specific, a DNN takes the raw format input (image or audio) $x\in R^n$ and maps it to the output layer using the parametric function $p=F_{\theta }(x)$, where $p\in [1,n]$, which is based on the network architecture and combined parameters of all of the layers in the network. When training the DNN, the goal is to minimize the loss between the predicted and ground truth labels and optimize the network parameters $\theta$. The back propagation algorithm is the widely used approach for training a DNN; in it, the loss gradient of the output layer is back propagated to update the network parameters. Training and adjusting the parameters of the DNN allows the network to learn, encode, and decode the watermark for embedding, extracting the watermark, and network protection. DNNs are widely-used in the field of watermarking for embedding and extracting the watermark.

Hou et al.⁶³ proposed a watermarking technique based on the enhanced version of multiple histogram modification. The method follows a particular criterion to embed a watermark into multiple histograms. Initially, the cover image decomposes into two different sets of pixels. Later, using the DNN, multiple histograms are produced by the classification method. For embedding the watermark data, the produced histograms’ optimal bins are selected from each. However, the visual quality between the cover and watermarked image is excellent. Compared with the existing technique,^64–66 the technique achieves a higher PSNR, but it is necessary to assess the technique’s robustness against various attacks. Further, the overall embedding cost needs to be computed. For copyright protection and ownership confirmation of DNN models, Zhang et al. presented a watermarking technique in Ref. 67. The Uchida et al.⁴⁶ method has several drawbacks; however, an upgraded threat model is employed to enable the black-box mode verification and application programming interface (API) access. The method shows good accuracy at the cost of small overhead. However, the authors do not conduct a transparent investigation of the overhead and security of the model. Another method is presented by Wu et al.⁶⁸ for deep model copyright protection and ownership verification using the watermarking technique. In this method, the output image from the deep-learning model is used to obtain the watermarked image. The embedded watermark can only be extracted by the extraction network for verification. The method’s usefulness for real-time applications is constrained by the fact that it is only evaluated for three different types of attacks and there is no transparent study of the computational time of the method. To preserve intellectual property rights and to protect the ownership of the DNN, Deeba et al.⁶⁹ proposed a watermarking technique. This technique generates and embeds the watermark data into the neural network. The ownership verification is done by a specific type of input–output pair. The technique performs better, but the performance is judged based on only two attacks, and the authors do not conduct a transparent investigation of the execution time of the technique. In Ref. 70, the authors proposed a watermarking technique based on a DNN for the ownership verification of the multimedia document. The cover image is divided into $8\times 8\text{pixels}$ to embed the binary watermark of 1-bit $\{-1,1\}$. After that, at the $\mu$’th block, the watermark image is embedded in the DCT coefficient. In the same procedure, the inverse function is applied for extraction. The technique produces a similar original and watermarked image with acceptable PSNR.

Some works based on the DNN model are mentioned below and are summarized in Table 5. The challenges of DNN models for watermarking are as follows:

Table 5

Summary of DNN-based watermarking.

3.4.

Others Perspectives

In addition to the above-mentioned work, some other deep-learning models such as RNNs, backpropagation neural networks (BPNN), and artificial neural networks (ANN) are also used for watermarking techniques. Some of these works are mentioned below and are summarized in Table 6.

Table 6

Summary of other deep-learning model-based watermarking.

For the ownership verification and copyright protection of digital color images, Sinhal et al.⁷¹ proposed a low-cost, blind watermarking scheme. The image is first converted into a YCbCr model using a selection of $4\times 4$ blocks of $Y$-component randomization. Later, using a low-cost ANN model to implant a binary watermark, the chosen component is dissected using integer wavelet transformation. In the LWT domain, Islam et al.⁷² suggested a reliable watermarking method utilizing an ANN. The watermark is embedded using the LWT cover image’s randomized coefficient, and it is extracted using an ANN model. The selected sub-band coefficient is first randomized using a key after the cover picture is modified using the LWT. Later, using a different key, the randomized coefficient is used to obtain the randomized blocks. The chosen sub-randomized band’s block is then used to incorporate the watermark. ANN is utilized for watermark extraction. Overall, the approach is reliable and blind, but the embedding and extraction costs were not transparently investigated in the study, and the watermark’s capacity is insufficient for practical use. The edge detection-based watermarking approach was suggested by Kazemi et al.,⁷³ and it involves embedding private information in the edge of the color picture. An edge detector is utilized to identify the edge of the color RGB cover media, and the resulting media is subjected to a contourlet transform to calculate the directional components. Using a genetic technique, the logo image is first scrambled to provide better verification before being embedded into the cover image. The hidden watermark is extracted via a combination of differential evolution and multilayer perceptron. The approach, however, exhibits weak resilience to the few attacks. A deep-learning-based watermarking scheme was proposed by Singh et al.⁷⁴ in which multiple watermarks are embedded into the DWT-DCT domain single value of the cover image. The three-level DWT is used to first break down the cover image. The low-frequency band (LL3) and low-high bands (LH2) are taken into consideration for embedding the watermark and encoded text data into the cover picture, respectively. Later, selective encryption is used to encrypt the watermarked picture to save cost. Finally, to reduce the distortion effect applied to the extracted watermark image, a BPNN is used. The technique ensures the confidentiality of the data and shows stronger robustness against different attacks compared with Refs. 7778.79.80.81.–82. However, a transparent investigation of security and cost is required. A data-hiding method utilizing the long short-term memory-based recurrent neural network (LSTM-RNN) was proposed by Singh et al.⁷⁵ A particular criterion is used to acquire the electrocardiogram signal (TP) section of the ECG signal, which is then used to encrypt watermark data. The distortion between the original and forecasted signals is reduced using LSTM-RNN. The suggested method outperforms the conventional methods indicated in Refs. 8384.85.86.87.88.89.90.91.–92. However, the watermark’s capacity is constrained, which limits the method’s applicability for real-life use. A blind DCT-SVD-based watermarking technique was proposed by Wang et al.⁷⁶ Initially using the median filter, the cover image is enhanced to improve the robustness of the watermark. Later, without altering the cover picture, RCNN is used to map the association between the watermark and cover images. The technique provides better robustness and stability than the works presented in Refs. 9394.95.96.97.–98.