Detection Method of Data Integrity in Network Storage Based on Symmetrical Difference

Abstract

In order to enhance the recall and the precision performance of data integrity detection, a method to detect the network storage data integrity based on symmetric difference was proposed. Through the complete automatic image annotation system, the crawler technology was used to capture the image and related text information. According to the automatic word segmentation, pos tagging and Chinese word segmentation, the feature analysis of text data was achieved. Based on the symmetrical difference algorithm and the background subtraction, the feature extraction of image data was realized. On the basis of data collection and feature extraction, the sentry data segment was introduced, and then the sentry data segment was randomly selected to detect the data integrity. Combined with the accountability scheme of data security of the trusted third party, the trusted third party was taken as the core. The online state judgment was made for each user operation. Meanwhile, credentials that cannot be denied by both parties were generated, and thus to prevent the verifier from providing false validation results. Experimental results prove that the proposed method has high precision rate, high recall rate, and strong reliability.

1. Introduction

In recent years, the cloud computing becomes a new shared infrastructure based on the network. Based on Internet, virtualization, and other technologies, a large number of system pools and other resources are combined to provide users with a series of convenient services [1]. The cloud computing has the advantages: providing convenient computing resource sharing pool, flexible resources, safe and controllable data, cost saving, unified management, and low-cost fault tolerance, so the related products of cloud service are more and more popular with users. With the rapid development of information technology, the traditional data storage mode cannot meet the new needs and challenges [2]. The cloud storage has attracted great attention due to its low cost and high efficiency. As a new storage mode, the cloud storage has attracted more and more attention with the rapid popularization. How to ensure the correctness and integrity of data files stored in cloud servers is one of the key issues in the development of cloud storage technology [3]. Based on the importance of data integrity detection, many excellent research results have been obtained in this field.

One study [4] proposed a method for security data integrity detection. The algorithm performs cross-validation by establishing a dual-evidence mode of integrity verification evidence and untrusted detection evidence. The integrity verification evidence is used to detect data integrity and use untrusted detection. Evidence determines the correctness of data verification results. In addition, the reliability of the verification results is ensured by constructing a detection tree. However, this method has poor recall. Literature [5] proposed a method for data integrity detection in big data storage. Cross-checking the integrity of data in the storage with a two-factor verification method to verify that the data integrity of the big data store is complete. A check tree is constructed to ensure the reliability of the verification results. The data storage integrity of this method is good, but the encoding time of this method is too long and the efficiency is low. Study [6] proposed a method for detecting the integrity of multi-copy data in a cloud storage environment. By introducing a classic B + tree authentication structure and designing a lightweight system model, users can timely grasp the integrity and consistency of multi-copy data on the cloud server side. Adding the rank value to the B + tree enables users to quickly and accurately locate the location of the wrong data, which is convenient for the next processing. However, after this method is used, the data availability rate is low, which cannot satisfy the effective storage of network data.

In order to improve the precision rate and recall rate during the data integrity detection, a method of detecting data integrity of network storage based on symmetric difference was put forward.

2. Research Status

Data has always been an important carrier of information. Traditional data storage methods mainly rely on semiconductors, optical, magnetic, etc. to achieve data storage, data is stored locally. With the rise of the Internet, storage technologies based on network communication emerge in endlessly, such as disk array, network access, storage area network and other storage technologies. However, with the advent of the era of cloud computing, cloud storage has emerged as the times require, and gradually developed into the mainstream storage technology [7]. The traditional method is likely to be replaced by cloud storage. In the cloud platform, the cloud service provider (CSP) is mainly responsible for storing and managing data, and maintaining the availability and reliability of data. However, the data stored in CSP by enterprises or individual users may involve customer related information, personal hobbies, and habits, which often has potential value and is easy to be coveted by people with ulterior motives. In addition to data security, the research on cloud storage also includes data integrity. Because of the important theoretical research and practical application value of data in storage security, it has been concerned by many experts in related fields at home and abroad. Ateniese et al. Took the lead in publishing the paper of PDP (proof of data possession) scheme, which can still ensure the data can be maintained on unreliable storage nodes. In this paper, T represents a combination of tags corresponding to the set of file blocks to be queried. ρ is a hash value under the set of file blocks to be queried. When data integrity is detected. First remove the index information of the tag in t to get the pre calculated tag information, and then compare it with ρ through some established calculation. As long as they are equal, it means that the data stored on the server side keeps the original state without any tampering, which proves the integrity of the file. However, this method ignores the user’s demand for dynamic data update, and there are many difficulties in the design level.

3. Method

3.1. Network Storage Data Acquisition

The network crawler is applied to the network storage data crawling, and thus to provide support for the data integrity detection. As a complete automatic image annotation system, it is necessary to use the crawler technology to automatically obtain the network image and build an image annotation database. On this basis, the web crawler technology is used to capture the image and related text information.

Web crawler design:

The main function of web crawler is to obtain the network data. It mainly uses the hypertext link in web page to roam, find, and collect information in the Internet, so as to provide the data source for the next stage of information extraction, organization, and management. Generally, the crawler starts from an initial URL set and uses some search strategy to traverse the web page and download various data resources along the URL of hypertext link, such as breadth-first strategy, depth-first strategy.

The web crawler system will maintain a URL table, including some original URLs. Based on these URLs, Robot downloads the corresponding pages and extracts new URLs from them and then adds them to the URL table. After that, Robot repeats the above process until the URL queue is empty.

On this basis, the basic framework of web crawler is shown in Figure 1 (source: author own conception, adapted from Wu Libing):

The working process of crawler is as follows:

Import the seed (the initial URL list) and import the agent. Import templates (regular expressions constructed by different web features). The agent is corresponded to the seed URL, and the seed is put into the crawling queue as the initial set of URL queue. Take the waiting URL from the queue, and then enter the crawling state. After that, the web source file corresponding to the URL is downloaded. Transfer the regular expression in the template object array and the downloaded web page source file into the web page analysis module for matching, so that the structured information can be obtained. The important links contained in the structured information, such as URL of the next page of post list and URL of the post, which continues to be put into the URL queue and waits for crawling. Information required by other users in structured information, such as post name, post sender, reply number, and reply content are stored in the information table.

Extraction of image and related text:

Through the research on the page, the text information related to the image mainly includes:

(1) The texts around the image in page, most of them are too long, including a lot of semantic information. During the page analysis, they are mostly related to the page structure, such as the adjacent texts in the same row or column of the table. When an image exists as an illustration, the surrounding words have limited contribution to the annotation on the image.

(2) File name, title, or description information of image are usually concise phrases or words, which have strong generalization ability.

(3) The title of the image link page. The content of image web page is highly relative to image, and some titles which are used to generalize the content of hyperlink web page. Meanwhile, they also have something to do with image semantics.

The image semantics mainly comes from the analysis of related text. Firstly, Chinese characters and English existing in related text are translated into Chinese, and then the automatic word segmentation and the part of speech tagging are carried out.

When the system receives a large amount of text information through the crawler program, the following problem is how to extract the keywords from documents. One of the obvious differences between Chinese and English is that in Chinese text, there is no obvious natural separator between Chinese characters or vocabularies. Meanwhile, the number of Chinese words is uncertain, and the collocation is flexible, and the semantics is diverse. Most of them are composed of two or more Chinese characters, and the writing is continuous, which brings more difficulties for Chinese understanding and keyword extraction.

Generally, users may retrieve in the form of words or single character when querying, so the system should label images with shorter words or single character as much as possible. When the annotation platform receives various long texts, it is necessary to divide the whole sentence into smaller vocabulary units at first, and then process them through the keyword selection module [8].

In Chinese word segmentation, it is necessary to consider the complexity of time on the premise of accuracy [9]. A simple way is the maximum matching method of positive word subtraction. The basic thought is to build a dictionary in advance, and then extract a preset length word string from the long sentences of natural language, and compare it with the dictionary. If the string belongs to the dictionary, it will be regarded as a meaningful word string. Then, the separator is used to split it and output it. Otherwise, it is necessary to shorten the word string and search again in the dictionary. Finally, we should move backward and repeat the above steps. The basic description of this algorithm is shown in Figure 2 (source: author own conception, adapted from Wang Ruilei).

With the distributed crawler as the core, an automatic image download program is implemented, and the main flow chart is shown in Figure 3. It mainly includes the web source file downloading module, regular matching module, next page URL construction and image URL construction module, and image download module whose entry parameter is image URL.

3.2. Data Feature Extraction Based on Symmetrical Difference

In the above process of data acquisition, the automatic word segmentation and pos tagging, Chinese word segmentation are used to achieve the feature analysis of the text data. Then, the symmetrical difference algorithm is taken as the main method and background subtraction is taken as the auxiliary method, so that the image data feature extraction and preliminary recognition of data integrity are achieved. Thus, the precision rate of data integrity detection is improved.

The symmetrical difference algorithm can remove the influence of the background revealed by the motion and draw the contour of the moving object accurately [10,11]. The basic algorithm is as follows: the source images of three consecutive frames in video sequence are $f_{\left(k-1\right)}\left(x,y\right)$,$f_{\left(k\right)}\left(x,y\right)$ and $f_{\left(k+1\right)}\left(x,y\right)$. The absolute difference gray-scale images of two adjacent source images are calculated respectively, namely $d_{\left(k,k+1\right)}\left(x,y\right)$ and $d_{\left(k,k+1\right)}\left(x,y\right)$.

$$\{\begin{array}{l}{d}_{\left(k-1,k\right)}\left(x,y\right)=\left|W\times f_{\left(k\right)}\left(x,y\right)-W\times f_{\left(k-1\right)}\left(x,y\right)\right|\\ d_{\left(k,k+1\right)}\left(x,y\right)=\left|W\times f_{\left(k+1\right)}\left(x,y\right)-W\times f_{\left(k\right)}\left(x,y\right)\right|\end{array}$$

(1)

where, $W$ is a window function to suppress noise. Because the mean filtering will blur the image, leading to the loss of edge information, the median filtering function with 3 × 3 window is chosen to suppress the noise.

The binary images $b_{\left(k-1,k\right)}\left(x,y\right)$ and $b_{\left(k,k+1\right)}\left(x,y\right)$ are obtained by calculating threshold values of $d_{\left(k-1,k\right)}\left(x,y\right)$ and $d_{\left(k,k+1\right)}\left(x,y\right)$ respectively. The binary image $d_S^{(k)}\left(x,y\right)$ of symmetrical difference result is obtained by logic operation of $b_{\left(k-1,k\right)}\left(x,y\right)$ and $b_{\left(k,k+1\right)}\left(x,y\right)$ at each pixel position. The formula is

$$d_S^{(k)}\left(x,y\right)=\{\begin{array}{l}\begin{array}{cc}1,& b_{\left(k-1,k\right)}\left(x,y\right)\cap b_{\left(k,k+1\right)}\left(x,y\right)=1\end{array}\\ \begin{array}{cc}0,& other\end{array}\end{array}$$

(2)

The basic idea of background subtraction is to subtract the current image from the background image stored in advance or real-time background image. The pixel point whose difference is greater than a certain threshold value is regarded as the point on the moving target. Otherwise, it is considered as the background point, which is very suitable for detecting the moving target when the background image changes less with time. By comparing the difference of gray values between the current source image $f_{\left(k\right)}\left(x,y\right)$ and background image $B_k\left(x,y\right)$, the foreground image $d_b^{(k)}\left(x,y\right)$ can be obtained. The formula is

$$d_b^{(k)}\left(x,y\right)=\{\begin{array}{l}1,\left|W\times f_{\left(k\right)}\left(x,y\right)-W\times B_k\left(x,y\right)\right|\ge T\\ 0,other\end{array}$$

(3)

where, $T$ denotes the threshold value.

The ideal background $B\left(x,y\right)=\cup \frac{n-1}{s=0}{B}_{s,s+1}$ is obtained on the basis of the given $N$ frame image, in which $\cup$ denotes the image splice operator and $B_{s,s+1}$ denotes the common background of frame $S$ and frame $S+1$.

Formula (4) is used to judge the attribution of sub block $B_K\left(s,j\right)$,

$$B_K\left(s,j\right)=\{\begin{array}{l}s,s+1\\ \begin{array}{cc}\mathrm{Frame}\mathrm{common}\mathrm{background}& C\left[D_{bk}\left(s,j\right)\right]\le TB\end{array}\\ \mathrm{target}\mathrm{area}other\end{array}$$

(4)

The moving object detection algorithm based on background subtraction and symmetrical difference can accurately extract and update the background model when the image has several moving targets.

3.3. Data Integrity Detection and Accountability Mechanism

Based on the data collection and feature extraction above, the random sentry data segment is introduced to achieve the final data integrity detection. Combined with the accountability scheme of data security of the trusted third party, the trusted third party was taken as the core. The online state judgment was made for each user operation. Meanwhile, credentials that cannot be denied by both parties were generated, so as to ensure the reliability of audit and accountability of trusted third party when the cloud is not trusted. In addition, it is able to prevent the verifier from providing the false verification results.

This scheme randomly selects the sentry data segment to detect the data integrity. Because the sentry data segment contains the selected sentry information, the effective data and other sentry data, this scheme can support the detector to carry out infinite detection, and thus to improve the data integrity detection and recall rate. It is not necessary to worry about the sentry leakage caused by multiple tests.

Next, the scheme flow is introduced according to the data preprocessing stage, the challenge initiation stage and the detection and verification stage.

3.3.1. Data Preprocessing Stage

The original data blocking: firstly, the network storage user runs the key generation formula: $KeyGen\left(p_u\right)\to \left(sKe{y}_u,pKe{y}_u\right)$. Secondly, the public key $sKe{y}_u$ and private key $pKe{y}_u$ held by the user are generated, and the private key $pKe{y}_u$ is saved as a secret pair. After that, the user uses the public key $sKe{y}_u$ to encrypt the original data file: $FileEncrypt\left(F,sKe{y}_u\right)\to \left(F^{\prime }\right)$. Then, the file block algorithm is carried out, $FileBlock\left(F^{\prime },m\right)\to \left\{F_1,\cdots ,F_m\right\}$. $F^{\prime }$ is divided into $m$ blocks. The sizes of blocks are the same. Finally, each data block in $\left\{F_1,\cdots ,F_m\right\}$ set is divided into $n$ blocks: $FileBlock\left(F_i,n\right)\to \left\{{F^{\prime }}_{1i},\cdots ,{F^{\prime }}_{ni}\right\}$. Finally, the data block matrix consisting of $m$ vectors is obtained: $F=\left[\begin{array}{ccc}{F^{\prime }}_{11}& L& {F^{\prime }}_{1m}\\ M& O& M\\ {F^{\prime }}_{n1}& L& {F^{\prime }}_{nm}\end{array}\right]$.

Erasure code: the network storage user performs the erasure code on the obtained data matrix, $ECoding\left(B,{\left\{{F^{\prime }}_{ij}\right\}}_{1\le i<n,1\le j<m}\right)\to A=\left[\begin{array}{ccc}{a}_{11}& L& a_{1m}\\ M& O& M\\ a_{k1}& L& a_{km}\end{array}\right]$. The vandermonde matrix $B^{\prime }$ of erasure correction code is a matrix with $k$ rows and $n$ columns, $\left(k>n\right)$. Its model is shown in Formula (5)

$$B^{\prime }=\left[\begin{array}{ccccc}1& a_1^1& a_1^2& L& a_1^{n-1}\\ 1& a_2^1& a_2^2& L& a_2^{n-1}\\ 1& a_3^1& a_3^2& L& a_3^{n-1}\\ M& M& M& O& M\\ 1& a_k^1& a_k^2& L& a_k^{n-1}\end{array}\right]$$

(5)

In fact, data matrix $\mathrm{A}=\left[\begin{array}{ccc}{a}_{11}& L& a_{1m}\\ M& O& M\\ a_{k1}& L& a_{km}\end{array}\right]$ is obtained by $m$ matrix multiplications, that is to say, each data block set $\left\{{F^{\prime }}_{1i},\cdots ,{F^{\prime }}_{ni}\right\}$ is multiplied by the vandermonde matrix $B^{\prime }$ of erasure correction code in the form of vector, and then these products constitute the data matrix.

$$\left[\begin{array}{ccccc}1& a_1^1& a_1^2& L& a_1^{n-1}\\ 1& a_2^1& a_2^2& L& a_2^{n-1}\\ 1& a_3^1& a_3^2& L& a_3^{n-1}\\ M& M& M& O& M\\ 1& a_k^1& a_k^2& L& a_k^{n-1}\end{array}\right]\times \left[\begin{array}{l}{F^{\prime }}_{1i}\\ {F^{\prime }}_{2i}\\ M\\ {F^{\prime }}_{ni}\end{array}\right]=\left[\begin{array}{l}{a}_{1i}\\ a_{2i}\\ M\\ a_{ki}\end{array}\right]$$

(6)

According to the vandermonde matrix $B^{\prime }$ with $k$ rows and $n$ columns, we can see that there are $\left(k-n\right)\times m$ redundant data blocks in this error correction code. In other words, for data vector ${\left(a_{1i},a_{2i},\cdots ,a_{ki}\right)}^T$, when the maximum loss is $k-n$ data blocks, the entire data vector can be recovered completely.

Sentry position generation: after the data matrix is generated, the network storage user sets the number of sentries which need to be placed in each data block, $l^{\prime }\left(l^{\prime }\ge 2\right)$. Meanwhile, the user uses the sentry insertion position generation algorithm $GuardPosition\left(i,j,l^{\prime }pKe{y}_u\right)\to \left(P_{i,j,1},\cdots ,P_{i,j,w},\cdots ,P_{i,j,l^{\prime }}\right)$ to calculate the position array. $P_{i,j,w}$ denotes the position of the $w$th sentry which needs to be inserted into the data block. ${a^{\prime }}_{ij}$ denotes the data block after inserting the sentry data. $len\left({a^{\prime }}_{ij}\right)=len\left(a_{ij}\right)+q\times l^{\prime }$ represents the bit length of ${a^{\prime }}_{ij}$ after inserting the sentry data. $q$ represents the bit length of each sentry. Thus, the proportion of the actual effective data in data block ${a^{\prime }}_{ij}$ to the total storage data is $len\left(a_{ij}\right)/\left(len\left(a_{ij}\right)+q\times l^{\prime }\right)$.

The generation algorithm $GuardPosition\left(i,j,l^{\prime }pKe{y}_u\right)\to \left(P_{i,j,1},\cdots ,P_{i,j,w},\cdots ,P_{i,j,l^{\prime }}\right)$ of sentry insertion position is to select the random number to hash the bit length of ${a^{\prime }}_{ij}$, namely $P_{i,j,w}=H^w\left(ran\left(i,j\right)\Vert pKe{y}_u\Vert User\_id\right)\mathrm{mod}len\left({a^{\prime }}_{ij}\right)$. The function $ran\left(i,j\right)$ denotes the random number generated by $i,j$. $User\_id$ denotes the unique ID of network storage user. $H(\cdot )$ is the hash function, and $H^1(\cdot )=H(\cdot ),H^2(\cdot )=H\left(H^1(\cdot )\right),\cdots ,H^{l^{\prime }}(\cdot )=H\left(H^{l^{\prime }-1}(\cdot )\right)$. Because $P_{i,j,w}$ is generated by random number, it is necessary to reorder the set of sentry positions from big to small. Finally, the orderly sequence of sentry insertion positions can be obtained [12,13].

Sentry data generation: network storage user uses the array $\left(P_{i,j,1},\cdots ,P_{i,j,w},\cdots ,P_{i,j,l^{\prime }}\right)$ of orderly sentry insertion position and preset length $q$ of sentry to calculate sentry at position $P_{i,j,w}$:$G_{i,j,w}:GuardGen\left(i,j,P_{i,j,w},q,pKe{y}_u\right)\to \left(G_{i,j,w}\right)$.

The sentry generation algorithm $GuardGen\left(i,j,P_{i,j,w},q,pKe{y}_u\right)\to \left(G_{i,j,w}\right)$ is defined as the binary 0/1 string $q$-bit data with the result $H^w\left(ran\left(i,j\right)\Vert pKe{y}_u\Vert User\_id\right)$. At this stage, network storage user needs to take $pKe{y}_u$ and $ran\left(i,j\right)$ as secrets and then save them locally. In addition, they are not disclosed to any other party in the data preprocessing stage.

Sentry insertion: after the network storage user generates the sentry, the sentry insertion algorithm is used to insert the sentry set $G_{i,j,w}$ into the data block $a_{ij}$:

$$Guardinsert\left(a_{ij},P_{i,j,w},G_{i,j,w}\right)\to A\prime =\left[\begin{array}{ccc}{a^{\prime }}_{11}& L& {a^{\prime }}_{1m}\\ M& O& M\\ {a^{\prime }}_{k1}& L& {a^{\prime }}_{km}\end{array}\right]$$

(7)

When calculating the position of sentry $\left(P_{i,j,1},\cdots ,P_{i,j,w},\cdots ,P_{i,j,l^{\prime }}\right)$, we do not consider that the insertion of previous $P_{i,j,w}$ will affect the change of sentry position $P_{i,j,w^{+}}$ behind. When $Guardinsert$ is implemented, it is necessary to move the original position $P_{i,j,w}$ back $w-1$ positions of $q$. After ${P^{\prime }}_{i,j,w}=P_{i,j,w}+\left(w-1\right)\times q$ transformation, we will insert $G_{i,j,w}$ into the data block matrix $A=\left[\begin{array}{ccc}{a}_{11}& L& a_{1m}\\ M& O& M\\ a_{k1}& L& a_{km}\end{array}\right]$ to generate the final confusion data matrix $A\prime =\left[\begin{array}{ccc}{a^{\prime }}_{11}& L& {a^{\prime }}_{1m}\\ M& O& M\\ {a^{\prime }}_{k1}& L& {a^{\prime }}_{km}\end{array}\right]$, and then upload the data matrix to the network server.

Upload parameter to trusted third party: the network cloud storage user uses the public key $sKe{y}_t$ of trusted third party to encrypt the parameters $\left(n,m,k,l^{\prime },q,ran{\left(i,j\right)}_{1\le i<k,1\le j<m},len\left({a^{\prime }}_{ij}\right),pKe{y}_u\right)$ which are used in the data pre-processing stage, $\sec =ParaEncrypt\left(\left(n,m,k,l^{\prime },q,ran{\left(i,j\right)}_{1\le i<k,1\le j<m}\right),len\left({a^{\prime }}_{ij}\right),pKe{y}_u\right)$. After that, we can store them in the trusted third party. In the subsequent data integrity detection, the network storage user authorization can directly use the private key $pKe{y}_u$ to decrypt $\sec$. According to the calculated parameters, the challenge can be sent to the cloud.

3.3.2. Start Challenge

Cloud storage users put forward detection request. When cloud storage users need to detect the integrity of data files stored in cloud server, they will send a detection request to the trusted third party, and then the trusted third party will perform the data integrity detection instead of users.

Detection parameter analysis: after receiving the detection request from the user, the trusted third party checks the user rights at first, so as to judge whether the user has the read permission for the data file. If the applicant has the read permission, the trusted third party uses $pKe{y}_u$ to decrypt the pre-processing parameter set $\sec$ to get $\left(n,m,k,l^{\prime },q,ran{\left(i,j\right)}_{1\le i<k,1\le j<m},len\left({a^{\prime }}_{ij}\right),pKe{y}_u\right)$. The random generation algorithm $RanPos\left(k,m,r\right)\to \left\{\left(i_1,j_1\right),\cdots ,\left(i_r,j_r\right)\right\}$ is performed. Where, $k$ is the number of rows and $m$ is the number of columns of matrix $A\prime =\left[\begin{array}{ccc}{a^{\prime }}_{11}& L& {a^{\prime }}_{1m}\\ M& O& M\\ {a^{\prime }}_{k1}& L& {a^{\prime }}_{km}\end{array}\right]$. $r$ denotes the number of data blocks ${a^{\prime }}_{ij}$ that the trusted third-party plans to detect. According to the detection intensity and detection environment, $r$ can be determined by the trusted third party. If the trusted third party wants to conduct comprehensive data integrity detection, $r$ can be bigger at this time. If the trusted third party wants to conduct periodic data integrity detection, the value of $r$ can be moderate at this time. If the current network fluctuates greatly or the soft and hard environment is lack, the value of $r$ can be reduced [14,15].

The output $I=\left[\left(i_1,j_1\right),\cdots ,\left(i_r,j_r\right)\right]$ of algorithm $RanPos$ denotes the subscript of the selected detection data block. Then, the trusted third party uses the array $I$ and algorithm $GuardPosition\left(i,j,l^{\prime }pKe{y}_u\right)\to \left(P_{i,j,1},\cdots ,P_{i,j,w},\cdots ,P_{i,j,l^{\prime }}\right)$ to generate $r\times l^{\prime }$ sentry positions of $r$ data blocks ${a^{\prime }}_{ij}\left(i,j\right)\in I$. Finally, the algorithm ${P^{\prime }}_{i,j,w}=P_{i,j,w}+\left(w-1\right)\times q$ is used to calculate the real positions of $l^{\prime }$ sentries in data block ${a^{\prime }}_{ij}\left(i,j\right)\in I$.

The generation of detection interval: the trusted third party uses the form of selecting data interval to determine the data range. That is to say, the data interval $\left[x_c,y_c\right],1\le c<r$ is selected from data block ${a^{\prime }}_{ij}$ for final detection of data integrity. The data block ${a^{\prime }}_{ij}$ corresponds to an interval, and the lengths of $r$ intervals $\left[x_c,y_c\right],1\le c<r$ are the same, but their positions are different in corresponding ${a^{\prime }}_{ij}$ (see Figure 4).

The selection of interval length $y_c-x_c$ is similar to the selection condition of the number $r$ of detection data block ${a^{\prime }}_{ij}$, which can be determined by the trusted third party through the detection intensity and detection environment. Meanwhile, the minimum length of interval $\left[x_c,y_c\right]$ should be bigger than the length of $len\left({a^{\prime }}_{ij}\right)/l^{\prime }$. Because $len\left({a^{\prime }}_{ij}\right)=len\left(a_{ij}\right)+q\times l^{\prime }$, $len\left({a^{\prime }}_{ij}\right)/l^{\prime }$ must be bigger than the position length $q$ of sentry. That is to say, the detection interval range must be bigger than the length of sentry, so that the reliability of data integrity detection can be guaranteed, and thus avoiding the sentry leakage caused by multiple detection [16,17,18].

In order to ensure that the data interval $\left[x_c,y_c\right],1\le c<r$ contains the sentry information, the trusted third party randomly selects $r$ random numbers $\left\{u_e\right\},1\le e<r$ from the integer interval $\left[1,l^{\prime }\right]$, and then extracts the sentry position ${P^{\prime }}_{i,j,e}$ of corresponding random number $\left\{u_e\right\},1\le e<r$ from the sentry position set $\left(P_{i,j,1},\cdots ,P_{i,j,w},\cdots ,P_{i,j,l^{\prime }}\right)$ of data block ${a^{\prime }}_{ij}$ corresponding to $I=\left[\left(i_1,j_1\right),\cdots ,\left(i_r,j_r\right)\right]$. That is the aggregation $\left({a^{\prime }}_{i_1,j_1}:{P^{\prime }}_{i_1,j_1,u_1},\cdots ,{a^{\prime }}_{i_w,j_w}:{P^{\prime }}_{i_w,j_w,u_w},\cdots ,{a^{\prime }}_{i_r,j_r}:{P^{\prime }}_{i_r,j_r,u_r}\right)$. See Figure 5:

The length of detection interval the trusted third party is set as $\gamma$. In order to put the sentry ${a^{\prime }}_{i_w,j_w}:G_{i_w,j_w,u_w}$ corresponding to the set $\left({a^{\prime }}_{i_1,j_1}:{P^{\prime }}_{i_1,j_1,u_1},\cdots ,{a^{\prime }}_{i_w,j_w}:{P^{\prime }}_{i_w,j_w,u_w},\cdots ,{a^{\prime }}_{i_r,j_r}:{P^{\prime }}_{i_r,j_r,u_r}\right)$ in the detection interval $\left[x_c,y_c\right],1\le c<r$, the distance $dis=random\left(0,\delta \right)\times \gamma ,0<\delta \le 1$ between ${a^{\prime }}_{i_w,j_w}:{P^{\prime }}_{i_w,j_w,u_w}$ and the left vertex of interval is defined. $\delta$ denotes the basic threshold which is randomly generated at $dis$. If $\delta$ is close to 1, the probability that the length of $dis$ is close to $\gamma$. ${a^{\prime }}_{i_w,j_w}:{P^{\prime }}_{i_w,j_w,u_w}$ is far away from the left vertex of interval with the increase of $dis$, and the probability that sentry in the interval is incomplete.

In the detection interval $\left[x_w,y_w\right]$ of data block ${a^{\prime }}_{i_w,j_w}$, the sentry $G_{i_w,j_w,u_w}$ is uncertain, so the number of sentries in an interval may be uncertain. A small number of sentries may only be included in some information [19,20]. The situation of sentry in interval is shown in Figure 6. (1) shows that sentry $G_{i_w,j_w,u_w}$ is located in the interval $\left[x_w,y_w\right]$ completely, namely $x_w\le {P^{\prime }}_{i_w,j_w,u_w}<{P^{\prime }}_{i_w,j_w,u_w}+q-1<y_w$. (2) shows that only partial data of sentry $G_{i_w,j_w,u_w}$ is located in the interval $\left[x_w,y_w\right]$, namely $x_w<{P^{\prime }}_{i_w,j_w,u_w}\le y_w<{P^{\prime }}_{i_w,j_w,u_w}+q-1$. Due to the uncertainty of the length $\gamma$ of interval, this interval may include several sentries of position $w$ in addition to sentry $G_{i_w,j_w,u_w}$, which is shown in (3) and (4) of Figure 6.

Start challenge request: the trusted third party encrypts the array $I=\left[\left(i_1,j_1\right),\cdots ,\left(i_r,j_r\right)\right]$ of the serial number of detected data block and the array $Q^{\prime }=\left[\left(x_1,y_1\right),\cdots ,\left(x_r,y_r\right)\right]$ which is composed of detection interval $\left[x_w,y_w\right]$ corresponding to serial number $\left(i_w,j_w\right)$ through the cloud public key $sKe{y}_c$ and then sends them to the remote server as the challenge request $chal$.

3.3.3. Detection and Verification Stage

Evidence generation: after receiving the challenge request $chal$, the server uses the private key $pKe{y}_c$ to decrypt $chal$, then uses the decrypted $I$ and $Q^{\prime }$ to perform the evidence generation algorithm $Gen\mathrm{Pr}oof\left(I,Q^{\prime }\right)\to {\left\{i,j,\mathrm{Pr}o\right\}}_{\left(i,j\right)\in I}$, so as to output the credentials $\mathrm{Pr}o$ which is used to submit the trusted third party for data integrity detection. Finally, the public key $sKe{y}_t$ of trusted third party is used to encrypt $\mathrm{Pr}o$ and return it to the trusted third party.

Evidence verification: after receiving the detection credentials $\mathrm{Pr}o$ sent from the cloud, the trusted third party uses the evidence validation algorithm $Check\mathrm{Pr}oof\left(I,Q^{\prime },P_{i,j,w},q,pKe{y}_u,\mathrm{Pr}o\right)\to \left\{"Su","Fa"\right\}$ to judge whether the data file is complete. The detailed process is as follows: the trusted third party locates the $\theta$ sentry positions included in the detection interval $\left[x_w,y_w\right]$ according to the sentry position array $\left(P_{i_w,j_w,1},\cdots ,P_{i_w,j_w,z},\cdots ,P_{i_w,j_w,l^{\prime }}\right)$ of data block ${a^{\prime }}_{i_w,j_w}$, namely $x_w+1-q\le {\left\{{P^{\prime }}_{i,j,m}\right\}}_{1\le m<\theta }<y_w$. Then, the trusted third party calculates the detailed information of sentry by the sentry generation algorithm $GuardGen\left(i,j,P_{i,j,w},q,pKe{y}_u\right)\to \left(G_{i,j,w}\right)$. By comparing with the corresponding position in detection evidence $\mathrm{Pr}o$, we can see that only partial data appears in the sentry of detection area. If $R^{\prime }$ data segments in $\mathrm{Pr}o$ are compared, it means that the data has not been modified or damaged in this time. Otherwise, it means that the data stored in remote server is incomplete. Finally, the trusted third party submits the data integrity detection result to cloud storage user.

(4) Data security accountability based on trusted third party

With the popularization and development of cloud storage, people pay more and more attention to the security of cloud data. When the data stored in the cloud is illegally modified, neither the user nor the cloud can provide cogent credentials to divide the responsibility. Therefore, a data security accountability scheme based on trusted third party. This scheme takes the trusted third party as the core and carries out the online status judgment in each user operation, and then the credentials that cannot be denied by both parties are generated, so as to ensure the reliability of the audit and accountability of trusted third party when the cloud is not trusted.

A credible third-party accountability system needs to provide the following functions:

Any operation on cloud data is recorded in trusted third party and cloud;

When disputes occur, the trusted third party can accurately find the responsible party and determine the responsibility;

The certificate that is used to judge the responsible party has non repudiation.

Figure 7 is the framework of data security accountability based on trusted third party

In Figure 7, the user operates the cloud files through the browser or other clients. When the users log in to the cloud through the browser or other clients in each time, they can get a temporary token from the trusted third party. The file key version is formed after each user operation. When the user and the cloud operate the data, the accountability voucher will be generated and saved in the table of trusted third-party voucher. When the accountability is proposed or some disputes between the two parties occur, the trusted third party can judge the responsibility based on the clouding file operation records and local vouchers.

4. Results

In the process of verifying the performance of the network storage data integrity detection method based on symmetric difference, the experimental environment was 1 PC and the operating system was Windows 10. The installed memory (RAM) is 16 GB and the processor is AMD Ryzen Threadripper [email protected]. The disk size is 250 GB and the disk read/write speed is 5400 RPM. Java language is adopted. In this experiment, the text data and image data are randomly crawled in the network storage database by the above method, so that the feasibility of the proposed method is verified by the precision rate of data integrity detection.

Figure 8 shows the precision rate of data integrity detection.

In Figure 8, the overall precision rate of the proposed method is high, showing good performance. This method achieves the feature analysis of text data by automatic word segmentation, part of speech tagging and Chinese word segmentation. Based on the symmetrical difference algorithm and the background subtraction, the feature extraction of image data is realized. Meanwhile, the preliminary recognition of data integrity is achieved, and thus to improve the precision rate of data integrity detection.

Three different methods are used to further verify the data integrity. In this paper, the availability of network storage data is verified, and the results are as follows.

Analysis of the above Figure 9 shows that the availability of different methods is different under different data volumes. When the data volume is 5 GB, the data availability rate of literature [4] method is 72%, the data availability rate of literature [5] method is 82%, the data availability rate of this method is 94%, while the data availability rate of this method is significantly higher than the other two methods, and the data integrity is better.

5. Discussion

The recall rate of data integrity detection is used as the discussion index to analyze the recall performance of the proposed method. The results are shown in Figure 10.

Figure 10 show that the proposed method uses the random sentry data segments to detect the data integrity. Because the sentry data segments not only contain the selected sentry information, but also contain the effective data and other sentry data, the scheme can support the infinite detection of detection party, and thus to improve the recall rate of data integrity detection.

In order to further verify the performance of this method, three different methods of coding time are used for detection, and the results are as follows.

Analysis of Figure 11 shows that different methods have different encoding times, that is to say, the shorter the encoding time, the higher the efficiency. When the amount of data stored is 14 MB, the coding time of document [4] method is 8 s, that of document [5] method is 12 s, and that of document [4] method is 5 s. The coding time of this method is the shortest and the efficiency is higher.

6. Conclusions

In order to solve the problem about network data security and integrity, a method to detect the network storage data integrity based on symmetric difference is put forward. According to data acquisition, feature analysis and extraction, the sentry data segment is introduced to detect the data integrity. The research results and work of this paper can be summarized as follows:

(1) Through in-depth study of data feature extraction and data integrity technology, this paper proposes a method of network storage data integrity detection based on symmetric difference. In this method, the integrity of the network storage data is detected according to the automatic image annotation system. For data recovery, this paper provides a strong anti-corruption ability by two rounds of coding for the original file. In the face of large area and high frequency file damage, it can still recover the original file with a high recovery rate, providing a high data availability. Every time the data recovery algorithm is called, this method is better.

(2) This paper presents an efficient data recovery algorithm. After locating the faulty storage nodes, it is necessary to recover the faulty data on these nodes. The algorithm ensures the recovery of data with low communication overhead and high recovery rate, and provides better security for network storage data.

(3) The integrity detection method proposed in this paper supports not only static data detection, but also dynamic data detection.

In this paper, the integrity detection method of network storage data based on symmetric difference is studied, and the phased research results are obtained. However, due to the limited time and energy and the criss-cross problems involved, the work done in this paper will inevitably be insufficient, which needs further improvement and improvement, and further research in the future. With the continuous growth of data volume, the development and expansion of trusted third party has become the next problem to be solved, and fault tolerance is also a very important research focus.