A Method for Vulnerability Database Quantitative Evaluation

: During system development, implementation and operation, vulnerability database technique is necessary to system security; there are many vulnerability databases but a lack of quality standardization and general evaluation method are needed. this paper summarized current international popular vulnerability databases, systematically introduced the present situation of current vulnerability databases, and found the problems of vulnerability database technology, extracted common metrics by analyzing vulnerability data of current popular vulnerability databases, introduced 4 measure indexes: the number scale of vulnerabilities, the independence level, the standardization degree and the integrity of vulnerability description, proposed a method for vulnerability database quantitative evaluation using SCAP protocol and corresponding standard, analyzed a large number of vulnerabilities in current popular vulnerability database, quantitative evaluated vulnerability database by the law of normal distribution, the experimental results show this method has strong versatility and science, and it is beneficial to improve the quality and standardization construction for vulnerability database development.

mechanisms, it is not realistic to completely eliminate vulnerability ; ], once exploited by an attacker, the damage and loss is difficult to repair, vulnerability database is a basic tool for vulnerability management [Ou, Hu, Zhang et al. (2007)], it can provide comprehensive functions, such as collection and release of vulnerabilities, vulnerability description, etc., it is the primary technology in the information security field. However, in development of current vulnerability database technology, there are some problems: • There are a lot of popular vulnerability databases with different characters, such as data scale, data source, standardization and integrity degree, it is a lack of a survey of vulnerability databases research to systematically summarize and introduce the characters of current popular vulnerability databases. • Due to various network equipment manufacturers, Internet companies and research institutions, in different vulnerability databases, the same vulnerability may have different release time and description data structure, heterogeneous data structures in different vulnerability databases prevent standardization construction and data sharing from each other. • Different vulnerability databases have different quality; there is a lack of a common evaluation method to evaluate the quality of vulnerability database, a common method for vulnerability database quality quantitative evaluation need to be proposed. To solve above problems, on the basis of summarizing a large number of vulnerability data of current popular vulnerability databases, this paper proposed a vulnerability database evaluation method to evaluate the quality of vulnerability database. The contributions of this paper are as follows: • This paper systematically summarized all popular vulnerability databases at home and abroad, systematically introduced the details of current popular vulnerability databases. • Based on SCAP protocol, this paper proposed vulnerability database evaluation method, analyzed a large number of vulnerability data in popular vulnerability databases, extracted the major features, such as data scale, the source independence level, integrity and standardization degree of vulnerability data as 4 measure indexes to quantify and grade the vulnerability database. • The evaluation method using normal distribution quantitative evaluated current popular vulnerability databases, the result can show that it is beneficial to regulate the vulnerability database construction and operations, promotes the quality standard construction of vulnerability database and provides a reference for vulnerability database standardized construction. Compared with method which needs to set fixed value for measure index, this method has the advantage of keeping high accuracy with the development of the vulnerability technology.  [Wang, Guo, Wang et al. (2009)].  The vulnerability entries of NVD vulnerability database are more than 80,000. NVD has its own us-cert vulnerability announcement, us-cert security warning, OVAL information and CPE information, NVD is one of the most complete vulnerability databases in the world. The NVD and CVE [NCVERC (2018)] vulnerability databases are compatible with each other and NVD contains all the vulnerability data of CVE database. In the NVD vulnerability database, the vulnerability information can be searched according to the CVE id. Therefore, the NVD vulnerability database has good universality. NVD mainly focuses on the vulnerabilities in system and protocol layer rather than Web vulnerabilities are less. The NVD vulnerability database adopts SCAP standard protocol. Each vulnerability has 15 fields, includes the CVE id, vulnerability title, vulnerability description, CVSS score, risk level, release date, update date, exploit method, risk type, reference, affected version and platform. Because of long time for auditing vulnerabilities, the timeliness of NVD is obviously insufficient.

CVN
In 1998, the U.S Defense Advanced Research Projects Agency (DARPA) at Carnegie Mellon University's software engineering institute set up Computer Emergency Readiness team/coordination center (CERT/CC) to collect and publish the Internet security incidents and security vulnerabilities, provide safety techniques, security update advice and safety emergency response. CERT/CC established CERT Vulnerability Notes (CVN). CVN has an authoritative data source, and proposed the risk metric method of vulnerabilities. However, CVN has only one data source, the number of vulnerabilities is not large enough, and the vulnerability data is not comprehensive enough.

CNVD
China National Vulnerability Database(CNVD) is constructed by China Research Center for Information Technology Security and CNCERT/CC, CNVD has rich vulnerability resource, the number of vulnerability entries is over 90000, its vulnerability identification form is: CNVD-YYYY-NNNN, each vulnerability information record 14 properties, risk evaluation has 3 grades: high, medium and low, update delay is 1-2 days.

NIPC Security vulnerability database (NIPC) is constructed by National Computer Virus Emergency Response Center (NCNERC), Anti-Virus Products Testing and Certification
Center, and Key Laboratory of Computer Network and Information Security Ministry of Education. The NIPC vulnerability database contains nearly 90,000 vulnerability entries which are in form of nipc-yyyy-nnnnn, and each vulnerability entry has 19 attributes. The risk evaluation has 3 grades: high, medium and low, and the update delay are 1-2 days. By studying relevant standards of vulnerability database and fusion algorithm for heterogeneous vulnerabilities, the vulnerability information of NIPC has a high exploit rate.

SCAP
The SCAP vulnerability database integrates the vulnerability data from a large number of vulnerability databases and some corresponding standards of security vulnerabilities, including NVD, OSVDB, Securityfocus, Packet-Storm, CNNVD and SCAP standards, etc. Through in-depth analysis, the security vulnerability data sharing and security services are established. SCAP is a vulnerability information sharing platform which provides vulnerability information query services. Vulnerability information includes detailed information and partial proof of concept (POC). SCAP introduced the SCAP standard in detail, such as CCE, CWE [TU (2018)], OVAL, Android special vulnerability databases [Yang, Wen and Zhang (2015)], and reclassified the vulnerabilities according to the structure level of Android system.

Enterprise vulnerability databases
For the propose of collecting the vulnerability information of the corresponding business system and establishing an emergency response center to minimize the loss caused by the vulnerability exploit, improving their products, sharing and trading of vulnerabilities, etc.. As shown in Tab

SecurityFocus
The SecurityFocus vulnerability database established by Symantec Company contains over 90,000 vulnerability entries. Its major feature is that the vulnerability information not only includes a brief description, but also contains many details, such as the attack method, script instance and other contents provide convenience for analyzing the vulnerability. Compared with the government vulnerability database such as NVD, the vulnerability release approach of SecurityFocus is more convenient and timelier; it has a greater international influence. However, SecurityFocus has some deficiencies in the processing of vulnerability data. It lacks standardized systematic vulnerability classification and authoritative vulnerability risk assessment.

X-Force
X-force vulnerability database updates data timely, through the web site xforce.iss.net, the majority of users can query the vulnerability information. Relying on the product platform of IBM, the x-force vulnerability database has been transformed into security products such as security scanner, and it is one of a few vulnerability databases that can transform security vulnerability data into security services.

EDB
The EDB database is a security vulnerability database developed and maintained by OffensiveSecurity, which provides vulnerability query services for free. EDB uses CVE id to identify vulnerabilities and provides verification code for a large number of vulnerabilities, it has great influence in the security field. The deficiencies of EDB are mainly reflected in the lack of natural language description of vulnerabilities, the classification and risk assessment of vulnerabilities.

NSFocus
NSFocus contains 36,000 vulnerability entries, and provides users with security scanning and protection services based on a large amount of vulnerability data.

Seebug
Seebug vulnerability information is released through manual processing, and the vulnerability data is authoritative. In addition, over 80% vulnerability entries provide proof of concept (POC) which brings convenient to security researchers to study the vulnerabilities.

Summary
The information of 15 international popular vulnerability databases is shown in Tab. 3, the details includes country, language, the number of vulnerabilities, proportion of vulnerabilities with CVE id, CVSS and CWE, etc. The vulnerability database with largest vulnerability number is OSVDB. It is shown that the number of vulnerabilities and fields in each vulnerability databases are different, only a few vulnerability databases contain proof of concept (POC), many vulnerability databases use the CVE id.

Quantitative evaluation method based on SCAP protocol
The Security Content Automation Protocol (SCAP), designed by the United States National Institute of Standards and Technology (NIST) [Mell and Grance (2002)], is a complete and mature mechanism for standardized vulnerability assessment, and its major feature is the standardized and automated architecture. SCAP integrates six methods: CVE, CVSS [Grance, Kuhn and Landau (2007)], OVAL, CPE [NISTCPET (2014); Zhang, Wu, Liu et al. (2011)], CCE, XCCDF. Standardization is the major advantage of SCAP, SCAP provides the solution for the field of security standardization, including input and output data format, standard processing method, uniform field and risk level measurement, it can automatically audit complex system configuration, and improve the degree of versatility and automation.
To better grasp the latest progress in the field of vulnerability databases research and implementation, this article analyzed popular vulnerability database from the aspect of operating agency, data features, operation, etc., proposed a metric for vulnerability database evaluation, extracted the vulnerability data scale, the data independence level, data standardization level, and integrity level as four measure indexes, it can provide the theoretical basis for rapid, comprehensive and accurate evaluation of vulnerability databases, and help to improve vulnerability database system, promote the development of vulnerability database technology.

Vulnerability database scale (VD) and data source independency level (SIL)
To a large extent, the number of vulnerability entries can reflect the vulnerability database's scale and how many CWE types the vulnerability database has. The independence of vulnerability data source can represent the viability of a vulnerability database. Currently, referring to each other has become a common phenomenon for vulnerability databases, the more independency the vulnerability data source has, the less data refers from other vulnerability database and the more vulnerability entries are obtained through its own way.

Vulnerability data integrity level (DI)
The more effective descriptive fields reflect more comprehensive vulnerability information, so number of effective descriptive fields can be used as an index to measure data integrity. Common fields include CVE id, vulnerabilities name, release and update time, risk level, classification, the affected version and platform, reference links, proof of concept (POC).
POC can greatly enhance the description ability, improve the efficiency of vulnerability analysis, it can attract more accession and more POC submission to form a benign circulation, improve the viability of the vulnerability database, therefore, having POC fields or not is an important standard of influence, and should be used as an index of integrity measure index. In addition, cross-referencing data among international popular vulnerability databases has become a common phenomenon. Data source statement can improve the data integrity of vulnerability databases; therefore, data source statement should also be used as an index to measure data integrity.

Vulnerability data standardization Level (DSL)
The standardization degree of vulnerability data represents the scientific and rationality of the design of vulnerability database data structure. A more standardized vulnerability database is usually designed according to the corresponding international standard database, standardized data structures can facilitate data fusion and sharing among different vulnerability databases.

SCAP-based quantitative evaluation grading method
The vulnerability database scale (VD), data source independency level (SIL), data standardization level (DSL) and data integrity (DI) can be taken as four measure indexes of vulnerability database metric, the data of 4 indexes satisfied the normal distribution rules after statistical analysis, therefore, it can be quantified according to the normal distribution equation: where is the expectation, and is the variance, is the specific value of measure index. The quantitative rules of each measure index are as follows: • High(H). ∈ [ + 2 , + ], Grade is 3, score is 3. • Middle(M). ∈ [ − 2 , + 2 ), Grade is 2, score is 2.
∈ [ , − 2 ), Grade is 1, score is 1. The overall evaluation score of the vulnerability database is calculated on the basis of the obtaining of four scores. The equation for calculating overall evaluation score is as the following equation:
Compared with the method of setting fixed reference value, the main advantage of this method using normal distribution rules can avoid the loss of accuracy with the development of vulnerability database technology.

Experiment and evaluation
This paper selected 15 international popular vulnerability databases for experiments, and obtained statistic results of vulnerability database scale, data source independence level, data standardization level, data integrity evaluation score to calculate the vulnerability database evaluation score.

The vulnerability database scale measurement results
The statistical results of the number of vulnerability entries are shown in Fig.1. Among them, OSVDB has the largest number of vulnerabilities, the databases with over 50,000 vulnerability entries includes OSVDB, x-force, CNNVD, CNVD, SecurityFocus, SCAP Chinese, NVD, NIPC, Secunia and Seebug. According to the normal distribution rules, the vulnerability database scale evaluation result is as Tab. 4.

The vulnerability data source independence measurement results
Statistic results of the number of popular vulnerability databases with CVE id is as shown in Fig. 2, The vulnerability data source of Wooyun, PacketStorm and Seebug have high independence level, while vulnerability entries of SCAP Chinese are referenced from other vulnerability databases. According to the normal distribution rules, the evaluation results of data source independence is shown as Tab. 5.

Figure 2:
The number of popular vulnerability databases with CVE id

Vulnerability data integrity measurement results
The statistic results of the number of fields in the popular vulnerability database are shown in Fig. 3. Most of popular vulnerability databases are more than 10 vulnerability description fields, such as NIPC, OSVDB, x-force, CNVD, Secunia, NVD, SecurityFocus, CXSecurity, NSFocus and SCAP Chinese. Only SecurityFocus, EDB, CXSecurity, PacketStorm and Seebug have POC field. At present, most of the vulnerability databases are short of copyright statement to descript the vulnerability data source. The popular vulnerability databases with copyright statement are shown in Tab. 6. The vulnerability data integrity measurement results are shown in Tab. 7.

Data standardization level measurement results
The statistic results of data standardization degree of vulnerability databases are shown in Tab.7. NVD, SCAP Chinese, CNNVD and NIPC have over 90 percent in the average coverage rate of SCAP protocols, and relatively high data standardization degree. According to normal distribution rules, the data standardization degree measurement results are shown as Tab. 8.   The result is shown as Tab. 9, although NVD and SCAP Chinese are the vulnerability databases with highest quality, both of them need to improve scale and integrity level. Every vulnerability database has its advantages which are found in its higher scores, and the advantages can provide guidance to other vulnerability databases for improvement. Every vulnerability database has its weakness which is found in its lower score, it needs to be improved for quality elevation.

Conclusion
With the rapid development of information technology, network security has become a hotspot of information technology. Vulnerability technology is the foundation of network security and occupies the primary position of network security research. Vulnerability database provides a feasible mechanism for vulnerability management and is one of the most important technologies in network security research. This paper systematically summarized current popular vulnerability databases, analyzed the characteristics of the current popular vulnerability databases, systematically introduced the scale, data source independence level, standardization level and data integrity of the vulnerability database, and proposed a method for vulnerability database quantitative evaluation based on the SCAP protocol, analyzed a large number of data in current popular vulnerability databases, extracted the scale, data source independence level, standardization and integrity level of the vulnerability database as four measure indexes to quantitative evaluate vulnerability database, the experiments proved that this method can quantitative evaluate vulnerability database scientifically and comprehensively, it is helpful to improve the quality of the vulnerability database, and promote vulnerability database standardization construction.