Harmonization of Product Classifications: A Consistent Time Series of Economic Trade Activities

1 Introduction

Innovation and international trade are considered a major engine of economic growth (Aghion et al. 2021; Wagner 2019). Innovation can take a variety of forms, including product, production or delivery, organizational, marketing or communication innovation (Gault 2018). When an economist thinks of innovation, product innovations, like (autonomous) automobiles, computers, smartphones, robots, or drones, come first to mind. We focus on enabling investigations concerning product innovation and firm-product relations in general. For the purpose of analyzing product innovation, studies mostly use the Community Innovation Survey (CIS)^[1] (e.g. Schubert and Tavassoli 2020). CIS is considered one of the richest sources of firm innovation data. Researchers use CIS data to investigate characteristics of firms introducing product innovations, like research and development (R&D) expenditures, firm size, firm age, industry, or profitability (e.g. Audretsch and Belitski 2020; Coad et al. 2016; Griffith et al. 2006; Mairesse and Mohnen 2010). CIS data is also used to conduct policy evaluations of large public R&D grants or public procurement for innovation on the firms’ innovation output (Mairesse and Mohnen 2002; Stojčić et al. 2020). Although the availability of CIS data has had a major impact on the field of innovation economics, CIS data suffers from at least two weaknesses. Firstly, only a small sample of relatively larger firms is collected each cycle, and secondly, variables are collected via survey, thus, they are self-reported by the firms and can be imprecise or suffer from a sample selection bias. There are also other weaknesses, such as the documented difficulties in merging CIS microdata with regional identifiers (Hauser et al. 2018).

Transaction data from customs (Combined Nomenclature, CN) and the Prodcom data can be used to investigate product innovation or firm-product relations in general (European Commission 2019, 2021). Research questions concerning determinants of firms’ changes in productivity over time, firms’ product mix strategy, firm markups, R&D grants impact on the number of newly exported products, or extensive and intensive margins of trade are just few examples that can be analyzed with firm-product data.^[2] These data provide the value and quantity of imported, exported, or sold products at a very detailed product level and enable the analysis of large firm samples. In addition, Prodcom data include a substantial part of all domestically traded products. Questions related to industry-level importance of innovative products can also be investigated using transaction and Prodcom data. Identified new products can be classified into new-to-firm and new-to-market products, combined with the value of these sold or exported products providing additional insights into the magnitude of innovation. Also, these data enable to consider the distance between exporter and destination market (Mayer and Zignago 2011), the type of exports (Wagner 2019), and frequency of simultaneous exports and imports (Bernard et al. 2018). Castellani and Fassio (2019) is one of the few innovation studies using firm-product-market level data.

Consistent time series of product codes are necessary for working with transaction and Prodcom data. However, several product codes are updated annually. Such updates include new product codes, dropped product codes, or product codes merged into a common code. Considering such changes harmonizing the product codes for a given period is mandatory. We contribute to the literature on the importance of harmonizing data to mitigate internal data validity issues (Bellert and Fauceglia 2019; Duprez and Magerman 2022; Liao et al. 2020; Pierce and Schott 2009, 2012; Van Beveren et al. 2012).^[3] Such harmonization is not only an important topic for product codes. Link (2020) demonstrated the necessity of harmonization for ifo Business Survey (IBS) data. Previous approaches to harmonize product codes are based on the idea of Pierce and Schott (2009, 2012, implemented in Stata or R, and are more or less flexible in terms of selecting the period and classification systems (and also with respect to software and programming skills required for users). We briefly explain the development of the preliminary work to better delineate our approach. Pierce and Schott (2009, 2012 suggested an approach to harmonize the international harmonized system product codes (HS) and provided Stata code to enable researchers to prepare the data consistently for further analyses. Van Beveren et al. (2012) picked up their idea and extended their Stata code from concording HS to CN8 and Prodcom codes. Duprez and Magerman (2022) updated the Stata do-files available from 1988 until 2010 up to 2014.

When using programming code in the form of do files or script files, adjustments to other periods, classification systems and program environments require additional programming effort and this may be error prone. Bellert and Fauceglia (2019) were the first to provide a Stata ado-file for various classification schemes without necessary internal code changes. Not having to intervene in the program structure for updates or change requests was also our objective for the implementation in R. Our approach offers new features and add-ons in several respects. In addition to the harmonized product codes, the display of the development of all product codes over time is the core of our approach. This allows the entire history of merged, split, dropped and new product codes to be easily tracked and interpreted. Also, special cases are marked, such as product codes that remained the same during the observation period, but to which other product codes have been added. We provide so-called utilize functions to count the number of added, kept, and dropped products of each firm from one year to the next. We provide the broad economic categories (BEC) (United Nations 2016) and the System of National Accounts (SNA) classification (European Communities et al. 2009). All necessary calculations are included in an open-source R package that is easy to use and can be adapted for different periods (or updated product code change lists) without internal code changes. The approach shows the links between different product codes over time and allows working with consistent time series for the selected period. The importance of harmonization is demonstrated by applying our approach on Croatian data. We obtain a similar substantial discrepancy between the raw time series and the harmonized time series as Van Beveren et al. (2012) for Belgian firm-product data.

In the remainder of the manuscript, we describe the classification systems, present the idea of harmonization, and demonstrate the importance of harmonizing product codes by applying the approach to firm-product data from Croatia. Finally, we summarize the main features of the approach.

2 Harmonization of Product Codes

Harmonization is based on classification lists and concordance tables provided by Eurostat. There are several classification systems used in the European Union (EU) for trade and production. Although the systems are designed to be similar, there are important differences between the two systems, both at a given time and across several years.

2.3 Classification by Broad Economic Categories

The broad economic categories (BEC) system classifies products into broad economic categories used internationally. This classification was introduced in 1971 and has undergone several revisions, with the latest revision, number 5, in 2016. Today, its development and maintenance are done by the United Nations Statistics Division and is used worldwide. Since the most recent revision, BEC codes have eight categories and contain up to six digits instead of three as they used to. The leading digit indicates the main economic class of the product. The second digit distinguishes between goods and services within the eight main categories. Moreover, BEC codes can be classified into three basic classes defined by the System of National Accounts (SNA), which focus on the end-use of the product. The classes are called “capital goods”, “intermediate goods”, and “consumption goods” in revision 4 of BEC, while they are called “intermediate consumption”, “gross fixed capital formation”, and “final consumption” since revision 5. The SNA classification is the third digit in the BEC codes. The last three digits are the “processing dimension”, the “specification dimension” and finally, the “durability dimension” (for an example, see Table 3).

Table 3:

Example of a BEC code for Rev. 5.

Classification system	Digits	Code	Description
BEC	1	6	ICT, media, computers, business and financial services
BEC	2	61	Goods
BEC	3	611	Intermediate consumption
BEC	4	6112	Processed
BEC	6	611220	Specified

1. Source: United Nations (2016).

Concordance files between HS6 and BEC Rev. 4 exist for 1996, 2002, and 2007. For 2012 and 2017, there exists a concordance between HS6 and BEC Rev. 5. Therefore, we provide BEC codes from Rev. 4 until 2011 and BEC codes from Rev. 5 thereafter. Figure 1 illustrates the connection as well as the availability of concordance files between the classification systems described in this paper.^[4] Recently (simultaneously to our approach^[5]) Duprez and Magerman (2022) provide concordance files for CN8 and PC8 on github for 2002–2014.^[6] By default, we use the original concordance files provided at the RAMON server.

Figure 1:

Connections between the different classification systems.

2.4 Harmonization

The basic idea behind harmonization is to keep track of every product code during a specific period. In the simplest case, a code does not change during the examined period, i.e. no harmonization is needed. In any other case, all changes associated with a specific code must be considered. There are different product code changes 1-to-1, 1-to-many, many-to-1, many-to-many, 1-to-none, and none-to-1. The last two changes indicate that a code was dropped or a new code was created. A 1-to-many or many-to-1 change can occur if two or more codes are split or merged.

Figure 2 visualizes possible product code histories. Diamonds in the figure indicate that product codes change in a given year. In case (I) code 01031000 does not change, and thus this CN8 code is the harmonized code too. Case (II) shows a simple code change, i.e. a renaming of the product code, and thus the former product code 01019090 becomes simply 01019000 (= harmonized code). The remaining lines show more complex changes. Cases (III) and (IV) demonstrate a merge of product codes 31056010 and 31056090 in 2011. They both are assigned at the end of the period the family code f1 (= harmonized code). The family code f2 in the cases (V) and (VI) results from a split of code 29309085 into 29309060 and 29309099 in 2010. One should note that a mixture of the cases ((II) until (VI)) is possible, resulting in another family code. The final artificial family code is denoted as “f” with a running number. Therefore, the largest number indicates the number of family codes.

Figure 2:

Example for real CN8 code changes for the period 2008–2013.

All changes above occur for CN8, HS6, and Prodcom product codes’ changes over time. Prodcom codes can be generated or dropped within the period of interest. If changes in these classifications are not taken into account, erroneous entries and exits of products, price bias, or incorrect price and quantity indices result. The harmonized product code for all product codes is provided in a separate column (CN8plus, PC8plus, and HS6plus, cf. Figure 3).

Figure 3:

Harmonized CN8 product code (CN8plus) and history of product codes from 2009 until 2012 as output by the package.

In contrast to the approaches that provide harmonized product codes (family codes) as output (Bellert and Fauceglia 2019; Duprez and Magerman 2022; Pierce and Schott 2009, 2012; Van Beveren et al. 2012), our approach provides, in addition, the entire history of each product code (CN8, Prodcom) in concordance with the HS6, BEC, and SNA classifications. The history matrix enables an easy-to-understand output (Figure 3) and allows the researcher to track splits and mergers and to understand the substantive implications.

In comparison to the previous product harmonization approaches (Bellert and Fauceglia 2019; Duprez and Magerman 2022; Pierce and Schott 2009, 2012; Van Beveren et al. 2012), our approach has further add-ons. (i) A product code can merge with another product code but remains the same product code afterward. These merged product codes belong by default to the same family. They are marked by a flag equal to 1 (cf. Figure 3) that allows the researcher to investigate the history and to decide whether to use the original product code or the family code. This choice has consequences, for example regarding the possibility to link HS6, BEC, or SNA classifications. The “flagyear” column has values only for flag values equal to 1, 2 or 3 and indicates the year in which the first change occurred. In this case, there was no change to this specific product code, so the value of the “flagyear” column corresponds to the last year of the period. (ii) New or dropped product codes are indicated by a flag equal to 2. A dropped product code, for example, may be linked to other product codes (due to splits or mergers) and is therefore assigned a family code. At the firm level, the drop of such a family code is interpreted as a product dropped by the firm, although it may be due to a product code dropped by Eurostat or the national Bureau of Statistics. The indicator (flag = 2) therefore makes it possible to remove the dropped (or also the new) product code from the family code. This prevents misinterpretations. The feature is particularly advantageous for Prodcom codes. (iii) Simple code changes, i.e. product codes have received unique but different product codes over time, are not marked as a family (in contrast to Bellert and Fauceglia 2019) but indicated by a flag equal to 3. (iv) Eurostat reports lists of product codes that changed as well as lists of product codes that existed in each year. Our method employs both lists. Thus, the complete information is used to identify especially new or deleted product codes. (v) Each approach deals with specific classification systems, reports different results, and is written for different programming environments. We summarize these features in Table 4.

Table 4:

Comparison of the approaches dependent on the classification systems, the programming environment, and output.

Approach	Pierce and Scott (2009, 2012	Van Beveren et al. (2012)	Bellert and Fauceglia (2019)	Duprez and Magermann (2022)	Our approach
Classification systems
CN8
PC8
HS6
HS10
BEC
SNA
Programming environment
Software	Stata	Stata	Stata	Stata	R
R-package/Stata-ado-file
Output
Harmonized product codes
History matrix

The complete harmonization approach is available as an open-source R package, called harmonizer.^[7] By default harmonization of CN8 codes is possible from 1995 to 2022 and Prodcom from 2008 to 2021, respectively. However, updated data is added regularly to the package by the maintainer and can even be conveniently added by the users themselves.^[8] Also, usage of product codes and product code change files provided by national Bureau of Statistics is feasible. Thus, harmonization is possible for these products too. In Appendix B the commands for computing the harmonized product codes and the history matrix with the R package harmonizer are provided.

3 Application

We apply harmonization on firm-product-market data of Croatia in the period from 2008 to 2016. For demonstration purpose Table 5 shows one possible (artificial) entry of raw data at firm level.

Table 6 provides the export value of all goods captured by CN8 codes. Columns (2) and (3) show the result without harmonization, while columns (4)–(7) take the harmonization into account. Almost 20 % of the total export value averaged across all years is associated with family product codes. If we do not harmonize the product codes, amount and value of products will be incorrectly attributed to individual product codes, while actually, the codes should be interpreted in the context of a family of product codes (cf. Figure 2). A direct consequence of this misinterpretation can be seen in Table 7.

Table 7 shows all added and dropped products in terms of export, i.e. firms started or stopped trading specific products. From 2008 to 2016, added or dropped products are counted from one year to the following. Thus, the firm had to exist in both years. Columns (2)–(5) represent the trading data without harmonization, while columns (7)–(11) summarize harmonized data. The percentages below the values denote the relative amount of added or dropped products for all products in the market. Comparing the value of added products columns (4) and (9), reveal the strong influence of the missing harmonization. One notices that the average amount sold of new-to-firm products without harmonization is 1.5 times higher than the value with harmonization. Without harmonizing the data, firms are interpreted as introducing more products than they do or being more innovative than they are. A similar pattern occurs if one looks at the value of dropped products, i.e. columns (6) and (11). By looking at non-harmonized data, one would overestimate the amount of dropped products of a firm even though some were not dropped at all. Similar findings are obtained for imports (cf. Tables A1 and A2).

We also investigated the firm sales data from products classified by Prodcom codes in Croatia. Table 8 shows the difference between harmonized and non-harmonized data. The raw data without any harmonization, which can be found in columns (2) and (3) of Table 8, may also include possible typos and national codes. Within the Prodcom classification system, every country can include codes that are only valid for the nation itself. These codes are not used internationally and can therefore only be harmonized with national concordance files. However, if concordance files are available, it is possible to use them within our package since the use of custom trade data as well as of custom concordance files is possible. Nevertheless, the harmonized results shown here do not include these national codes, since we have not yet had access to the national concordance files of Croatia.

Similar findings to Table 7 providing dropped and added products for Prodcom codes can be found in the Appendix in Table A3. Valid identification of the changes in the firms’ product mix allows developing innovation measures and therewith enhancing the second wave of innovation and firm literature. Identified new products can be classified into new-to-firm and new-to-market products, and can also be combined with the value of these products sold or exported – thus providing additional insights into the magnitude of innovation. Researchers can work with the harmonized data to not only identify new-to-firm products but combining these data with firm-level geographical data and identify new-to-firm-region or new-to-firm-national market products. In addition, with the export or sales value, the importance of these new products can be quantified.

4 Conclusions

Similar to research on firms and international trade (Wagner 2019), research on firms and innovation has begun to use transaction data at the level of firms and products or even at the firm-product-market level (Castellani and Fassio 2019). This type of data can be used for many purposes, such as developing innovation measures at different regional levels or identifying characteristics of innovative or high-growth firms. However, such data must be harmonized; otherwise, the validity of the results is compromised. If changes in these classifications are not taken into account, erroneous entries and exits of products, price bias, or incorrect price and quantity indices result.

The open-source R package harmonizer comprises all our harmonization efforts. The main output consists of the harmonized product code and our unique history matrix of product codes of various classification systems (CN8 and Prodcom together with HS6, BEC, and SNA). The history matrix reflects the complex temporal links between product codes that are not readily apparent in the change lists, product code lists, and concordance files. We provide not only the necessary links between product codes but also information to identify special cases, such as deleted or new products in a family of linked product codes. Due to this information, product code changes or drops are not misinterpreted as products discontinued by firms. Furthermore, our R package allows for computing the number of added, kept, or dropped products for each firm each year. In addition, updating the harmonization period is conveniently possible as new concordance files can be provided as input files.

The application of our approach to Croatian firm-product-market data demonstrated the importance of product code harmonization. In our application, the average export value of new products in the firms’ product mix without harmonization is 1.5 times higher than the value using harmonized data. The extent of the significance of harmonizing is in line with the application of Van Beveren et al. (2012) on the data of Belgium.

Establishing a standardized harmonization procedure for trade (CN8) and sales (Prodcom) data is a desirable goal. For this, different researchers across the EU countries have to contribute to obtaining common knowledge about the sensitivity of handling product code families differently. The goal of our approach is a step towards this standardization, as it enables researchers to investigate the sensitivity with respect to this issue. However, further research is still needed to establish a standardized harmonization procedure for these rich data.

With the harmonization of trade and sales data, new and dropped products can be counted reliably, changes in export and import values can be computed, price indices can be derived, and new products can be distinguished between new-to-firm products or new-to-market products. The harmonizer package can be conveniently applied. Our contribution may push forward the research in international trade and innovation economics by enabling research with consistent firm-product data.