Introduction

In Japan, the Feed-in Tariff (FIT) system was introduced in 2012 to promote renewable energy (RE). This system ensured that electricity generated from renewable sources would be purchased at a fixed price for a certain period. Thanks to the system, RE suppliers could avoid market risks and stabilize their prospects for investment recovery. As a result, by the end of FY 2021, the operated renewable energy generation capacity had increased by approximately 3.3 times compared to before the introduction of the FIT system [1]. However, some issues have also been identified, such as the increased burden on the public associated with purchasing RE electricity and the persistently high cost of generating renewable energy. In response to these issues, in 2022, the Japanese Feed-in-Premium (FIP) system has begun operating to encourage market trading of RE electricity and make RE a competitive power source. When RE power suppliers sell electricity in the market through the FIP system, a certain amount of payment (premium) is given on top of the market price [2]. However, if power suppliers cause a discrepancy between planned and actual generation (imbalance) in market trades, they must incur the cost of compensating for the imbalance as a penalty [3]. In other words, RE suppliers with high uncertainty in power generation are likely to cause imbalances. Therefore, the risk of electricity price spikes should be avoided to prevent profit deterioration.

Regarding Japan’s electricity market, several studies focus on predicting RE generation and spot prices. Nakayama et al. incorporate Geographic Information System (GIS) information into a prediction model for photovoltaic power generation and spot prices and mention that GIS information provides better prediction accuracy [4]. Kaneko et al. apply sparse modeling to select important variables for spot price prediction in the Japanese electricity market and show that the importance of calendar dates is relatively higher than that of other variables [5]. Ohta et al. develop a novel price prediction model based on neural networks and show that it could forecast well except during price spikes [6]. Yamada et al. apply a generalized additive model to develop a spot price forecasting model for the Japanese electricity market and mention that challenges remain in forecasting when prices spike [7]. Adline et al. focus on price spike in JEPX and try to developed a price spike predicting model applying a stochastic process called the Hawks process [8]. Maekawa et al. clarify that speculative actions of electricity suppliers can change inelasticity of the demand in electricity market based on mathematical modeling considering merit order curves [9]. On the other hand, it is also suggested that price spikes in the electricity market are difficult to predict because of the significant influence of human behavior and intention, such as the bidding strategies of market participants [10]. However, these studies focus on directly measurable information, such as weather and fuel prices, as explanatory variables for electricity price prediction, or try to model the electricity market mathematically; the influence of human behavior is not considered in their models. Therefore, their application to predicting electricity prices considering human uncertainty will be limited while they are effective in addressing spot price under economically rational operation and understanding past phenomena.

As mentioned above, RE companies must face the uncertainty of human behavior, in addition to variable electricity generation in the electrical market. However, decision making under uncertainty is generally risk-averse [11, 12], and uncertainty can hinder economically rational decisions by RE companies [13,14,15]. In addition, electricity consumers tend to be risk-averse when they select electricity plans [16]. Given these factors, renewable electricity suppliers are expected to be more reluctant to bid for renewable electricity because of concerns about the uncertainty of price spikes. This could be one factor preventing the promotion of market purchases and sales of renewable electricity. Considering the future increase of JEPX participants and the growth of uncertainty due to human factors in addition to variable RE power generation, incorporating measurements of human behavior into predictive models can be an effective approach. Then, how can we measure human behavior and mitigate the uncertainty of price spikes?

Owing to the development of information technology in recent years, large-scale social data on human behavior and communication have become available through websites and social networking services, and research on the social phenomena using these data is attracting attention [17]. Social media is expected to become a “social telescope” that provides a bird's-eye view of society and human behavior [18,19,20], and a wide range of research fields utilize social data. In the field of market transactions, the application of social data through the Web is relatively well discussed in financial markets [21,22,23]. Ersan et al. have noted that reports are increasing to engage financial data with social media information for the purpose of predicting stock prices [24]. Bollen et al. clarify that daily fluctuations in sentiment, as derived from Twitter data, exhibit a statistically significant correlation with the daily closing prices of the Dow Jones Industrial Average [25]. Of particular note is the attempt to understand market behavior from keyword search volumes provided by Google [26], given that actors begin their decision-making processes by attempting to gather information [27]. Preis et al. provide evidence of correlation between the weekly transaction volumes of S&P 500 companies and the weekly search volumes of the corresponding company names [28]. In addition, changes in the volume of Google keyword searches may identify signs of falling stock prices [29]. These reports suggest that some data in social media show a correlation with financial data, and keyword search volumes can be one of indicators in the financial market.

Considering the insights into the financial market stated above, keyword search volumes can be surrogate indicators of human behavior in electricity markets and can mitigate the uncertainty of price spikes because electricity demand is closely related to human behavior, and social conditions influence corporate strategies (Fig. 1). However, no studies explore the utilization of social data in the electricity market, where participants are fewer than in financial markets and the mechanisms are different. Therefore, this study clarifies the relationship between keyword search volumes and electricity market prices by examining the Japan Electric Power Exchange (JEPX); it employs keyword search volumes obtained from Google Trends as representative social data on human behavior.

Fig. 1
figure 1

Hypothesis of this study: Web search behavior may help to capture signs of spikes in electricity prices if the relationship between web search volumes and electricity prices becomes clear

Full size image

This study introduces a novel approach to analyzing electricity market prices using web search data. The methodology developed, particularly the use of time-series clustering and statistical analysis, is not region-specific and can be applied to different markets and geographic locations. In addition, the proposed approach highlights the impact of human behavior, as captured through online searches, on electricity market prices. The knowledge obtained by this study will contribute to understanding markets influenced by consumer behavior and sentiment.

Methodology

Overview of the evaluation procedure

This study focuses on the relationship between keyword search volumes on the Web, which is large-scale social data, and electricity market prices. The section "Conditions for the analysis" defines the conditions for analysis such as target data. The section "Categorization of search volumes by time series clustering" categorizes keyword search volumes based on the similarity of waveforms using a machine learning method to understand the time-series characteristics of the keywords. The sections "Stationarity evaluation of time series data" and "Evaluation of the relationship between area prices and search volumes using statistical methods" examine the relationship between keyword search volumes and electricity market prices using statistical methods.

Conditions for the analysis

  1. (1)

    Scope of analysis

This study selects the Tokyo and Kansai regions in Japan, which are the first and second most populous regions in the country, respectively. We analyze the FY2022 area prices of the JEPX spot market in the two regions and search volumes for keywords obtained from Google Trends.

  1. (2)

    Overview of the electric utility system and electric power exchange in Japan

The Japanese power grid is characterized by a longitudinal transmission system without international connections and is split into ten regions: Hokkaido, Tohoku, Tokyo, Hokuriku, Chubu, Kansai, Chugoku, Shikoku, Kyushu, and Okinawa. The utility frequency of the first three regions is 50 Hz, while that of the remaining regions is 60 Hz. The transmission system in each area is largely independent because the power interconnection capacity is limited to neighboring areas. However, that of the Okinawa region is completely isolated from the other regions [30].

The JEPX has been established in 2003 as part of the deregulation of Japan’s electrical utility system. Approximately 300 electric companies in all regions except Okinawa have joined JEPX [31]. Because Tokyo and Kansai are the most populated areas at 50 and 60 Hz, respectively, they are considered as case study areas. Figure 2 presents the power supply composition of each region. Approximately 85% of total electricity generation in Tokyo comes from thermal power. This is because, as of 2023, nuclear power plants, including the Fukushima Daiichi power plant damaged in the Tohoku earthquake in 2011, are not permitted to restart in the 50 Hz area [1, 30]. However, in Kansai, which is part of the 60 Hz area, nuclear power plants are being restarted and account for approximately 20% of the total power generation. Regarding RE, more electricity is generated in Tokyo than in Kansai.

Fig. 2
figure 2

Electricity generation of Tokyo and Kansai regions in FY 2022 (Visualized from Refs. [34, 35])

Full size image

As of 2023, JEPX is the only exchange that operates a wholesale electricity trading market, where approximately 40% of the electricity sold in Japan is traded [31]. JEPX operates several electricity markets: spot market, intraday market, forward market, and baseload market. Because approximately 98% of the total trading volume of JEPX is in the spot market, we focus on the spot market [31]. The spot market, also called the “day-ahead market,” is where the electricity to be delivered the next day is traded, and the next day is divided into 48 frames by 30 min. The auction process in the spot market is facilitated by a mechanism called the blind single-price auction. Within the single-price auction methodology framework, all submitted bids are segregated into distinct “sell” and “buy” categories after the bid is closed. These categories are then amalgamated to form comprehensive supply demand curves. The point at which these curves intersect establishes both the contract price and corresponding quantity. The contract price is applied to all successful bidders, regardless of their bidding prices [32].

The contract price, which is determined at the point of intersection between the comprehensive supply and demand curves encompassing all selling and buying bids within the nine regions, is referred to as the system price. However, because there are capacity limitations on the interconnection lines in each region of Japan's power grid, the supply quantity from one area to another corresponding to the system price may exceed the limitations. In this case, a supply–demand curve is formulated by considering bids from sellers and buyers within each region. The contract price, which is derived from the intersection of these bids of each region, is called the area price [32]. Given that Tokyo and Kansai are the focal points of the investigation, we explore the relationship between the area price and search volume within these two regions.

In the context of actual supply and demand on trading days, when deviations between the planned and actual generated power occur, which are called “imbalances,” electricity distribution companies adjust their own regulating capacities to rectify the imbalances. Power generation and retail companies responsible for causing imbalances are required to settle the regulating cost corresponding to these imbalances with electricity distribution companies post facto [3]. In particular, RE companies face a higher risk of imbalanced penalties during periods of escalated area prices owing to uncertainty in their power supply. Therefore, this study focuses on the peak values of area prices for each day in FY2022 [33].

  1. (3)

    Google trends

Google Trends provides search volumes for keywords on the Google search engine [36]. Search volumes from Google Trends are not absolute values of the number of searches but relative search volumes [%] with the maximum number of searches in a specified period as 100. Google Trends allows users to specify keywords, periods, and regions from which search volumes must be obtained. This study targets the search volume of “Tokyo Prefecture” and “Osaka Prefecture” as representative districts of Tokyo and Kansai, respectively (see Appendix A for the relevance of target regions to obtain data). Figure 3 shows the time-series trend of the search volume of “electric power” as an example keyword, with the area price also shown. This study hypothesizes that the search volumes of some keywords help capture signs of sudden changes in electricity prices and elucidates the relationship between JEPX area prices and search behavior. To achieve this, search keywords are selected based on the following criteria to evaluate the characteristics of a diverse range of keywords:

  1. 1.

    Context of energy: 40 keywords of high importance calculated by the TF-IDF method for each chapter and section of the Energy White Paper 2022 (Table 1, left side)

  2. 2.

    Social interest: The 25 most frequently searched keywords in Google in FY2022 (Table 1, right side).

Fig. 3
figure 3

Time-series trends of area price (in red) and search volume of “electric power” as an example of keywords (in blue) in FY 2022 (Visualized from Refs. [33, 36])

Full size image
Table 1 The keyword list for the evaluation chosen from Energy White Paper 2022 and the top search keywords by Google
Full size table

Regarding criterion 1, keywords with high importance in the context of “energy” are expected to include those that are relevant to the area price trend. The TF-IDF method [37,38,39], a quantitative method to evaluate the importance of words in documents, is applied to extracting specific keywords, using each chapter and section of the Energy White Paper 2022 [1] as input. The TF-IDF value is obtained for each keyword in each chapter and section, and the sum of the TF-IDF values is calculated for each keyword. The top 40 keywords with the highest TF-IDF values are designated as the keywords of the Energy White Paper 2022 (Table 1, left side). The keyword list includes energy-related terms such as “energy,” “electricity,” and “nuclear power,” as well as business-related terms such as “business,” “price,” and “demand”. We refer to these keywords as “Group A.”

Regarding criterion 2, the trend of area prices might correlate not only with keywords related to energy or electricity but also with keywords linked to events of societal interest. The 25 keywords with the highest search frequency on Google in FY 2022 are selected. Table 1 (right side) presents the various keywords selected: daily searched keywords such as “weather” and “news;” web services such as “YouTube” “Twitter,” and “Amazon;” “COVID-19;” “Corona;” keywords reflecting the current times; manga and anime titles; and others that attract attention in the world. We call these keywords “Group B.”

Categorization of search volumes by time series clustering

In the section "Conditions for the analysis", 65 keywords are assessed for their relevance to JEPX area prices. Although this study represents a novel endeavor to elucidate the connection between JEPX area prices and keyword search volumes, the sheer volume of time-series data poses a challenge for characterization. Consequently, we extract overarching patterns from this extensive time-series data through the utilization of “time-series clustering,” a form of unsupervised machine learning.

TimeSeriesKMeans employs the k-means algorithm [40, 41] to analyze time-series data. The k-means algorithm, a commonly used clustering technique, functions by grouping data in a manner that minimizes the cumulative distance between the center point of each cluster and the data points within that cluster [42]. The simplicity of this algorithm makes it appropriate for clustering large amounts of data. Notably, TimeSeriesKMeans diverges from the conventional k-means methodology by adopting Dynamic Time Warping as its distance metric instead of the Euclidean distance [41, 43]. This choice allows TimeSeriesKMeans to effectively measure the similarity between time-series data, thereby facilitating the clustering process. This study segments the dataset into five distinct clusters based on the results obtained from the analysis of cluster numbers using the elbow method. Appendix B presents the results of the elbow method.

Stationarity evaluation of time series data

In general, the statistical analysis of time-series data assumes that the process is stationary. Process stationarity signifies that both the expected value and variance of data remain constant across time intervals. This stationarity implies that such processes exhibit an absence of trends and tend to revert to mean values over extended periods. Nevertheless, a considerable portion of time-series data within the domains of economics and finance deviate from the ideal stationary process. Therefore, nonstationary data are typically converted into stationary data for analysis or modeling. Given the current context in which the understanding of the time-series characteristics of JEPX area prices and keyword search volumes remains limited, assessing the stationarity of these datasets becomes important. This evaluation is essential to ensure the appropriate assessment of the relationship between area prices and search volumes.

Time-series data in economics and finance such as stock prices are often non-stationary in the original series but stationary in the difference series. Additionally, in the assessment of the profitability of economic and financial data, the logarithmic difference series is often adopted as a measure of the rate of return. The process of taking the logarithm of the original series and obtaining the difference between the logarithms of consecutive data points is referred to as a logarithmic difference series. Therefore, in this study, stationarity is evaluated for both the original series and the logarithmic difference series pertaining to area prices and keyword-specific search volumes. This approach aligns with typical practices in analyzing time-series data in economics and finance.

The augmented Dickey-Fuller (ADF) test, a method commonly employed for testing time-series data stationarity, is used to assess the stationarity of the area prices and keyword search volumes. The significance of the ADF test determines whether the null hypothesis is rejected, indicating that the time-series data are statistically stationary and do not include a unit-root process. Notably, the ADF test necessitates selecting of an appropriate model for both null and alternative hypotheses, considering the characteristics of the data. Consequently, the critical value for the statistical hypothesis test varies based on the selected model. Given the area prices and keyword search volumes analyzed in this study, three types of models are adopted: (1) without a constant term, (2) with a constant term, and (3) with a constant term and a first-order trend.

Evaluation of the relationship between area prices and search volumes using statistical methods

Considering the outcomes of the stationarity test of the time-series data in the previous section, a statistical assessment of the relationship between area price and keyword search volume is conducted. This study examines the relationship between keyword search volumes and area prices to identify keywords that can predict price spikes in the spot market (day-ahead market). To achieve this goal, it is important to evaluate from the perspective of keywords that (1) correlate with the area prices for the day and (2) change in search volumes for the previous day before the change in area price for the day.

For (1), the Pearson correlation coefficient between the area price for the day and the search volume for each keyword is calculated to confirm the correlation. For (2), Vector Autoregressive (VAR) models are developed for area prices for the day. This model incorporates the area price and one of the keyword search volumes for the previous day as explanatory variables, as shown in Function (1). The VAR model is an extension of the autoregressive (AR) model that encompasses multiple explanatory variables with different time lags. Nevertheless, since this study specifically examines the change in area price for the day and in search volume for the previous day, VAR models comprising two variables with a single lag (t-1) are developed: the area price and one of the keyword search volumes with one day lag (i.e., the previous day). The statistical significance of the regression coefficients obtained from the VAR model is evaluated to demonstrate the relationship between electricity prices for the day and keyword search volumes for the previous day.

$$Y\left(t\right)={\beta }_{0}+{\beta }_{1}Y\left(t-1\right)+{\beta }_{2}{X}_{n}\left(t-1\right)+\epsilon$$
(1)

where Y is the area price, Xn is the search volume for each keyword, n is each keyword (n = 0, …, 64), t is the day, β0, β1, and β2 are the regressive coefficients, and ε is the error term with mean 0 and variance constant.

Results and discussion

Keyword categorization by time series clustering

Figure 4 presents the clustering results using TimeSeriesKMeans, and Table 2 summarizes the corresponding clusters for each keyword. Figure 4 comprises five graphs, designated as clusters 0 to 4, from top to bottom. In each graph, the thick colored line represents the trend of the cluster center, and the thinner gray lines represent the search volume trends associated with keywords classified within that specific cluster. As Table 2 shows, 14 of the 40 keywords in Group A are classified into different clusters in Tokyo and Kansai, whereas only 2 of the 25 keywords in Group B belonged to different clusters. Therefore, Group A, which represents the context of energy, contains a relatively large number of keywords with different types of waveforms for each region. However, the waveform of Group B, which represents social interest, shows similarity by region.

Fig. 4
figure 4

Results of time-series clustering: the thick-colored lines are the trend of cluster centers, and the thinner gray lines are the search volume trends in assigned clusters

Full size image
Table 2 Results of time-series clustering by keywords, keywords without “x” share the same cluster both in Tokyo (T) and Kansai (K) regions
Full size table

Figure 5 shows the standard deviation of the search volume for each keyword on the horizontal axis, and the mean value on the vertical axis. The color of each data point corresponds to the cluster to which it belongs. Since TimeSeriesKMeans clusters similar time series trends based on the distance between the data, the standard deviation and mean values of the time-series data do not necessarily directly explain the characteristics of each cluster. However, Fig. 5 indicates that the mean and standard deviation of each keyword, along with the clustering results, contribute to a comprehensive understanding of the time-series attributes of each keyword. The characteristics of each cluster are as follows.

Fig. 5
figure 5

The distribution of standard deviation and mean of search volumes and the colors in the plots show assigned clusters

Full size image

Cluster 0

  • Median of mean: 32.2 (2nd lowest); median of standard deviation: 20.5 (1st highest)

As the waveforms of this cluster show, daily search volumes are relatively low; however, interest in keywords frequently increases and decreases over large ranges.

  • Keywords (Groups A, B): Tokyo (6, 0), Kansai (11, 0)

All keywords in this cluster belong to Group A indicating shared themes related to energy. Keywords such as “nuclear power,” “supply,” “implementation,” “crude oil,” “energy saving,” and “department” are common to both Tokyo and Kansai. Nevertheless, Kansai has more Group A keywords, including “fuel,” classified within this cluster compared to Tokyo.


Cluster 1

  • Median of mean: 56.4 (2nd highest); median of standard deviation: 16.9 (2nd highest)

Keywords belonging to this cluster maintain consistent popularity, but also have a relatively large range of increases and decreases depending on the interest in the keyword.

  • Keywords (Groups A, B): Tokyo (17, 2), Kansai (10, 1)

All keywords in this cluster belong to Group A, except for “Google” and “translation” in Tokyo and “translation” in Kansai from Group B. Tokyo and Kansai share keywords such as “energy,” “business,” “technology,” and “economy.” The keywords represent the most energy context for Tokyo in particular.


Cluster 2

  • Median of mean: 41.7 (middle); median of standard deviation: 9.65 (2nd lowest)

Keywords belonging to this cluster are widely distributed in mean values, but show moderate popularity and relatively stable volatility overall. However, waveform spikes are occasionally observed.

  • Keywords (Groups A, B): Tokyo (6, 9), Kansai (9, 10)

Tokyo and Kansai share several keywords, such as “power generation” and “electric power” from Group A, and “weather” and “Yahoo” from Group B, but relatively more Group B keywords are assigned.


Cluster 3

  • Median of mean: 20.25 (1st lowest); median of standard deviation: 12.2 (middle)

Interest in keywords in this cluster is relatively low; however, spike-shaped and characteristic mountain-shaped waveforms are occasionally observed.

  • Keywords (Groups A and B): Tokyo (6, 3), Kansai (4, 3)

Though Tokyo and Kansai share several keywords, this cluster is characterized by keywords such as “COVID-19,” “Pokémon,” and “ONE PIECE” from Group B, whose search volumes increase when some events which many people attention to increase in infected patients and release of movies.


Cluster 4

  • Median of mean: 62.5 (1st highest); median of standard deviation: 7.5 (1st lowest)

The keywords in this cluster maintain high popularity, and the validation is relatively smaller than that in other clusters.

  • Keywords (Group A, Group B): Tokyo (5, 11), Kansai (6, 11)

This cluster includes the highest count of Group B keywords and is presumed to be the cluster that best capture daily search behavior. For instance, “YouTube,” “Twitter,” “Instagram,” and “sports” are assigned in both Tokyo and Kansai.

Evaluation of stationarity of time series data

Table 3 shows the results of the ADF test for area prices and search volumes of the original series (Table 3b) and the logarithmic difference series (Table 3c) for Tokyo. The results of time-series clustering, which correspond to Table 2, are also shown in Table 3a for reference. Table 4 presents the results for Kansai. With respect to the original series of area prices, the results for Tokyo are significant at the 5% level for (1) and significant at the 1% level for (2) and (3), indicating that they may be stationary. However, for Kansai, the statistical significance of the ADF test could not be confirmed and nonstationarity is not rejected. These findings indicate the presence of regional differences in the stationarity of the original series. Conversely, in the logarithmic difference series for both Tokyo and Kansai, the results are significant at the 1% level for (1), (2), and (3), indicating the likelihood of the stationary process.

Table 3 Results of ADF test for Tokyo Region
Full size table
Table 4 Results of ADF test for Kansai Region
Full size table

As for the original series of the keyword search volumes, the test statistic of (1) for all keywords is non-stationary except “deal” and “coal” in the original series for both Tokyo and Kansai. For (2) and (3), a mix of keywords exhibit statistical stationarity and nonstationarity. Although no clear distinction based on clusters or regions is observed, fewer keywords belonging to Group B and clusters 2, 3, and 4 statistically deny nonstationarity. For the logarithmic difference series, except for “COVID-19,” the results indicate that null hypotheses for (1), (2), and (3) are rejected at the 1% level in both Tokyo and Kansai, suggesting a high likelihood of statistical stationarity. Therefore, some keywords may be stationary even in the original series in terms of search volume; however, it is difficult to clearly deny non-stationarity in light of the results in (1).

Based on the above results, the logarithmic difference series of area prices and search volumes is applied to the analysis in the next section. This allows for a more rigorous statistical evaluation and direct comparison between area prices and keyword search volumes across regions.

Evaluation of the relationship between area prices and search volumes using statistical methods

Table 5 presents the analysis results of the relationship between area prices and search volumes. Table 5b shows the Pearson correlation coefficient (R) of search volumes with the area price in each region. L(0) indicates that the number of lags is zero, that is, the area price and search volume on the same day. In Table 5, keywords with a correlation coefficient of 0.4 or higher in either Tokyo or Kansai are extracted. Appendix C shows the outcomes for all keywords. Table 5c shows the statistical estimates based on the VAR(1) models, which predict the area price on the day (L0) from the area price and the search volume for one of the keywords on the previous day (L1). A logarithmic difference series is employed for the evaluation. The time-series clustering results corresponding to Table 2 are also shown in Table 5a for reference purposes.

Table 5 Statistical analysis results with Pearson correlation coefficient and VAR mode
Full size table

When a correlation coefficient of 0.4 or higher is used as the criterion for a relatively high positive correlation between area price and search volume, applicable keywords are “business,” “electric power and,” “industry” from Group A, and “Google,” “Yahoo,” “translation” from Group B common for Tokyo and Kansai, and “technology,” “development” and “plan” from Group A only for Tokyo are applicable. As for the assigned clusters, for Tokyo, seven out of the nine keywords are found in Cluster 1 and one each in Clusters 2 and 3. For Kansai, three of the six keywords come from Clusters 1, two from Cluster 2, and one from Cluster 4. Keywords that are highly correlated with area prices are found most frequently in Cluster 1.

Macroscopically, with regard to the VAR model analysis, only 12 models for Tokyo confirm the 1% or 5% significance of the regression coefficients for area price (L1). Thus, t the regression coefficient of area price (L1) is undeniably zero in many models. However, in Kansai, the significance of the regression coefficients for area price (L1) is confirmed for all models. Consequently, this result implies that the area price on the day in Tokyo is less influenced by the previous day's price than in Kansai (see also Table 7 in Appendix C).

As for search volume (L1), when the models with regression coefficients of 1% and 5% significance are extracted, there are 30 keywords (Group A: 21, Group B 9) for Tokyo and 35 keywords (Group A: 20, Group B 15) for Kansai, and 24 keywords (Group A:15, Group B 9) are common for both regions. The significance of the regression coefficient is confirmed for the keywords that show relatively high positive correlations with area prices, such as “business” and “google,” except for “electric power” in Tokyo. Comparing Groups A and B, the regression coefficients tend to be greater in Group B (see also Table 7 in Appendix C). Although no clear regional differences are identified, the regression coefficients are generally higher for Kansai.

These results suggest that the logarithmic difference series of the following keywords are partially linked to the area price on the day, and changes in the area price for the day may be captured from changes in the search volume for the previous day:

Tokyo Region

  • Cluster 1 – Group A: “business,” “technology,” “development,” “industry,” “plan”

    – Group B: “Google”, “translation”.

  • Cluster 2 – Group B: “Yahoo”

Kansai Region

  • Cluster 1 – Group A: “business,” “industry”

    – Group B: “translation”

  • Cluster 2 – Group A: “electric power”

    – Group B: “Yahoo”

  • Cluster 4 – Group B: “Google”

Examining Fig. 5, I observe that they are plotted in the middle or higher mean and standard deviation of search volume. When Groups A and B are compared, the keywords in Group A have higher standard deviations. Although there are some differences in the words extracted from Group A between Tokyo and Kansai, the search volumes of these keywords may move relatively significantly before the area price changes. In Group B, the common keywords "Google," "translation,” and "Yahoo" are extracted from Tokyo and Kansai. These are popular keywords associated with general Web search behavior with high absolute search volumes.

Figures 6 and 7 show orthogonalized impulse response analysis of each area price in case a one standard deviation shock in each keyword search volume is given to the VAR models. The analysis helps us understand the influence of keyword search volumes on area price by isolating the error terms associated with each variable within the model. Figure 6 is for the Tokyo region, and Fig. 7 is for the Kansai region. The dotted lines in each graph indicate a confidence interval of 95%. Since this study assumes the use of a keyword search volume as a one-day leading indicator of each area price, the magnitude corresponding to one on the horizontal axis of each graph explains the lag effect of the keyword search volume on the area price. As Fig. 6b shows, the lower dotted line is under zero even though the upper dotted line is above zero. This means search volumes of “electric power” in the Tokyo region may not be a leading indicator of the area price by the confidence interval of 95%. On the other hand, the one-day lag effect is observed in search volumes of the keywords in Figs. 6 and 7 other than “electric power” in the Tokyo region from orthogonalized impulse response analysis point of view. Moreover, the lag effect in the Kansai region is greater than that in the Tokyo region. These results implies that human search behavior is linked to area prices to some extent, and the observation of these keywords may help detect sudden changes in area prices.

Fig. 6
figure 6

The impulse response analysis based on the VAR models in the Tokyo region. The vertical axis represents the magnitude of the response in the area price to a one standard deviation shock in each keyword search volume. The horizontal axis shows the day lag of the area price

Full size image
Fig. 7
figure 7

The impulse response analysis based on the VAR models in the Kansai region. The vertical axis represents the magnitude of the response in the area price to a one standard deviation shock in each keyword search volume. The horizontal axis shows the day lag of the area price

Full size image

Conclusion

This study highlights the lack of sufficient discussions on whether social data through the Web can help mitigate the uncertainties of human behavior’s influence on the electricity market, even though the uncertainties in the market may discourage RE companies’ proactive electricity deals. This study hypothesizes that keyword search volumes, which are social data taken from the Web, might help identify signs of changes in electricity prices, and clarifies the relationship between keyword search volumes and electricity market prices. Keyword search volumes are categorized by the time-series clustering method. The relationship is evaluated by statistical methods, and the Tokyo and Kansai regions are the focus of this study.

This study draws several conclusions. First, new insights into the stationarity of the time-series data on area prices and keyword search volumes are obtained. Regarding area prices, regional differences are observed; the area prices of Tokyo are possibly stationary in both the original series and logarithmic difference series. However, for Kansai, the original series is likely to be nonstationary, whereas the logarithmic difference series is likely to be stationary. Although some previous analyses in other countries suggest that electricity prices are non-stationary in the original series [44, 45] and become stationary by taking the difference series [45], no studies explore the difference in regions. Here, looking at the composition of power sources in Japan, thermal power generation accounts for a large proportion of power generation in the Tokyo region, with natural gas in particular accounting for more than 50% of the total [34, 46]. On the other hand, nuclear power generation is relatively larger in the Kansai region, and natural gas and coal are almost equally represented in thermal power generation [35, 47]. The Kansai region has more varied power sources, which means that the cost of power generation changes more sensitively with changes in demand. This may bring the differences of stationarity in the original series of area prices though future investigations are needed. The result that there are regional differences in the stationarity of electricity prices implies that the composition of power sources needs to be considered when constructing the electricity price forecasting models. As for search volumes, although some differences by keyword are observed, no clear regional differences are confirmed, and the possibility of non-stationarity for the original series could not be clearly rejected. Meanwhile, the logarithmic difference series is likely to be stationary. Although the stationarity of keyword search volumes has not been fully discussed in previous studies, the fact that it can be made stationary by taking logarithmic difference series is an important finding for building statistical forecasting models in the future using keyword search volume as an explanatory variable. Considering above, the logarithmic difference series is preferable for a statistically reliable time-series data analysis on area prices and search volumes, including economic indicators such as stock prices.

Second, the correlation of area prices on the day with search volumes on the day is evaluated using the Pearson correlation coefficient of the logarithmic difference series. In addition, the regression analysis with the VAR model is examined to explain the changes in area prices for the day by those in keyword search volumes for the previous day. We find that the area prices moved, linking search volumes of some keywords and whose changes could be captured by the search volumes of the keywords on the previous day. Specifically, "business," "technology," "development," "industry," and "plan" for Tokyo and "business", "industry," and "electric power" for Kansai are extracted from the keywords used in the context of energy. As for keywords showing social interest (Group B), "Google," "translation," and "Yahoo" are common for both regions; those are popular keywords associated with general web search behavior that have high absolute search volumes. Moreover, the mean and standard deviation of the search volumes for these keywords are above moderate, and these keywords are categorized into specific clusters based on the similarity of the waveform point of view. Although similar keywords are extracted from both Tokyo and Kansai, the regression coefficients tend to be larger for Group B than for Group A and larger for Kansai than for Tokyo. As mentioned in the introduction, previous studies in the stock market have suggested that changes in the volume of some keyword searches correlate with stock trading and identify early warning signs of stock [26, 28, 29]. It is thought-provoking that the results of this study are consistent with the previous analysis even though the JEPX has fewer participants and the different trading rules compared to the stock markets. Investigating the background events and social conditions that link the search volume of these keywords with a day-ahead lag effect could provide a better understanding of the potential impact of public interest and sentiment on electricity market prices.

These results suggest that the logarithmic difference series of specific keywords used in energy contexts such as “business” and “industry” and popular words in daily search behavior such as "Google," "translation," and "Yahoo" can be effective as explanatory variables for area price prediction models and help capture the signs of area price spikes. Despite the significant and valuable results of this study, the following conditions are worth mentioning for future studies.

  1. 1.

    This study applies statistical methods to investigate the relationship between keyword search volumes and area prices and shows that the area prices for that day “tend to change when” search volumes of some keywords for the previous day have changed. While this result is statistically appropriate, this does not show a causal relationship. It does not necessarily indicate that the area prices on the day changes “because” search volumes of some keywords in the previous day have changed.

  2. 2.

    This study focuses on Tokyo and Kansai in FY 2022 as a case study because they have the largest web search volumes and electricity deals. However, future studies can examine the differences between other regions over multiple years to obtain more general insights. The evaluation procedure and findings of this study are helpful for future investigations.

Considering the outcomes of this study and the conditions mentioned above, we expect to develop a price spike prediction model using search volumes to mitigate the uncertainties caused by human behavior in the market as future work.

Lastly, the findings of this study offer several policy implications, particularly in shaping decision-making processes within the electricity market, although the development of the prediction model for price spikes in the electricity market is future work. Firstly, the correlation between online search behavior and electricity market prices can be a valuable tool for policymakers and market regulators. Understanding the potential influence of public interest and sentiment, as reflected in web searches, on market prices can aid in developing more responsive and adaptive market strategies. Secondly, regulators can use these insights to anticipate market fluctuations. Incorporating web search data into predictive models makes it possible to foresee and mitigate abrupt price spikes, thus ensuring market stability and consumer protection. Thirdly, observation of regional differences in the correlation between search volumes and market prices can suggest region-specific policy approaches. While this study focuses on Tokyo and Kansai, the methodology is adaptable to other regions in Japan and globally. The unique market dynamics and consumer behavior patterns in each region or country can be analyzed similarly to derive region-specific insights. Policymakers may need to consider these regional nuances when designing regulations and interventions in the electricity market. In summary, integrating web search data analysis in the electricity market opens new avenues for informed decision-making and policy development. Future research in this area can further refine these approaches, offering more nuanced insights into the interplay between public sentiment and market dynamics.