The UV Luminosity Function of Protocluster Galaxies at z ∼ 4: The Bright-end Excess and the Enhanced Star Formation Rate Density

T18 use the HSC–SSP S16A internal data release for selecting g-dropout galaxies. HSC is the prime focus camera of the Subaru Telescope (Komiyama et al. 2018; Miyazaki et al. 2018).

The HSC–SSP survey is a wide and deep survey carried out over 300 nights by the HSC collaboration (Aihara et al. 2018a). The target fields are divided into three layers (Wide, Deep, and UltraDeep), and five broad bands (grizy) and three narrow bands are used (for more details on the HSC filter system, see Kawanomoto et al. 2018). The Wide layer has a 5σ limiting magnitude of i ∼ 26 mag. HSC–SSP data are processed via hscpipe (Bosch et al. 2018), which is a modified version of the Legacy Survey of Space and Time software (Axelrod et al. 2010; Jurić et al. 2017; Ivezić et al. 2019). In the S16A data release, the total survey area of the Wide layer observed in all bands and reaching to the full depth is 178 deg², and the average seeing is 056 in i band and 065–07 in other bands.

T18 construct a sample of g-dropout galaxies from the gri-band photometry. Only five regions in the Wide layer have enough depth (XMM-LSS, WIDE12H, GAMA15H, HECTOMAP, and VVDS) to construct a homogeneous map of the galaxy distribution. T18 impose color criteria (for g − r and r − i) and a limiting magnitude cutoff (5σ significance in the i band and 3σ significance in the r band), based on the Cmodel magnitudes (Bosch et al. 2018). Various flags are used to select objects with clean photometry and that not affected by cosmic rays, and so on (for more details, see T18).

T18 select protocluster candidates according to the peak value of the overdensity significance. The overdensity map of g-dropout galaxies is drawn from their surface number density using the fixed aperture method. This method distributes circular apertures on every 1' grid cell and estimates the surface number density of galaxies from the number of galaxies inside the apertures. They define an aperture size of 18, which corresponds to ∼0.75 physical Mpc at z ∼ 3.8. This size is the smallest one expected for protoclusters of "Fornax-type" clusters (M_halo ∼ 1–3 × 10¹⁴ M_⊙ at z ∼ 0), as predicted by simulations (Chiang et al. 2013).

T18 only focuses on regions whose limiting 5σ magnitudes for the g, r, and i bands are deeper than 26.0, 25.5, and 25.5 mag, respectively, giving an effective survey area of 121 deg². To draw the overdensity map, T18 uses the g-dropout galaxies that are brighter than 25 mag in i band. T18 select as protocluster candidates 179 overdense regions whose peak overdensity significance is greater than 4σ, following Toshikawa et al. (2016). T18 determine that about ≥76% of such regions will evolve into halos with a mass greater than 10¹⁴ M_⊙ at z ∼ 0.

This large sample of protoclusters allows T18 to conduct an angular clustering analysis and estimate the mean dark matter halo mass as $\langle {M}_{\mathrm{halo}}\rangle ={2.3}_{-0.5}^{+0.5}\times {10}^{13}\,{h}^{-1}\,{M}_{\odot }$. According to the extended Press–Schechter model, halos with such a large mass are indeed expected to evolve into those with $\langle {M}_{\mathrm{halo}}\rangle \,={4.1}_{-0.7}^{+0.7}\times {10}^{14}\,{h}^{-1}\,{M}_{\odot }$ at z ∼ 0.

We have to define the volume of protoclusters in order to measure the UVLF. It should be noted that these protocluster candidates and their members have a redshift uncertainty (δz ∼ 1) because this method is based on the dropout technique. We approximate the shapes of protoclusters as cylinders. The cross section of the cylinder is a circle with a radius of 18, corresponding to 0.75 physical Mpc, which is the same size as the aperture in the overdensity map. The line-of-sight length is equivalent to the diameter of the cross section. Therefore, we select protocluster member galaxies from galaxies that are located within a projected distance <18 from the center of the overdensity peak. We consider a masked region in the determination of protocluster volumes. Note that we do not consider the particular morphology of each protocluster. For example, some protoclusters, particularly the more massive ones, can be bigger (e.g., Chiang et al. 2013; Muldrew et al. 2015). Some studies also argue that the shape of protoclusters can be described with a triaxial model (Lovell et al. 2018). The radius used for selecting member galaxies in the study is the minimum size of protoclusters as predicted by the simulation (Chiang et al. 2013), therefore our selected regions are expected to contain pure protocluster members, but we might miss some member galaxies that are located in the outermost regions of protoclusters. As we discuss in Section 5.2, our results for the shape of UVLF do not significantly change even if we change the radius of the cross section and the line-of-sight depth.

We estimate the UV absolute magnitude, which is the absolute magnitude at 1500 Å in the rest-frame, from the apparent magnitude. As mentioned in Section 2.2, our protocluster galaxies have a significant redshift uncertainty because they are selected from g-dropout galaxies. Therefore, we fix $\bar{z}=3.8$ as the typical redshift. We convert the i-band magnitude (m_i) by using the following equation:

$$\begin{eqnarray}\begin{array}{rcl}{M}_{\mathrm{UV}} & = & {m}_{i}+2.5\mathrm{log}(1+\bar{z})\mbox{--}5\mathrm{log}\left(\displaystyle \frac{{d}_{L}(\bar{z})}{10\ \mathrm{pc}}\right)\\ & & +\ ({m}_{1500(1+\lambda )}-{m}_{i}).\end{array}\end{eqnarray} \tag{ 1 }$$

Here, ${d}_{L}(\bar{z})$ is the luminosity distance at $z=\bar{z}$ in parsecs. We assume that the SED of g-dropout galaxies in the rest-UV is flat in f_ν, which leads to a k-correction factor (${m}_{1500(1+\lambda )}-{m}_{i}$) of zero, following Ono et al. (2018).

We measure only the projected number density from the photometric data; therefore, our protocluster galaxy sample has some possible contaminants. One is fore/background g-dropout galaxies outside the protocluster regions, hereafter called "field galaxies." The effective redshift range of g-dropout galaxies is significantly larger than the protocluster's transverse size, so we must subtract the contribution of field galaxies from the measured surface number density in protocluster regions. The number density of field galaxies can be approximated by the UVLF of field galaxies (field UVLF), as the volume fraction of protoclusters is small compared to the total survey volume. In addition to field galaxies, g-dropout galaxies themselves may inevitably have some contaminants such as stars and low-redshift galaxies due to the color selection uncertainties, which should be removed from the sample. These objects can be assumed to be homogeneously distributed if we combine all protoclusters, which are separated on the whole sky; therefore, their contamination rate should be the same both inside and outside of the protocluster regions. This implies that the subtraction of the field UVLF without the contamination correction provides a clean estimate of the number density of protocluster galaxies.

One possible contamination source that is unlikely to be homogeneously distributed is low-z galaxy clusters at 0.3 < z < 0.6, where Balmer breaks are hardly distinguishable from Lyman breaks at z ∼ 4. Oguri et al. (2018) construct a galaxy cluster sample at 0.1 < z < 1.1 from 232 deg² HSC–SSP data. They find 620 clusters at 0.3 < z < 0.6, indicating their surface number density is 2.67 deg⁻². The possibility that our protoclusters are overlapped with galaxy clusters at 0.3 < z < 0.6 within 18 (i.e., the protocluster size) is only 0.59%. Therefore, we conclude that all contamination is negligible in the estimation of the UVLF of protocluster galaxies (PC UVLF).

We correct the effective volume of g-dropout galaxies to the protocluster effective volume by a factor F defined as;

$$\begin{eqnarray}&&F({M}_{\mathrm{UV}})=\displaystyle \frac{\left\langle C({M}_{\mathrm{UV}},z)\tfrac{{dV}(z)}{{dz}}\delta z\right\rangle }{{V}_{\mathrm{eff}}({M}_{\mathrm{UV}})}.\end{eqnarray} \tag{ 2 }$$

Here, C(M_UV, z) is the completeness function of the g-dropout selection estimated in Section 3.2. The term δz is the redshift interval that corresponds to the depth of the cylinder volume of protoclusters (see Section 2.2). The differential comoving volume is dV(z)/dz. The term V_eff(M_UV) is the effective volume for g-dropout galaxies in the 18 aperture, which is defined as follows (e.g., Hogg 1999):

$$\begin{eqnarray}&&{V}_{\mathrm{eff}}({M}_{\mathrm{UV}})=\int C({M}_{\mathrm{UV}},z)\displaystyle \frac{{dV}(z)}{{dz}}{dz}.\end{eqnarray} \tag{ 3 }$$

The numerator of F(M_UV) corresponds to the effective volume of a protocluster, whose shape is defined in Section 2.2. Therefore, F(M_UV) is the ratio of the effective volume of the protoclusters and the effective volume of the redshift range of the entire g-dropout selection. Since we do not know the exact redshift of each system, we use the average numerator weighted by the redshift selection function (i.e., the completeness function).

Then, the PC UVLF is described as follows,

$$\begin{eqnarray}\begin{array}{rcl}{{\rm{\Phi }}}_{\mathrm{PC}}({M}_{\mathrm{UV}}) & = & \displaystyle \frac{1}{F({M}_{\mathrm{UV}})}\\ & & \times \ \left(\displaystyle \frac{{n}_{\mathrm{obs},\mathrm{PC}}({M}_{\mathrm{UV}})}{{V}_{\mathrm{eff}}({M}_{\mathrm{UV}})}-{{\rm{\Phi }}}_{\mathrm{field}}({M}_{\mathrm{UV}})\right),\end{array}\end{eqnarray} \tag{ 4 }$$

where n_obs,PC(M_UV) is the observed number of g-dropout galaxies in protocluster regions as defined in Section 2.2 in each magnitude bin. The term Φ_field(M_UV) is the field UVLF without the contamination correction (see Section 3.3). In order to determine Φ_PC(M_UV), we estimate the completeness function of g-dropout galaxies C(M_UV, z) and the field UVLF without contamination treatment in the following section.

As in previous studies of UVLFs of field LBGs (e.g., Yoshida et al. 2006; van der Burg et al. 2010; Ono et al. 2018), we insert mock galaxies into actual images and estimate a completeness function. This is derived as a function of the redshift and the magnitude.

Mock galaxies are inserted into the coadd images of the g, r, and i band images of the HSC–SSP products. We generate mock images with Balrog¹⁴ (Suchyta et al. 2016), which inserts mock galaxies with the help of galsim¹⁵ (Rowe et al. 2015), and proceed with their detection and measurement using SourceExtractor. However, the HSC–SSP source catalog is constructed based on hscpipe; therefore, we detect and measure the photometry of the mock galaxies with hscpipe, instead. We use hscpipe version 4, which is the same software used for the HSC–SSP S16A data release.

We assume that the surface brightness profile follows the Sérsic profile (Sérsic 1963) with a fixed Sérsic index of 1.5 for mock galaxies. In addition, the effective size distribution is assumed to be consistent with that of Shibuya et al. (2015). The real profiles of the mock galaxies are convolved with the point spread functions (PSFs) of the fields taken from from PSFEx¹⁶ (Bertin 2011) and inserted. The SEDs of mock galaxies are generated using CIGALE¹⁷ (Boquien et al. 2019). We assume a constant star formation and use the single stellar population models of Bruzual & Charlot (2003). We adopt the Salpeter initial mass function (IMF; Salpeter 1955) with an age of 100 Myr and metallicity of Z/Z_⊙ = 0.2. The dust extinction follows Calzetti et al. (2000) with E(B − V) = 0.0–0.4 mag. The IGM absorption is accounted for according to Meiksin (2006). We change their redshift from 3.0 to 5.0 with an interval of δz ∼ 0.1.

Due to the slight differences in depths among the five fields, we estimate the completeness function for each field. We select one region called tract, with an area of 2.3 deg², for each field to execute the procedure. The number of inserted galaxies is about 35 per arcmin². From the detected catalogs, we select mock g-dropout galaxies by the same criteria as used in T18, including color, magnitude, and flags selection.

For each field, we calculate the completeness as the number ratio of selected mock g-dropout galaxies to all inserted objects in each magnitude and redshift bin. Figure 1 shows the completeness function of each field, demonstrating that the five fields have almost the same completeness.

Zoom In Zoom Out Reset image size

From the completeness function and Equation (5) below, we estimate the field UVLF without contamination treatment.

$$\begin{eqnarray}&&{{\rm{\Phi }}}_{\mathrm{field}}({M}_{\mathrm{UV}})=\displaystyle \frac{{n}_{\mathrm{obs},\mathrm{field}}({M}_{\mathrm{UV}})}{{V}_{\mathrm{eff}}({M}_{\mathrm{UV}})}\end{eqnarray} \tag{ 5 }$$

Here, n_obs,field(M) is the observed number of field galaxies and contaminants of M_UV = M. Before deriving n_obs,field(M), we remove all known low-z galaxies, stars, or QSOs from the available spectroscopic survey archives, such as Sloan Digital Sky Survey DR12 (Alam et al. 2015), HectoMAP cluster survey (Sohn et al. 2018), and VIPERS DR1 (Garilli et al. 2014). The majority of matched objects are galaxies at 0.3 < z < 0.6 and QSOs at the same redshift distribution as the g-dropout galaxies. Only two QSOs, which overlap with the protoclsuter region, are removed from the sample (Uchiyama et al. 2018).

We compare the input total magnitude and the measured 20 aperture magnitude of mock galaxies used in Section 3.2, and find that the 20 aperture magnitude has a +0.08 mag offset on average from the input magnitude. Therefore we apply a 0.08 mag aperture correction to our measured 20 magnitudes to derive the total magnitudes. We confirm that the derived total magnitudes are consistent with the aperture magnitudes measured with larger apertures, such as 30, 40. We also correct for the galactic extinction using the extinction map from Schlegel et al. (1998).

This UVLF is not necessarily the same as the field UVLF derived in previous studies (e.g., Bouwens et al. 2015; Ono et al. 2018) as our function includes contaminants, as seen in Figure 2. We derive the field UVLFs for each field.

Zoom In Zoom Out Reset image size

The bottom panel of Figure 2 shows the difference between the average of the UVLFs and that of Ono et al. (2018) normalized by this average UVLF. Since the UVLF of Ono et al. (2018) excludes contaminants, this represents the expected fraction of contaminants among our g-dropout galaxies. We can find that this ratio is consistent with that in Ono et al. (2018), overplotted in the bottom panel of Figure 2. We conclude that our completeness function is consistent with that of previous studies. Hereafter, we will use these UVLFs and the completeness function to estimate the PC UVLF.

Here, we estimate the PC UVLF according to Equation (4). Two protoclusters are excluded because they are located in low-quality regions with quite shallow limiting magnitudes (m ∼ 25.6 mag for the 5σ i-band limiting magnitude); therefore, 177 protocluster regions are used to estimate the PC UVLF. Because the completeness function and Φ_field(M_UV) have been determined for each field, the PC UVLF is also estimated for each field separately, and we take the average weighted by the total area for each field as our final PC UVLF. We note that all PC UVLFs for each field are overall consistent with each other within the uncertainty.

We show the average PC UVLF of the HSC–SSP protocluster candidates in Figure 3. Our PC UVLF has apparent discrepancies with the field UVLF in the literature (e.g., Ono et al. 2018). First, the amplitude is much higher than for the field UVLF: the integrated value of the PC UVLF at M_UV ≤ −20.3 is about 230 times higher than that of the field UVLF of Ono et al. (2018). Second, its shape is remarkably different from that of the field UVLF. The amplitude-matched field UVLF is also shown in the top panel of Figure 3 for reference, and compared with that, the PC UVLF has a significant excess toward the bright end (M_UV ≤ −20.8). The trend can also be seen on the bottom panel of Figure 3, which shows the ratio of the PC and the field UVLF. We see that the excess gets larger toward the brighter bins. If the shapes are identical, this ratio should stay constant at any brightness.

Zoom In Zoom Out Reset image size

Since the number density of galaxies decreases toward the bright end, so the photometric error of each galaxy might enhance the amplitude of the bright end of the UVLF, which is known as the "Eddington Bias" (Eddington 1913). We estimate the effect of this bias by convolving the error distribution of magnitude to that of the field UVLF of Ono et al. (2018). The detail of this analysis is described in Appendix A. We confirm that the Eddington bias is not significant enough to generate the shape of our PC UVLF.

Because contributions from low-z contaminants, which are distributed homogeneously, are statistically subtracted from the sample as mentioned in Section 3.1, the bright-end excess is not due to low-z galaxy contaminants. Also, the PC UVLF may depend on F(M_UV), which is the ratio of the effective volume of protoclusters and g-dropout galaxies. We confirm that the bright-end excess of the PC UVLF does not change even if we fix F (M_UV) = 1, as seen in Appendix B.

The rest-UV luminosity of galaxies represents their SFR. Therefore, this result indicates that overdense regions at z ∼ 4 have not only a high SFRD caused by the excess of the number of galaxies, but also a higher fraction of galaxies with high SFR compared to those in the blank field. This trend is also seen in some protoclusters at lower redshifts. For example, Shimakawa et al. (2018) estimate the SFR of HAEs in a protocluster at z = 2.5, and they also find that HAEs in the densest regions tend to have a higher SFR than those in the outskirts. Koyama et al. (2013) report a similar trend from HAEs in protoclusters at z ∼ 2. This paper, for the first time, shows that the enhancement of star formation of UV-bright galaxies in overdense regions can already be seen as early as z ∼ 4. We have to note that some bright (M_UV < −23.0) LBGs can be AGNs, whose UV emission cannot be a proxy for the SFR of their host galaxies (e.g., Ono et al. 2018; Adams et al. 2020). We discuss a possible contribution from AGNs in Section 5.3.

To compare the shape of the PC UVLF with that of the field UVLF more quantitatively, we fit the Schechter function (Schechter & Press 1976), which is defined as follows:

$$\begin{eqnarray}&&\phi (L){dL}={\phi }^{* }{\left(\displaystyle \frac{L}{{L}^{* }}\right)}^{\alpha }\exp \left(-\displaystyle \frac{L}{{L}^{* }}\right)d\left(\displaystyle \frac{L}{{L}^{* }}\right),\end{eqnarray} \tag{ 6 }$$

where α is the faint-end slope, L* is the characteristic luminosity, and ϕ* is the overall normalization. This function can be also expressed as a function of the absolute magnitude M_UV,

$$\begin{eqnarray}\begin{array}{rcl}{\rm{\Phi }}({M}_{\mathrm{UV}}) & = & \displaystyle \frac{\mathrm{ln}10}{2.5}{\phi }^{* }{10}^{-0.4({M}_{\mathrm{UV}}-{M}_{\mathrm{UV}}^{* })(\alpha +1)}\\ & & \times \ \exp (-{10}^{-0.4({M}_{\mathrm{UV}}-{M}_{\mathrm{UV}}^{* })}).\end{array}\end{eqnarray} \tag{ 7 }$$

We fit the Schechter function in terms of absolute magnitude to the PC UVLF using the χ² minimization method. We show the best-fit Schechter function in Figure 4 and the parameters in Table 1. Compared to the best-fit parameters of the field UVLF in previous studies (Yoshida et al. 2006; van der Burg et al. 2010; Bouwens et al. 2015; Ono et al. 2018), our PC UVLF has a less steep faint-end slope, as shown in Figure 5. Our best-fit ${M}_{\mathrm{UV}}^{* }$ is consistent with that of the field UVLFs at the 68/95% confidence level. This implies that the PC UVLF has a different shape compared to the field UVLF, although the discrepancy between our PC UVLF and the best-fit Schechter function is large, particularly at the bright end (M_UV < −23).

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Table 1.  The Best-fit Parameters and the Reduced χ² of the Schechter and DPL Functions Fitted to the PC UVLF

${M}_{\mathrm{UV}}^{* }$	ϕ*	α	β	${\chi }_{\nu }^{2}$
(mag)	(Mpc−3)
Schechter function
$-{20.61}_{-0.14}^{+0.12}$	${0.48}_{-0.02}^{+0.02}$	$-{0.16}_{-0.25}^{+0.25}$	⋯	11.2
Double power-law function
$-{21.13}_{-0.04}^{+0.04}$	${0.31}_{-0.01}^{+0.01}$	$(-0.16)$	$-{3.59}_{-0.11}^{+0.08}$	5.5

Note. We fix the faint-end slope in the case of the DPL to the best-fit value in the case of the Schechter function.

Download table as:  ASCIITypeset image

The large reduced χ² shown in Table 1 indicates this failure to fit the bright end. This may be because the PC UVLF does not seem to have a clear exponential decrease at the bright end. Therefore, we try to fit another functional form. Recent UVLF studies of field galaxies at higher redshifts (z ≥ 4) have suggested that the galaxy UVLF can be well described by a double power-law (DPL) function (e.g., Ono et al. 2018; Bowler et al. 2015, 2020). The DPL function is defined as follows:

$$\begin{eqnarray}&&\phi (L){dL}={\phi }^{* }\ {\left[{\left(\displaystyle \frac{L}{{L}^{* }}\right)}^{-\alpha }+{\left(\displaystyle \frac{L}{{L}^{* }}\right)}^{-\beta }\right]}^{-1}\displaystyle \frac{{dL}}{L* },\end{eqnarray} \tag{ 8 }$$

where β represents the power-law slope at the bright end (${M}_{\mathrm{UV}}\lt {M}_{\mathrm{UV}}^{* }$). We fit this function in terms of absolute magnitude, also. We fix the faint-end slope α to be the same as that of the best-fit Schecter function. We also show the best-fit DPL function in Figure 4 and its parameters in Table 1. The DPL function fits better than the Schechter function, even though the best-fit DPL function still has some deviation from the observed PC UVLF at M_UV < 23.

The excess over the best-fit Schechter/DPL function of the UVLFs of field galaxies is often explained by the presence of AGNs. Ono et al. (2018) claim that the gap between the UVLFs of field galaxies and their best-fit Schechter function at z ∼ 4–7 is explained by the contribution of AGN UVLFs at the same redshift. Also, Konno et al. (2016) construct the Lyα luminosity function of LAEs at z = 2.2 and argue that the gap between it and its best fit at the brightest end is due to AGNs. We discuss the possible contribution from AGNs in Section 5.3 and do not reject the possibility that the gap in both best-fit results is due to AGNs. However, we cannot conclude which function represents the galaxy UVLF more precisely. Therefore, we use both fitting functions in the following sections.

We estimate the SMF based on the measured PC UVLF, assuming that all protocluster g-dropout galaxies are located on the star formation main sequence of field galaxies at the same redshift. We utilize the main sequence estimated by Song et al. (2016), who determine the main sequence by applying an SED-fitting analysis to field photo-z selected galaxies from Finkelstein et al. (2015). We assume the main sequence is equivalent between protoclusters and the field, which is supported by observational studies (e.g., Koyama et al. 2013; Shi et al. 2019b; Long et al. 2020) and a theoretical study (e.g., Lovell et al. 2020), though some studies report a large contribution from starburst galaxies in protoclusters (e.g., Miller et al. 2018), leading to the possibility of a different main sequence from that of field galaxies.

We use the "constant-scatter galaxy SMF" method, which is conducted in some previous studies (e.g., Song et al. 2016). First, M_UV is randomly assigned. The probability distribution for each M_UV is approximated by the PC UVLF, in which Gaussian random errors for each bin are assigned, whose 1σ is equivalent to that of the observed PC UVLF. The M_UV is converted into the stellar mass M_* according to the M_*–M_UV relation of Song et al. (2016) with a constant scatter of 0.4 dex, and finally, the stellar mass distribution is obtained. This procedure is repeated 1000 times, and the SMF of protocluster galaxies (PC SMF) is obtained by taking their average. The uncertainty of the SMF is taken from the variation among 1000 results. The SMF of field galaxies (the field SMF) is also estimated from the field UVLF of Ono et al. (2018) in the same manner. We find that the estimation of SMFs has only a negligible change within the uncertainty when we use the main sequence of Tomczak et al. (2016), which has a flatter massive end ($\mathrm{log}({M}_{* }/{M}_{\odot })\gt 10.5$), compared to the main sequence of Song et al. (2016), as shown in Figure 6.

Zoom In Zoom Out Reset image size

Figure 6 shows our SMF estimates. We normalize them to fix their values at $\mathrm{log}({M}_{* }/{M}_{\odot })=10$ for easy comparison. The gray shaded region ($\mathrm{log}({M}_{* }/{M}_{\odot })\lt 9.72$) in Figure 6 shows the incomplete mass range due to the limiting magnitude (M_UV > −20.3). We hereafter discuss the SMF in the stellar mass range of $\mathrm{log}({M}_{* }/{M}_{\odot })\gt 9.72$. The PC SMF shows a clear excess from that of field galaxies toward the massive end, suggesting that protoclusters contain a relatively high fraction of massive galaxies compared to the field. Here, we mention three notes. First, this SMF only includes g-dropout galaxies, which are typically star forming, and we do not consider quiescent galaxies. Recent studies report the existence of massive quiescent galaxies even at z ∼ 4 in the blank field (e.g., Tanaka et al. 2019; Valentino et al. 2020), but the fraction of them is expected to be small (<5%) according to field SMFs (e.g., Davidzon et al. 2017), though the value in overdense environments is uncertain. Therefore, we ignore the effect of quiescent galaxies. Second, the bend of the PC UVLF around M_UV < −23 is not seen in the PC SMF. This is because the SMF is estimated from the main sequence with a constant scatter, which is the so-called "Eddington Bias." Third, the most massive end ($\mathrm{log}({M}_{* }/{M}_{\odot })\gt 11.15$) is dominated by objects with M_UV ≤ −23. As we mention in Section 5.2, objects in such a magnitude range can be AGNs; therefore, values of the SMF in this mass range can have uncertainty.

We fit the Schechter function to the measured PC SMF as well as to the field SMF at z ∼ 4. We can see that the PC SMF has a higher characteristic stellar mass and faint-end slope than does the field SMF, as seen in Figure 7. Protocluster galaxies have about 2.8 times higher characteristic stellar mass than do field galaxies. This also supports the result that protocluster galaxies are more massive than field galaxies.

Zoom In Zoom Out Reset image size

The difference between the PC SMF and the field SMF is also seen in simulations at z ∼ 4 (Muldrew et al. 2015; Lovell et al. 2018). In Figure 6, we compare our PC SMF and field SMF with those predicted in Lovell et al. (2018). Lovell et al. (2018) use the semi-analytical model (SAM) from Henriques et al. (2015) and trace the evolutionary track of halos with ${M}_{200}/{M}_{\odot }\gt {10}^{14}$ at z ∼ 0 to higher redshifts. M₂₀₀ is the mass within r < r₂₀₀, where the density is 200 times the critical density. We use their predicted SMFs as constructed from galaxies with $\mathrm{SFR}\gt 5\ {M}_{\odot }\ {\mathrm{yr}}^{-1}$ at z = 3.10 and 3.95. The average redshift of our protocluster sample is between the redshifts of these predicted SMFs. Our SMF is found to be almost consistent with the theoretical predictions and is located between the predicted SMF at z = 3.95 and that at z = 3.10. Though the PC SMF has a higher amplitude than the theoretical prediction at the most massive end ($\mathrm{log}({M}_{* }/{M}_{\odot })\gt 11.15$), this can be explained by the contribution of AGNs, as mentioned above.

We compare our PC SMF with those of (proto)cluster galaxies at lower redshifts. Shimakawa et al. (2018) estimate an SMF of HAEs in a protocluster called USS1558–003 at z ∼ 2.5. Nantais et al. (2016) focus on four galaxy clusters at z ∼ 1.5 from the Spitzer Adaptation of the Red-sequence Cluster Survey (SpARCS; Muzzin et al. 2009; Wilson et al. 2009; Demarco et al. 2010). van der Burg et al. (2013) present an SMF of galaxies of 10 rich clusters in the Gemini Cluster Astrophysics Spectroscopic Survey (GCLASS) at 0.86 < z < 1.34. The SMF of galaxies in 21 clusters detected with the Planck satellite at 0.5 < z < 0.7 is also presented in van der Burg et al. (2018). Calvi et al. (2013) estimate an SMF of cluster galaxies from the WIde-field Nearby Galaxy-cluster Survey (WINGS) at 0.04 ≤ z ≤ 0.07 (Fasano et al. 2006), and compare with that of field galaxies at the same redshift. Figure 8 shows our PC SMF with other SMFs and the field SMF. As in Figure 6, we normalize the amplitude of all the SMFs at $\mathrm{log}({M}_{* }/{M}_{\odot })=10$. This is because the definition of the (proto)clusters' volume varies by study, making amplitude comparison difficult. Therefore, we only focus on the shape differences of these SMFs. We also convert their assumed IMF to a Salpeter IMF, which is used in Song et al. (2016).

Zoom In Zoom Out Reset image size

We can see that there is a dearth of massive galaxies in the SMF of our protoclusters at z ∼ 4 compared to those at lower redshifts. This suggests that our protoclusters at z ∼ 4 are still in the process of mass growth. Particularly, from z ∼ 4 (HSC–SSP protoclusters) to z ∼ 1 (van der Burg et al. 2013), SMFs shows a monotonic growth at the massive end. At z ∼ 0–1, the ratio of SMFs at the massive end to that at the low-mass end decreases toward lower redshifts. This may be due to the significant contribution of less massive infalling galaxies. We discuss it in more detail in Section 5.4.

We note that these SMFs are based on galaxy clusters selected by different methods. They might be at different stages of the evolution of clusters (Toshikawa et al. 2020), which may make it difficult to compare them with each other. Moreover, the protocluster samples of this study and Shimakawa et al. (2018) only focus on star-forming galaxies, while others contain quiescent galaxies. The fraction of quiescent galaxies at z > 2 is known to be smaller than that at lower redshift, so we ignore the effects of this difference. Also, as mentioned in Section 2.1, our protocluster candidates are overdense regions expected to evolve into clusters with $\langle {M}_{\mathrm{halo}}\rangle ={4.1}_{-0.7}^{+0.7}\times {10}^{14}\,{h}^{-1}\,{M}_{\odot }$ at z ∼ 0. The majority of clusters from WINGS are as massive as M₂₀₀ ∼ (1–10) × 10¹⁴ M_⊙ (Biviano et al. 2017), which is the same mass range as for the expected halo masses of our protoclusters. On the other hand, the cluster halo mass of other studies is M₂₀₀ ∼ 3 × 10¹⁴ M_⊙ for SpARCS (Lidman et al. 2012) and GCLASS (van der Burg et al. 2013), and M₂₀₀ ∼ (3–13) × 10¹⁴ M_⊙ in van der Burg et al. (2018). These clusters are already as massive as WINGS clusters, even at z ∼ 1, so they may grow more by z ∼ 0, making their comparison to the WINGS clusters and our sample difficult. In addition, the halo mass of USS1558–003 is not estimated; therefore, it is still under debate whether HSC–SSP protoclusters at z ∼ 4 are progenitors of protoclusters such as USS1558–003.

Our protocluster sample has some variation in terms of overdensity. As shown in Figure 1 of Uchiyama et al. (2018), the overdensity of protoclusters ranges from 4σ to 9.5σ, and the overdensity and descendant halo mass are broadly positively correlated (Toshikawa et al. 2016). Here, we make subsamples of protoclusters according to the overdensity and construct UVLFs for each subsample.

We divide protocluster samples into four groups according to their overdensity δ: 4σ ≤ δ < 5σ; 5σ ≤ δ < 6σ; 6σ ≤ δ < 7σ, 4); and 7σ ≤ δ. The numbers of protoclusters in each subgroup are 120, 37, 13, and 7, respectively. In Figure 9, we show the PC UVLF for each subsample. The amplitude of the faint end (M_UV > −21.2) is almost the same among subsamples, while the amplitude at the bright end (M_UV < −21.2) depends on the overdensity of protoclusters. More overdense protoclusters tend to have a higher bright-end amplitude compared to less massive protoclusters. These protoclusters can be more spatially extended, which could cause such a dependency on overdensity; however, we find that this is unlikely, as discussed in Section 5.2.

Zoom In Zoom Out Reset image size

The dependency of the bright-end excess on overdensity can be seen even for each protocluster separately. Figure 10 shows the cumulative UVLF of galaxies in each protocluster. The bright-end amplitude of more overdense protoclusters tends to be higher than those of less massive protoclusters, suggesting that protoclusters with higher overdensity significance values have brighter objects. More interestingly, almost all of the protoclusters at z ∼ 4 have this excess at the bright end compared to field galaxies, and the variation is seen even if we focus on only protoclusters with the same overdensity. Therefore, we conclude that the bright-end excess is ubiquitously seen for protoclusters at z ∼ 4.

Zoom In Zoom Out Reset image size

In Ito et al. (2019), we investigate the significantly UV-brightest galaxies (proto-BCGs) in this protocluster sample. We find that galaxies in protoclusters containing proto-BCGs are brighter than other protocluster galaxies. This can be due to the overdensity dependence of the bright-end excess because the average overdensity of protoclusters containing proto-BCGs is slightly higher ((5.068 ± 0.149)σ) than that of all protoclusters ((4.767 ± 0.069)σ). To determine the cause of the bright-end excess, we divide a subgroup, which is made in this subsection, into two according to whether protoclusters contain proto-BCGs. At a fixed overdensity, the UVLF of members of protoclusters containing proto-BCGs has the same bright-end amplitude as those of protocluster not containing proto-BCGs. The brighter galaxies of protoclusters containing proto-BCGs are therefore due to the higher overdensities of these protoclusters.

We estimate the SFRD of protocluster galaxies, based on a combination of the PC UVLF and the far-IR (FIR) luminosity density. The PC UVLF is approximated by the best-fit Schechter/DPL function. Parameter spaces with a 68% confidence level as estimated in Section 3.5 are employed for the PC UVLF.

We first estimate the UV luminosity density ρ_UV from the PC UVLF as ${\rho }_{\mathrm{UV}}={\int }_{{L}_{\mathrm{faint}}}^{{L}_{\mathrm{bright}}}\,{L}_{\mathrm{UV}}\phi ({L}_{\mathrm{UV}}){{dL}}_{\mathrm{UV}}$. We set L_faint = 2.7 × 10²⁷ erg s⁻¹ Hz⁻¹, corresponding to M_UV = −17 mag, which is the same as that applied in Bouwens et al. (2015), and L_bright = 1.1 × 10³¹ erg s⁻¹ Hz⁻¹, corresponding to M_UV = −26 mag.

The FIR (8–1000 μm) luminosity density ρ_FIR is estimated as ${\rho }_{\mathrm{FIR}}={\int }_{{L}_{\mathrm{faint}}}^{{L}_{\mathrm{bright}}}\,{L}_{\mathrm{FIR}}\phi ({L}_{\mathrm{UV}}){{dL}}_{\mathrm{UV}}$ with the use of the IRX–β–M_* relation of z ∼ 3 LBGs (Álvarez-Márquez et al. 2019). The β−M_UV relation is known to exist even in protocluster galaxies at z ∼ 4 (Overzier et al. 2008). The β distribution is determined by using the conversion equation from i–y color to β in Bouwens et al. (2012). We linearly fit the median value of the β distribution in each 0.2 mag magnitude bin of M_UV ≤ −20.3. We use its best-fit parameters with their 1σ errors for the β−M_UV relation. We also estimate the β–M_UV relation of our field galaxies in the same manner and compare it with the literature in Appendix C. Our estimation is consistent with the literature within the uncertainty, suggesting that our measurements and the sample selection are robust. The stellar mass M_* is estimated from the UV absolute magnitude in the same method described in Section 4.1, with the correction of the IMF from a Salpeter IMF to that used by Álvarez-Márquez et al. (2019; a Chabrier 2003 IMF) by dividing the stellar mass by 1.74. From the β– M_UV relation and the estimated stellar mass, L_UV is converted into L_FIR.

We derive the average ρ_UV and ρ_FIR weighted by the likelihood obtained in the fitting. We employ their minimum and maximum values to estimate the errors by varying the parameters of the UVLF/β–M_UV relation in the range of their 16th and 84th percentiles, respectively. As a result, we estimate the UV and FIR luminosity densities of HSC–SSP protocluster galaxies as ${\rho }_{\mathrm{UV}}\,={3.46}_{-0.29}^{+0.35}\times {10}^{28}$ $({3.53}_{-0.16}^{+0.17}\times {10}^{28})$ $\mathrm{erg}\ {{\rm{s}}}^{-1}\ {\mathrm{Hz}}^{-1}\ {\mathrm{Mpc}}^{-3}$, and ${\rho }_{\mathrm{FIR}}={1.7}_{-0.9}^{+0.9}\times {10}^{11}$ $({2.5}_{-1.0}^{+1.8}\times {10}^{11})$ ${L}_{\odot }\,{\mathrm{Mpc}}^{-3}$ in the case of the Schechter (DPL) function, respectively.

Kubo et al. (2019) conduct a stacking analysis of FIR images taken from Planck, AKARI, IRAS, and Herschel at the position of HSC–SSP protoclusters, which is the same sample used in this study. Based on their best-fit of the SED model composed of star, dust, and AGN flux components, the total FIR luminosity from all galaxies per protocluster is inferred to be ${L}_{\mathrm{FIR}}={1.3}_{-1.0}^{+1.6}\,\times {10}^{13}\,{L}_{\odot }$. In the case of the SED model without the AGN component, it is estimated as ${L}_{\mathrm{FIR}}={19.3}_{-4.2}^{+0.6}\times {10}^{13}\,{L}_{\odot }$. As mentioned in Kubo et al. (2019), the best-fit L_FIR has a degeneracy between two cases, so the uncertainty is quite large. Considering this point and the effective volume of our protoclusters, our estimation of ρ_FIR is consistent with these estimations.

To derive the SFRD, we apply the conversion equation from Kennicutt (1998) to ρ_UV and ρ_FIR, as described below:

$$\begin{eqnarray}&&\mathrm{SFRD}=1.73\times {10}^{-10}{\rho }_{\mathrm{FIR}}+1.4\times {10}^{-28}{\rho }_{\mathrm{UV}}.\end{eqnarray} \tag{ 9 }$$

As a result, our protocluster galaxies are estimated to have an SFRD corresponding to ${\mathrm{log}}_{10}\mathrm{SFRD}/({{\rm{M}}}_{\odot }\,{\mathrm{yr}}^{-1}\ {\mathrm{Mpc}}^{-3})\,={1.54}_{-0.20}^{+0.16}$ $({1.68}_{-0.17}^{+0.16})$ in the case of the Schechter (DPL) function. This value is roughly ∼2.5 dex higher than that of field galaxies (e.g., ${\mathrm{log}}_{10}\mathrm{SFRD}/({{\rm{M}}}_{\odot }\,{\mathrm{yr}}^{-1}\ {\mathrm{Mpc}}^{-3})=-1.00\,\pm 0.06$ in Bouwens et al. 2015), suggesting that our protocluster regions have active star formation.

Previous studies estimate the SFRD of field LBGs by assuming the IRX–β relation of local starburst galaxies in Meurer et al. (1999). For reference, the SFRD of our protocluster members estimated with this IRX–β relation is ${\mathrm{log}}_{10}\mathrm{SFRD}/({{\rm{M}}}_{\odot }\,{\mathrm{yr}}^{-1}\ {\mathrm{Mpc}}^{-3})$ = ${1.61}_{-0.45}^{+0.33}\ ({1.71}_{-0.31}^{+0.26})$ in the case of the Schechter (DPL) function, which is consistent with the original result.

Next, we estimate the fraction of the cosmic SFRD from progenitors of massive halos (M_halo > 10¹⁴ M_⊙). We convert the estimated SFRD, which is per unit volume of the protocluster, to that per unit of cosmic volume, and divide it by the field SFRD. The field SFRD is taken from Bouwens et al. (2015); (${\mathrm{log}}_{10}\mathrm{SFRD}/({{\rm{M}}}_{\odot }\,{\mathrm{yr}}^{-1}\ {\mathrm{Mpc}}^{-3})=-1.00\,\pm 0.06$). Using other estimates (e.g., Bouwens et al. 2009; van der Burg et al. 2010) changes the result by only ∼0.1 dex.

In addition, our protocluster sample is not complete for all progenitors of halos with M_halo > 10¹⁴ M_⊙ at z ∼ 0. Some fraction of dark matter halos with an overdensity below 4σ at z ∼ 4 will also evolve into such halos. We can identify such progenitor halos in the simulation of Toshikawa et al. (2018, 2016). The fraction of halos that can be observed by our protocluster selection with a galaxy overdensity significance greater than 4σ at z ∼ 4 is about 6.2% ± 1.0%, suggesting that our sample has a very high purity but low completeness. The fraction of halos can be translated to the fraction of member galaxies based on the overdensity distribution of progenitor halos, which is equivalent to 9.67% ± 0.41%. Most of the nonobserved member galaxies should be hosted by progenitor halos whose overdensity significance is less than 4σ. With a simple assumption that the UVLF of these galaxies is the same as our PC UVLF, we can derive the intrinsic contribution of progenitors of massive halos to the cosmic SFRD by dividing by this completeness. We mention that the shape of the PC UVLF depends on the overdensity, but the main difference of the shape occurs at M_UV < −22, which does not significantly affect the SFRD measurement.

Moreover, 76% of the objects in our protocluster sample are expected to evolve into galaxies with M_halo > 10¹⁴ M_⊙ at z ∼ 0 (T18), so we correct the purity by multiplying by this ratio. Finally, we estimate that the ${9.4}_{-3.4}^{+4.7} \%$ (${13.9}_{-4.9}^{+6.5} \%$) of the cosmic SFRD occurs in progenitors of massive halos in the case when we use the best-fit of the Schechter function (the DPL function).

We compare this measurement with the prediction from the SAM in Chiang et al. (2017). They focus on galaxies with $\mathrm{log}({M}_{* }/{M}_{\odot })\gt 8.5$ in progenitors of clusters with M₂₀₀ > 10¹⁴ M_⊙ at z ∼ 0, and estimate that the contribution of protocluster galaxies is about 24% and 19% at z ∼ 4 when they use the Henriques et al. (2015) and Guo et al. (2013) SAM, respectively.

The comparison between the observed and predicted fraction of protocluster galaxies to the cosmic SFRD is shown in Figure 11. Our result is close to the theoretical prediction but slightly smaller. There are two possible explanations. First, we only focus on UV-bright galaxies and miss some other galaxy populations, such as SMGs, which are not selected by LBG selection. Though it is not yet clearly understood how much we miss such galaxies by g-dropout selection, Wang et al. (2019) argue that the optically dark but submillimeter-bright galaxies have a significant contribution to the cosmic SFRD. Marrone et al. (2018) report two SMGs are close to each other, implying that they are located in a massive halo. Also, some studies report highly overdense regions of SMGs (e.g., Miller et al. 2018). Although the FIR luminosity of galaxies from Kubo et al. (2019) has a large degeneracy depending on the SED model they use, the SFRD combined with the UV luminosity density estimated in this work and the stacked FIR luminosity from Kubo et al. (2019) are consistent with the theoretical prediction for the SFRD within the uncertainty. This FIR luminosity, estimated from the stacking, includes the contribution of SMGs, so this does not reject that SMGs may be one of the reasons for the difference between the measured and prediced SFRD values. Second, we may miss some members located on the outskirts of more massive protoclusters. This is because we define protocluster members according to the predicted size of the progenitor of "Fornax-type" clusters, which can be small for progenitors of more massvie clusters, like "Coma-like" clusters.

Zoom In Zoom Out Reset image size

We note that even if we estimate the SFRD using the best-fit PC UVLF in the magnitude range of M_UV < −19, which corresponds to $\mathrm{log}({M}_{* }/{M}_{\odot })\gt 8.5$ according to Song et al. (2016), the result does not change significantly.

We have evaluated the sample incompleteness in the same manner as most of the other studies of field LBGs (Section 3.2), and find that it is consistent with previous studies by comparing it with the field UVLF. However, another possible incompleteness could be caused by object confusion in crowded regions, such as in protoclusters. In some overdense regions, some fraction of galaxies will be mixed with nearby objects, which could lower the completeness. Our finding of a flatter UVLF in protoclusters than in the field UVLF could be due to this confusion effect, which might more significantly affect fainter galaxies. The luminosity function shape could change, as seen in this study.

We check for this effect by inserting mock galaxies into an overdense region to compare the completeness function in overdense regions with that in the blank field, as estimated in Section 3.2. We summarize the detailed procedure in Appendix D and find that there is no additional incompleteness due to object confusion in regions with an overdensity significance up to ∼8σ.

Though we see a deficit in the faint end of the PC UVLF compared to the field UVLF, the cause of this will be further investigated in future studies after we construct a protocluster sample in the Deep layer of the HSC–SSP, which has deeper image than the Wide layer. However, we now see that the completeness function of g-dropout galaxies estimated in this study is consistent with that of previous studies and that the blending due to focusing on overdense regions like HSC–SSP protoclusters does not lower the completeness. These results imply that the deficit is at least not due to incompleteness.

We have selected protocluster members from galaxies located within a distance of 18 from each overdensity peak. Since protoclusters with more significant overdensity levels tend to be more extended, we may miss some protocluster members on the outskirts of protoclusters, and this could lead to the bright-end excess. To examine this possibility, we redefine protocluster members as galaxies that are located within 42 from the overdensity peak, which corresponds to the size of progenitors of only the most massive halos (M_halo > 10¹⁵ M_⊙), like the Coma cluster at z ∼ 4. We find that the shape of the PC UVLF does not change from the case of 18, suggesting that the trend is not caused by the differences in the typical spatial dimensions of protoclusters of different masses. We also check the case of a smaller protocluster radius (∼1') and find that the trend does not change.

Recent studies have argued that the bright end (M_UV ≤ −23.0) of the UVLF at z ∼ 4 is mainly dominated by AGNs (e.g., Ono et al. 2018; Adams et al. 2020). Here, we discuss how well the contribution due to the AGNs can explain the bright-end excess that we find in the PC UVLF for M_UV ≤ −20.8.

First of all, we compare our PC UVLF to the field quasar UVLF. Akiyama et al. (2018) construct the quasar UVLF at z ∼ 4. The number density of quasars based on the best-fit DPL function for the magnitude range of −25.8 < M_UV < −20.8, which is the range where our PC UVLF has an excess, is about (0.9–10) × 10⁻⁷ Mpc⁻³ mag⁻¹. This value is (1–240) × 10³ times lower than the excess at the bright end that we find in the study. In addition, we have found that UV-luminous quasars scarcely exist in the protoclusters at z ∼ 4 (H. Uchiyama et al. 2020, in preparation), suggesting that the number density of luminous quasars in protoclusters should not be larger than that in the field.

The difference between the PC UVLF and the field UVLF in the magnitude range of M_UV ≤ −20.8 corresponds to 16 objects per protocluster. The expected total number of members in a protocluster is about 50, indicating that the bright-end excess corresponds to about 32% of the total protocluster members. If we assume that all of the excess at the bright end is due to AGN, such a high AGN fraction in protoclusters is inconsistent with the findings of previous studies. For example, Toshikawa et al. (2016) perform follow-up spectroscopy observations for protocluster member candidates, and they do not find any AGN in 11 members in a protocluster at z ∼ 3, suggesting that the AGN fraction is less than 9%. Assuming the same upper limit for the AGN fraction, the expected number of AGNs in a protocluster is fewer than five out of 50 members. Other studies show similar AGN fractions for protoclusters from X-ray counterparts. Lehmer et al. (2009) estimate an AGN fraction of ${9.5}_{-6.1}^{+12.7}$ percent for LBGs in the SSA22 protocluster at z = 3.09. Macuga et al. (2019) estimate the AGN fraction as ${2.0}_{-1.3}^{+2.6}$ percent for HAEs in the USS1558–003 protocluster at z = 2.53. Krishnan et al. (2017) investigate AGNs in a protocluster called Cl 0218.3−0510 at z = 1.62 and estimate that the AGN fraction of massive ($\mathrm{log}({M}_{* }/{M}_{\odot })\gt 10$) protocluster galaxies is ${17}_{-5}^{+6}$ percent. Although they argue that this value is high compared to that of the blank field at the same redshift, it is not enough to explain the bright-end excess of our PC UVLF. It should be mentioned that the AGN fraction estimated from X-ray detections can be sensitive to their depth, but these comparisons imply that protoclusters at z ∼ 4 are less likely to host such a large amount of UV-bright AGNs.

We note that residuals at M_UV < −23.0 of the PC UVLF from the best-fit of the Schechter (DPL) function correspond to 1.5 or 0.5 objects per protocluster. These seem to be reasonable values for the AGN fraction in a protocluster; therefore, a part of the bright-end excess can be contributed by the AGN.

Therefore, we conclude that AGNs are unlikely to explain all of the bright-end excess in the PC UVLF. It should be noted that we here discuss the UV-bright AGNs, and we do not include obscured AGNs. As mentioned in Section 4.3, Kubo et al. (2019) stack IR images of various surveys and estimate the total FIR luminosity of the same protocluster sample used in this study. Their results indicate that HSC–SSP protoclusters can include a population of UV-dim AGNs.

Some studies suggest that star formation is enhanced in overdense regions at high redshift compared to that in the blank field, as we mentioned in Section 3.4. For example, HAEs in protoclusters at z ∼ 2–2.5 show an enhancement of high SFR galaxies (Koyama et al. 2013; Shimakawa et al. 2018). In addition, Shi et al. (2019a) report tentative evidence of a higher SFR for Lyα emitting galaxies in protoclusters at z = 3.13. Saito et al. (2015) also report the enhancement of the bright end excess of Lyα luminosity function in a protocluster region around a radio galaxy at z = 4.11, which can be interpreted as a signature of an enhanced star formation activity. On the other hand, local galaxy clusters show the opposite trend. For example, cluster galaxies at 0.18 < z < 0.55 have SFRs from roughly 0.00 ± 0.11 h⁻² M_⊙ yr⁻¹ to 0.17 ± 0.02 h⁻² M_⊙ yr⁻¹, which are always lower than those of field galaxies (Balogh et al. 1998). Similarly, low star formation activity in a cluster is also reported at z = 1.6 (Kurk et al. 2009). Combining our results with those from the literature, the enhancement of SFR in overdense environments has already started at z ∼ 4, and the star formation activity drops some time between z ∼ 0 and z ∼ 2, which is earlier than for field galaxies. This is supported by the fact that massive quiescent galaxies have rapidly emerged in overdense regions in the era from z ∼ 2.5 to z ∼ 1.5 (e.g., Newman et al. 2014; Cooke et al. 2015; Wang et al. 2016).

Focusing on the stellar mass, there are several reports that there are more massive galaxies in protoclusters at z ∼ 2–3 (Hatch et al. 2011; Koyama et al. 2013; Cooke et al. 2014; Shimakawa et al. 2018), similar to our results at z ∼ 4. At lower redshift (z < 1.5), the situation is controversial. Many studies report that the shape of the SMFs of star-forming and quiescent galaxies in clusters are similar (e.g., Calvi et al. 2013; van der Burg et al. 2013; Lin et al. 2017), while for those of all cluster galaxies, it is argued that there are significant differences not only in the normalization but also in shape of the SMF at z ∼ 1 in van der Burg et al. (2013), at z ∼ 0.5–0.7 in van der Burg et al. (2018), and at z ∼ 0 in Balogh et al. (2001). In addition, Kovac et al. (2010) report a difference between the SMFs of galaxies in a group environment and those in the blank field. On the other hand, Calvi et al. (2013) suggest that the shape of the SMF is independent of the environment for z ∼ 0, as do Nantais et al. (2016) for z ∼ 1.5.

It should be noted that some studies report almost no difference from field galaxies in terms of the SFR and stellar mass of protocluster galaxies at z = 2.9 (Cucciati et al. 2014), and at z = 4.57 (Lemaux et al. 2018). These studies are based on only spectroscopically confirmed members, which are free from contamination; however, the sample of members is small (∼10 objects) and may not reveal the differences that we find in this study.

These comparisons suggest that galaxies in overdense regions are more massive and have more active star formation compared to galaxies in the blank field at z > 1.5. At lower redshift, these trends change: galaxies in overdense regions have lower SFR, and their SMF can be identical to that of the field at least when focusing on the same galaxy population. In addition, star-forming galaxies in protoclusters tend to be located at the main sequence at z ∼ 4 (Shi et al. 2019b; Long et al. 2020) and z ∼ 2–2.5 (Koyama et al. 2013; Shimakawa et al. 2018). This means that the majority of protocluster members are normal galaxies, and the starburst activity is not significant. Therefore, these results may point to the earlier onset of star formation in protoclusters.

This early formation scenario is consistent with theoretical predictions. Chiang et al. (2017) suggest three phases for the evolution of (proto)clusters. Galaxies in protoclusters already begin star formation in an "inside-out" manner from z ≥ 10 to z ∼ 5. Then, they continue star formation from z ∼ 5 to z ∼ 1.5. At z ≤ 1.5, star formation in galaxies is finished, and infalling galaxies into (proto)clusters dominate the main stellar mass growth in protoclusters. Such infalling galaxies are one of the possible reasons why the differences between SMFs of galaxies in local clusters disappear (Vulcani et al. 2013). The steeper SMFs for cluster galaxies at lower-z seen in Section 4.1 can also be explained by the effect of infalling galaxies. In addition, they also imply that ∼20% of the cosmic SFRD is contributed by protocluster galaxies, which is roughly consistent with our estimation, as discussed in Section 4.3.

The differences in shape between the PC UVLF and the PC SMF seen in this study can also be related to frequent mergers or an increase in gas supply toward the center of the connection of several connected filaments in an overdense region, as suggested in Shimakawa et al. (2018). Indeed, Tomczak et al. (2017) show that "top-heavy" SMFs may originate from the enhancement of mergers in overdense regions. They first construct SMFs for star-forming galaxies and quiescent galaxies at z ∼ 1 subdivided by their local environment. They find that shapes of the SMFs in more overdense regions tend to be more top heavy. They try to explain this trend by a simple semiempirical model. This model first generates ∼10⁶ galaxies at z = 5. For each redshift slice, some fraction of galaxy pairs are selected for the merger, and some fraction of galaxies are selected for quenching. The only free parameter is the merged galaxy fraction. The model shows that the observed SMF in overdense regions can be explained by a high merger rate (80–90%). In addition, the increase of gas supply can allow galaxies that are too massive to be star-forming galaxies in the blank field to maintain star formation. This effect also makes the SMF of protocluster galaxies, which consists only of star-forming ones, top heavy.

We find in Section 4.2 that all protoclusters follow the same trend that galaxies in more massive overdense regions tend to have a flatter UVLF, though diversity exists even if we focus on protoclusters with the same overdensity. The trend implies that more massive regions have generally experienced earlier structure formation, but their evolutionary stage has a significant variation even at the same epoch. This indicates that a large sample at each redshift is critically essential for tracing the general evolutionary sequence of protoclusters within this diversity.