A Census of Galaxy Constituents in a Coma Progenitor Observed at z > 3

We use the PSFEx software (Bertin 2011) to measure the point-spread function (PSF) of each image out to a radius of 3''. The two-dimensional PSFs are radially averaged to obtain the circularized PSF. Taking the worst-seeing data (K_S band) as the target PSF, we derive the noiseless convolution kernel for each image using the IDL routine MAX_ENTROPY. We use the full shape of the observed stellar profiles rather than assuming a function form, such as Moffat profiles. The details of the PSF-matching procedure are given in Xue et al. (2017). All optical and near-IR images are convolved with the appropriate kernels to create a set of PSF-matched science images.

Source detection and photometric measurements in the B_W RIyHK_S bands are carried out by running the SExtractor software (Bertin & Arnouts 1996) in dual mode on the PSF-matched images with the K_S-band data as the detection band. At the protocluster redshift (z = 3.8), the K_S band mainly samples the continuum emission at the rest frame ≈4400 Å. The SExtractor parameter MAG_AUTO is used to estimate the total magnitude, while colors are computed from fluxes within a fixed isophotal area (i.e., FLUX_ISO). The colors measured in FLUX_ISO and FLUX_APER are in agreement with each other within 0.1 mag. As the images are PSF-matched, aperture correction is constant in all bands and is given by the difference between the MAG_AUTO and MAG_ISO estimated in the K_S band.

For the Spitzer IRAC images, we take a different approach as it is not practical to convolve all images to the FWHM of any IRAC PSF, which is much broader (∼2''). We use the TPHOT software (Merlin et al. 2015), which performs "template-fitting photometry" similar to TFIT (Laidler et al. 2007; Lee et al. 2012). The software uses the information (source shape and position) supplied by higher-resolution data and simultaneously fits the fluxes of multiple nearby sources to minimize residual flux. Given that the FWHM of the K_S band's PSF is not negligible compared to that of the IRAC data, we also derive the convolution kernel using the same procedure above. For the effective PSF of the IRAC bands, we rotate the published IRAC PSF by a series of position angles with which the SDWFS data were taken, and create a weighted-average image.

Finally, all photometric catalogs are merged together to create the final multiwavelength catalog, where the TPHOT-measured fluxes are considered identical to the MAG_ISO fluxes of the optical/near-IR bands. Given the completeness of the K_S-band data, we only consider sources that have signal-to-noise ratios (S/N) greater than 10, roughly corresponding to a K_S,AB magnitude of 24.0 mag. The final multiwavelength catalog contains 27,845 sources.

We derive photometric redshifts with the CIGALE code (Noll et al. 2009) using the full photometric information. The reliability of the photo-z estimates is evaluated using existing spectroscopic sources, which target a subset of UV-bright galaxies satisfying the LAE or Lyman-break galaxy (LBG) color selection over a 1.2 × 0.6 deg² contiguous region in the PC 217.96+32.3 field. The details of these selection methods in our survey field are discussed in Lee et al. (2013) and Lee et al. (2014). Of the 164 sources at z_spec = 3.4–4.2, 48 galaxies lie within the NEWFIRM coverage. Of those, only 17 galaxies are bright enough to be detected in the K_S-band catalog with a photo-z estimate.

We find a redshift dispersion σ_z/(1+z_spec) = 0.15, where σ_z is the standard deviation of Δz (≡z_spec−z_phot). The large dispersion is due to three outliers which have (z_spec−z_phot)/(1+z_spec) > 0.2. We find that their redshift probability density functions (PDFs) have two peaks, one at z < 1 and another at z ∼ 4; given that they are fainter than other galaxies, redshift degeneracy is caused by the fact that the spectral break between B_W and R is not strong enough to be unambiguously determined as a Lyman break. However, for all three galaxies the probability of lying at z = 3.4–4.2 (computed by integrating the photometric redshift PDF in the interval, which we denote as p_z) is greater than 50%. Excluding these three galaxies, the redshift dispersion σ_z/(1+z) is 0.06.

After considering the photometric redshift constraints of our spectroscopic sources, we select protocluster candidate galaxies by requiring that z_phot = 3.4–4.2 for sources whose redshift PDFs are singly peaked, and p_z ≥ 0.5 for those with doubly peaked PDFs. All of the 17 spectroscopic members meet these criteria. The range z_phot = 3.4–4.2 is chosen based on the photometric redshift error as discussed previously. A similarly inclusive range was used by Kubo et al. (2013), who studied the stellar populations in and around another protocluster. After visual inspection, we remove sources with potential contamination in the photometry, including those that are too close to brighter sources or to the edges of the images. Our protocluster galaxy sample consists of 263 sources, which also include the spectroscopic members of the structure.

We examine the rest-frame UV colors of our protocluster candidates to assess their overall similarities to broadband color-selected LBGs. We match their positions to the I band–selected photometric catalog used for the B_W-band dropout selection (Lee et al. 2013, 2014). Of the 263 sources, 202 galaxies (77%) are detected in the I band with S/N ≥ 7. In Figure 2, we show their locations on the B_W − R versus R − I color diagram. Of the 202 galaxies, 135 galaxies (67%) satisfy the formal LBG criteria, and an additional 22 galaxies (11%) are within 0.25 mag of the formal B_W − R color cut. The galaxies outside the selection window tend to be fainter in the R and I bands (R ≲ 25.5) while dropping out of the B_W band, which results in a weaker constraint on the B_W − R color. However, their UV colors are generally similar to those of their UV-brighter cousins. Their R − I colors are redder than those within the LBG selection criteria, which likely contribute to the weaker spectral break in the B_W band. Thus, we conclude that our photo-z estimate works relatively well for moderately dust-obscured star-forming galaxies whose spectral energy distributions (SEDs) are similar to those of LBGs.

Zoom In Zoom Out Reset image size

We determine the physical properties of the photo-z protocluster candidates using the CIGALE software. We use the stellar population synthesis models of Bruzual & Charlot (2003), Calzetti et al.'s (2000) reddening law with A_V values ranging from 0 to 5 in steps of 0.1 mag, the solar metallicity, and Salpeter's (1955) initial mass function. We carry out three separate runs assuming a constant star formation history (SFH), an exponentially declining SFH with τ values of 50 Myr to 10 Gyr in steps of 200 Myr, and a delayed star formation model. In Figure 3 (two left columns), we show the best-fit SED model, population parameters, and redshift PDF (inset) for a subset of our sample. As can be seen in the figure, these photo-z candidates are mostly star-forming galaxies with a strong Lyα break and relatively blue UV slope. The best-fit SFR implies they are actively forming stars at a relatively high level (up to several hundreds of solar mass per year) with a moderate amount of dust (E(B − V) ∼ 0.1–0.2).

Zoom In Zoom Out Reset image size

As discussed in Section 2.2, the photometric redshift technique is most effective in selecting LBG-like galaxies. Here, we use a set of color selection criteria tuned to isolate galaxies with a strong Balmer/4000 Å break, a feature strongest in old stellar populations dominated by A and F stars. In this work, we use the following color criteria, which are similar to those found in the literature (e.g., Franx et al. 2003; Labbé et al. 2005; Kajisawa et al. 2006; Brammer & van Dokkum 2007; Wiklind et al. 2008; Huang et al. 2011; Nayyeri et al. 2014; Mawatari et al. 2016):

$$\begin{eqnarray*}&&H-{K}_{S}\gt 1.2;[3.6]-[4.5]\lt 0.5.\end{eqnarray*}$$

The first condition imposes that a strong Balmer/4000 Å break fall between the H and K_S bands, which would occur in the redshift range z = 3.6–4.2. Using the EZGAL software¹⁶ (Mancone & Gonzalez 2012) with the stellar population synthesis models of Bruzual & Charlot (2003), we compute the H − K_S colors of the stellar population as a function of age, assuming three families of SFHs: (1) an instantaneous burst; (2) constant SFHs (CSFs); and (3) exponentially declining τ models ($\mathrm{SFR}\propto \exp [-t/\tau ]$) with τ values of 0.1 Gyr and 0.5 Gyr. As illustrated in the left panel of Figure 4, a stellar population formed via a single instantaneous burst would meet this condition at age 250 Myr, while galaxies formed through a more extended star formation episode (τ = 100 Myr) would take ≈400 Myr to attain the same strength.

Zoom In Zoom Out Reset image size

The second criterion requires that the continuum slope at λ_rest = 7000–9000 Å be relatively flat, ensuring that the red H − K_S color is not due to dust reddening. In the middle panel of Figure 4, we illustrate the effect of interstellar dust assuming reddening parameters E(B − V) = 0, 0.5, 1.0 and the Calzetti et al. (2000) extinction law.

Using the above criteria, 56 galaxies are identified. Thirteen of them have power law–like SEDs in the mid-infrared with typical brightness of ≈21 AB in the 5.8 μm or 8.0 μm band; the H − K_S colors range from 1.2 to 1.3, on the low end of the color distribution. Four of them are also present in the Spitzer MIPS 24 μm source catalog provided by Vaccari (2015). These sources are likely to be heavily dust-obscured AGNs which scatter into our selection. Of the 13 galaxies, seven (54%) and eight (62%) meet the IRAC color criteria for high-redshift AGN selection proposed by Stern et al. (2005) and Donley et al. (2012), respectively. We remove all 13 galaxies from our sample.

The final sample consists of 43 galaxies, which we refer to as Balmer-break galaxy (BBG) candidates hereafter. Seven galaxies are also our photo-z protocluster candidates. In the rightmost panel of Figure 4, we show the H − K_S and [3.6]–[4.5] colors of all K_S band–detected sources with reliable color measurements. BBGs without (with) a photo-z estimate are shown in red (green), while the distribution of the remainder is indicated as grayscale and contours, where the contour lines enclose 68% and 95% of all the galaxies.

Most BBGs are very faint at observed optical wavelengths (i.e., faint at rest-frame UV wavelengths). Of the 43 galaxies, only seven (16%) are detected at the 5σ level (R ≤ 26.2 AB) in the R band while dropping out of the B_W band. The three brightest galaxies (in the R band) formally meet the LBG color criteria. The remaining four likely have SED shapes similar to those of their R-brighter counterparts but are simply too faint to place strong enough constraints on the B_W − R colors. All seven have photo-z probability distributions with a single peak at z > 3.5. The remaining 36 galaxies (84%) are formally undetected in the R band with a few detected at a lower significance.

In the K_S band, the BBGs have a mean $\langle {K}_{S}\rangle =22.94\pm 0.37$ AB (median = 22.92), significantly brighter than the photo-z members, which have $\langle {K}_{S}\rangle =23.44\pm 0.40$ (median = 23.50). In Figure 5, we show the K_S-band distribution of the photo-z (solid gray) and BBG candidates (hatched), where the BBGs are further split based on optical detection (labeled "UV-faint" or "UV-bright"). The disparity between their K_S-band brightness is driven by a selection effect: the IRAC color cut applied to the latter requires that they be bright enough in both the IRAC 3.6 and 4.5 μm bands.

Zoom In Zoom Out Reset image size

We construct the median SED for each BBG subsample by creating median image stacks and measuring the aperture photometry (see Lee et al. 2011 for a full description of the stacking procedure). The CIGALE software is run in the same manner as done for individual galaxies. The SED fitting results are summarized in Table 1. We find that optically faint BBGs are well fit by old stellar populations with little to no star formation, while the remaining seven galaxies are best fit as young post-starburst systems. We refer to the two groups as "quiescent" and "post-starburst" BBGs, respectively. In the next two subsections, we discuss each category in further detail.

Table 1.  Physical Properties of BBGs (Stacked Photometry)

SFH	Quiescent	Two Populations	Post-starburst	CSF
	exp. decl.	$={C}_{\mathrm{old}}\exp [-(t-{t}_{\mathrm{old}})/{\tau }_{\mathrm{old}}]$	exp. decl.	=const
	$\propto \exp [-t/\tau ]$	+ ${C}_{\mathrm{new}}\exp [-(t-{t}_{\mathrm{new}})/{\tau }_{\mathrm{new}}]$	$\propto \exp [-t/\tau ]$
zphot	3.58 ± 0.37	3.96 ± 0.26	3.95 ± 0.26	3.95 ± 0.26
$\mathrm{log}[{M}_{\mathrm{star}}/{M}_{\odot }]$	11.20 ± 0.07	10.99 ± 0.09	10.99 ± 0.10	10.95 ± 0.09
SFR (M⊙ yr−1)	0 ± 2	114 ± 61	110 ± 69	172 ± 68
Age (Myr)	984 ± 324	395 ± 141	405 ± 154	358 ± 127
E(B − V)	0.08 ± 0.08	0.16 ± 0.04	0.16 ± 0.05	0.19 ± 0.03
τ (Myr)	50	100, 300	500	$\infty$
fnewa	⋯	≲0.1	⋯	⋯
${\chi }_{r}^{2}$	6.39	5.22	5.29	5.38

Note.

^aThe fraction of stellar mass formed in the second burst relative to the older population.

Download table as:  ASCIITypeset image

In Figure 6, we show sample postage-stamp images of the quiescent BBGs. Most of the quiescent BBG candidates are detected only in three or four bands; the limited dynamic range in wavelength coverage and the shallow depths in the IRAC 5.8 and 8.0 μm bands result in poorly constrained photometric redshift estimates. While we return to the issue of redshift degeneracy later in this section, we fix the redshift of all quiescent BBGs to z = 3.8 in deriving their physical parameters, which is motivated by the redshift of the protocluster in the field. Changing the redshift by Δz = ±0.1 would result in a 5% change in mass.

Zoom In Zoom Out Reset image size

Twelve BBGs show an excess flux in the 5.8 and 8.0 μm bands, suggesting possible contamination by warm dust emission possibly arising from a hidden starburst or AGN. When we exclude the 5.8–8.0 μm data from the SED fitting and refit their masses, the change in stellar mass is minimal (6%). This is consistent with the expectation based on the infrared SEDs of high-redshift starburst/AGN systems that the flux contribution by AGNs at λ_rest ≤ 1–2 μm is not significant (e.g., Sajina et al. 2012; Kirkpatrick et al. 2015). The median value of the individual stellar mass measurements is $\mathrm{log}[{M}_{\mathrm{star}}/{M}_{\odot }]$ = 11.30 (σ = 0.29), consistent with that obtained from the stacked photometry (Table 1). The four most massive galaxies lie in the range $\mathrm{log}[{M}_{\mathrm{star}}/{M}_{\odot }]$ = 11.7–11.9 (see Figure 3); if confirmed, their masses already rival those of some of the brightest cluster galaxies in the local universe.

The stacked SED of the UV-faint BBGs is consistent with an old (980 Myr) and very massive ($\approx 2\times {10}^{11}\,{M}_{\odot }$) galaxy with little star formation. When combined with the best-fit photometric redshift at z_phot = 3.58, the formation redshift is at z_f ≈ 6. While similarly massive and old galaxies have been reported in the literature (Marchesini et al. 2010; Nayyeri et al. 2014; Glazebrook et al. 2017), the presence of such massive galaxies in large numbers may pose a considerable challenge to the hierarchical theory of galaxy formation.

The redshift PDF of the stacked SED (Figure 7, left inset) is singly peaked at z ≈ 3.6, strongly ruling out a lower-redshift (z < 3) solution. This gives us confidence that our quiescent galaxy sample is not dominated by heavily obscured lower-redshift sources. However, the photo-z constraints on individual galaxies are more ambiguous (Figure 3). Only 11 of the 36 galaxies have a singly peaked PDF at z > 3; the remainder show a rather flat z distribution or have two peaks. For the latter, the low-redshift solution typically lies at z_phot = 1.0–1.5, and the high-z solution lies at z_phot = 3.5–4.0.

Zoom In Zoom Out Reset image size

The color degeneracy between an old quiescent galaxy at a high redshift and a very dusty galaxy at a lower redshift is well known. Dunlop et al. (2007) reanalyzed the photometric data of a putative massive and quiescent galaxy at z = 6.5 named HUDF-JD2 (Mobasher et al. 2005) and showed that a very dusty (A_V = 3.8) galaxy at z ∼ 1.5–2.5 is equally likely. The galaxy was later detected in the 16 μm and 22 μm bands, lending further credence to the lower-z solution (z ≈ 1.7; Chary et al. 2007). Similarly, Marchesini et al. (2010) selected a sample of massive galaxies at z = 3–4 using the photometric redshift technique and noted that nearly half are equally well fit by very old and very dusty (A_V ≈ 3) galaxies at z < 3 if such models are included in the spectral template set.

In the redshift range to which our BBG selection is sensitive, the strongest constraint comes from the IRAC 5.8 μm and 8.0 μm photometry. For galaxies at z ≲ 1.5, these bands sample beyond the 1.65 μm stellar bump which arises from the declining H⁻ ion opacity in the stellar atmosphere (see, e.g., Sawicki 2002). Indeed, when we repeat the SED fitting procedure while limiting the redshift range to z < 2, the best-fit solution is an old and heavily reddened galaxy at z ≈ 1.2 (A_V = 3.1 ± 0.4, 2.5 ± 1.4 Gyr), which is shown in Figure 7 (left). Given the similarity of the rest-frame UV and optical colors of the two model fits, it is evident that flux measurements in the 5.8 and 8.0 μm bands are important in breaking the redshift degeneracy.

Indeed, all of the BBGs with 8.0 μm detection have singly peaked redshift PDFs. In the image stack of the remaining 25 galaxies, we do not obtain a clear detection, and as a result, the redshift PDF is doubly peaked, confirming our expectation. However, the nondetection is not surprising considering the sensitivity of the SDWFS data (the 5σ limit for a point source is 20.25 mag). If we assume Poisson noise (i.e., the most optimistic case), stacking 25 sources will result in a limiting magnitude of 22.0, which is very close to the measured 8.0 μm flux from the full stack (see Figure 7). Thus, the nondetection does not rule out the possibility that these 25 galaxies have SEDs similar to those of their 8.0 μm brighter counterparts but with slightly lower fluxes.

As a final check, we repeat our image stacking, photometry, and SED fitting procedure 200 times while each time randomly excluding seven BBGs (20% of the sample). We integrate the redshift PDF above z = 3 to obtain the formal probability P₃ for the high-redshift solution. In 73% of the time, the photometric redshift solution prefers the high-z solution (P₃ ≥ 0.5). We conclude that the redshift ambiguity of the BBGs is mainly driven by the existing depth at the 8.0 μm band and that deeper data will be necessary to place more stringent constraints on their redshift distribution.

Finally, we note that several studies have reported a significant fraction of MIPS 24 μm detections among massive quiescent galaxies (Mancini et al. 2009; Marchesini et al. 2010; Nayyeri et al. 2014; Marsan et al. 2015). At z = 3.0–4.5, the 24 μm samples λ_rest ≈ 4–6 μm, where warm–hot dust continua or polycyclic aromatic hydrocarbons excited by star formation or AGN activity could contribute significantly to the flux. The exploration of such possibilities offers a promising avenue to learn about how "red and dead" galaxies form and what roles AGN activity and nuclear starbursts play in the process. We also note a submillimeter (ALMA+SCUBA-2) detection a fraction of an arcsecond away from a confirmed post-starburst galaxy has been reported recently (Glazebrook et al. 2017). However, given the shallow MIPS coverage in the Boötes field (the 5σ detection limit is 250 μJy), we are unable to quantify what fraction of our BBG candidates may harbor hidden AGNs or starbursts.

The relatively strong Lyman break present in the seven optically bright BBGs places their redshift in the range z_phot = 3.6–4.0, giving us confidence that the H and K_S bands straddle the Balmer/4000 Å break. The overall chi-square distribution obtained from our SED fitting procedure suggests that either delayed or exponentially declining SFH models with relatively short τ values (100–300 Myr) are preferred over CSF models, which typically return larger ${\chi }_{r}^{2}$ values. The median fit values are $\mathrm{log}[{M}_{\mathrm{star}}/{M}_{\odot }]$ = 11.0 (σ = 0.2) for stellar mass, 145 (σ = 42) M_⊙ yr⁻¹ for SFR, and 433 Myr (σ = 23 Myr) for population age. In comparison, the CSF model returns higher values of the SFR, 205 $(\sigma =67)\,{M}_{\odot }\,{\mathrm{yr}}^{-1}$, but similar stellar masses and ages. These values are also consistent with the stacking results shown in Table 1.

In Figure 7 (right), we show the stacked photometry together with the best-fit SEDs assuming CSF (blue) and exponentially declining (red) models. The overall SED shapes are very similar in the entire range of the rest-frame UV-to-IR wavelengths with the exception of the K_S band sampling the rest-frame 4500 Å. A zoom-in on the wavelength range near the Balmer/4000 Å break is shown in the far right panel.

We also consider a scenario in which the galaxies are composed of two stellar populations formed at different times where the old population dominates the rest-frame optical emission, while the UV emission originates from newly formed stars (e.g., Kriek et al. 2009). We explore a range of "double-burst" models as follows: The SFHs of both populations are modeled as exponentially declining functions with τ values ranging across τ = 10–1000 Myr. The ages of the two populations are also allowed to vary. The fraction of stellar mass formed in the second burst relative to the old population, f_new, is varied from 0.01 to 0.50. The minimum χ² is achieved at f_new ≈ 0.05, where a 200 Myr old new burst is currently forming stars at a rate of 114 ${M}_{\odot }\,{\mathrm{yr}}^{-1}$ (green line in Figure 7, right panel). The χ² values are similar out to f_new ≲ 0.1 but increase more rapidly at f_new ≥ 0.2 (Δχ² = 0.4 and 0.9 at f_new = 0.2 and 0.3, respectively). Thus, we conclude that the mass formed during the recent star formation episode is small (<10%) compared to that of the evolved stellar population. The stellar population parameters obtained for all three different SFHs are listed in Table 1.

Finally, we consider the possibility that the seven optically bright candidates are normal star-forming galaxies misclassified as BBGs due to the contamination of the K_S-band flux by strong nebular emission, such as [O iii] and Hα (Shim et al. 2011; Schenker et al. 2013; Stark et al. 2013). Of particular interest in relation to the present sample is the [O iii] λλ4959, 5007 doublet, which falls into the K_S band at z = 3.1–3.6. For all but two, the redshift PDFs peak at z ≥ 3.7 even though the majority have a nonzero probability of lying in the range z_phot = 3.5–3.6. For the remaining two, the peak of the PDF is at z ∼ 3.6. Schenker et al. (2013) measured the rest-frame [O iii] equivalent widths (EWs) and determined the median value of 280 Å (see also Holden et al. 2016). At z = 3.5, it would lead to a substantial overestimation of the K_S-band continuum flux by 0.37 mag.

However, Malkan et al. (2017) recently noted that a strong anticorrelation exists between the [O iii] EW and stellar mass such that the EW could be as low as 80 Å in ${M}_{\mathrm{star}}\approx {10}^{10}\,{M}_{\odot }$ galaxies. The median stellar mass of our UV-bright BBGs is nearly an order of magnitude larger than this value. Similarly, only two galaxies in the Schenker et al. sample have UV brightness similar to our sample's¹⁷ ; their EWs are 100 Å and 150 Å, corresponding to a much less severe contamination of Δm of 0.15 and 0.21 mag, respectively. The trend of decreasing EWs with increasing mass and UV luminosity is likely the same, given the relatively tight correlation between the two quantities among star-forming populations (e.g., Stark et al. 2009; Lee et al. 2012; Salmon et al. 2015).

We measure the mean [3.6]–[4.5] color to be 0.24 ± 0.16; Stark et al. (2013) reported that the median [3.6]–[4.5] color is ≈−0.23 mag for 3.8 < z < 5.0 galaxies, significantly bluer than the ∼0.1 mag color measured in their sample for the systems at 3.1 < z < 3.6. The color difference is attributed to the presence of strong Hα emission in the former. The lack of excess 3.6 μm band flux corroborates the possibility that nebular line contamination is not significant.

Given the color degeneracies between the above possibilities, discriminating a young post-starburst from a rejuvenated old galaxy will be harder, requiring detection of their respective spectroscopic signatures; these will include Balmer absorption lines for post-starburst galaxies (e.g., Kriek et al. 2009; Glazebrook et al. 2017) and nebular lines, such as [O ii], [O iii], Hβ, and Hα, from the H ii regions. Future James Webb Space Telescope spectroscopy will help resolve this issue unambiguously (Kalirai 2018).

Regardless of their nature, we have uncovered a rare population of ultramassive galaxies ($\gtrsim {10}^{11}\,{M}_{\odot }$) which may have recently halted their star formation or are nearing the end of their star formation activity.

A summary of all the 43 BBG candidates is given in Table 2.

We show the sky distribution of the photo-z protocluster candidates in the left panel of Figure 8; the surface density enhancement relative to the mean density is shown as both grayscale and contour lines. The true density enhancement is expected to be higher as the mean density computed from the entire field includes the galaxies in the overdense region. There is a clear indication of a large overdensity slightly south of the field center, outlined by the $1.5\bar{{\rm{\Sigma }}}$ and $1.7\bar{{\rm{\Sigma }}}$ lines. A smaller, less significant one is found north of the field center.

Zoom In Zoom Out Reset image size

In the same figure, the sky distribution of known members of PC 217.96+32.3 is shown in the middle panel; the spectroscopic sources (which include both LAEs and LBGs) are color-coded by redshift. The density contour of the protocluster is constructed as before, but only using the LAE positions. Because our spectroscopic efforts are heavily focused on the LAE overdensity region, including non-LAE members in the density calculation would artificially increase the overdensity.

Comparison of the density maps of the photo-z and protocluster LAEs shows that they are not co-spatial. We perform a two-dimensional Kolmogorov–Smirnov (K-S) test (Peacock 1983; Fasano & Franceschini 1987) to compare the photo-z distribution with the LAE distribution, and find a p-value of 2.9 × 10⁻⁷. Thus, it is extremely unlikely that they are drawn from the same parent distribution at random. The 2D K-S test was also used in Kuiper et al. (2012) to compare different structures.

The largest photo-z overdensity runs in the NW–SE direction. While it partially overlaps with the southern end of PC 217.96+32.3, it stretches further west to a region devoid of LAEs. A smaller and less significant overdensity lies just north of the main overdensity, also overlapping slightly with a small LAE group north of the main LAE overdensity. The larger overdensity is also closer to PC 217.96+32.3 and thus more likely to have a physical connection to the confirmed protocluster. Being separated from each other, the two photo-z overdensities have an unclear physical association. Thus, we focus on the larger overdensity in this work.

The shapes and locations of the photo-z and LAE surface overdensities are suggestive of a possible physical connection. One hypothesis is that they are part of a single structure where the photo-z overdensity lies in the foreground of the LAEs (i.e., at z < 3.76), and as a result any Lyα emission from galaxies in this region is missed by the LAE selection filter. In the right panel of Figure 8, we show the narrowband filter bandpass converted to the redshift selection function (dashed line) together with the distribution of all spectroscopic sources in the range of z = 3.70–3.90. Most of the known members residing in the LAE overdensity lie at z = 3.775–3.785, i.e., in the blue half of the filter response. The southern end of PC 217.96+32.3 is composed of galaxies in the redshift range where the filter response falls off steeply. Existing spectroscopy reveals that three LAEs there have the line centroids outside the narrowband filter but are selected as LAEs because of their high line luminosities and broad line widths. The high concentration of z ≈ 3.77 LAEs where the two overdensities overlap provides circumstantial evidence that the LAEs only partially trace the true extent of a single very large structure.

Another possibility is that the photo-z overdensity is located further in the foreground of PC 217.96+32.3 near an LBG overdensity at z = 3.72. Of the 10 galaxies at z = 3.721 ± 0.04 within our NEWFIRM coverage, three reside within the ${\rm{\Sigma }}=1.7\bar{{\rm{\Sigma }}}$ region, and an additional five lie just outside the ${\rm{\Sigma }}=1.5\bar{{\rm{\Sigma }}}$ contour line. The significance of this spectroscopic overdensity is difficult to assess given the limited extent and depth of the existing spectroscopy. All confirmed LBGs—including the eight galaxies at z ≈ 3.72—have either relatively strong Lyα emission or high UV continuum luminosities. Lending further support to this possibility is G6025 (large white circle in Figure 8), one of the eight galaxies that is unusually large (end-to-end length of ∼20 kpc; Lee et al. 2018). The ground-based morphology and large angular size are consistent with two UV-luminous galaxies involved in a major merger, a type of event that should occur more frequently in a dense environment.

Finally, the photo-z overdensity may represent a protocluster with no physical connection to PC 217.96+32.3. The stacked redshift PDF of protocluster candidates peaks at z ≈ 3.75, but its width is not narrow enough to rule out the possibility that the true peak may be Δz ≳ 0.1 away from the peak value, which would place the structure at ≳20 Mpc away from PC 217.96+32.3.

Multiple protoclusters in close proximity are unlikely but not impossible. Kuiper et al. (2012) noted that there may be two separate galaxy overdensities near MRC 0317-257, a radio galaxy at z = 3.13. Similarly, a string of galaxy overdensities in the COSMOS field has been found spanning over a line-of-sight distance of ∼25 Mpc (z = 2.42, 2.44, 2.47, and 2.51 as reported by Chiang et al. 2015; Diener et al. 2015; Casey 2016; Wang et al. 2016, respectively). With the limited spectroscopy, their physical connection remains unclear. Two additional LAE overdensities of smaller magnitudes exist just outside our NEWFIRM field north of PC 217.96+32.3 (Lee et al. 2014), one of which has been confirmed to be a galaxy overdensity (Dey et al. 2016a). In Section 5.5, we return to this topic and evaluate the likelihood of multiple protoclusters in our survey volume.

Several previous studies have reported a high concentration of galaxies dominated by old stellar populations near known massive protoclusters (Steidel et al. 2005; Kubo et al. 2013, 2015; Wang et al. 2016). While those galaxies still await spectroscopic confirmation, such information would have important implications to the formation histories of massive cluster ellipticals. In this context, we investigate whether our BBGs are physically associated with the structure revealed by the photo-z overdensity.

In Figure 9 we show the locations of the 43 BBG candidates overlaid on the photo-z (left) and LAE (right) density maps. The BBGs seem to avoid the most overdense regions of both the LAEs and the photo-z candidates. Only one quiescent galaxy candidate lies near the LAE core, and two additional sources are near the $3{\bar{{\rm{\Sigma }}}}_{\mathrm{LAE}}$ line. Only three BBG candidates are located near the $2{\bar{{\rm{\Sigma }}}}_{\mathrm{photoz}}$ contour line. The relative void of all types of galaxies potentially associated with the structure in the southern corner and northern end of the field is also noteworthy. The fact that the same regions are well populated with lower-redshift sources perhaps suggests that the void is not artificially created by the presence of bright sources such as saturated stars or large galaxies.

Zoom In Zoom Out Reset image size

We perform a 2D K-S test using the BBG and LAE distributions and find a p-value of 3 × 10⁻⁴, indicating a significant disparity between the spatial distributions of the two samples. The same test using the BBG and photo-z distributions results in a p-value of 0.05. The K-S test evaluates the similarity of two univariate samples by constructing their cumulative distributions and computing their maximum distance. Because multivariate samples can be ordered in more than one way, multidimensional K-S tests lack the statistical rigor of a 1D test and thus need to be understood in the context of carefully controlled tests. To this end, we populate the survey field with two a priori known distributions and perform the 2D K-S test to quantify the range of the p-values. For each test, we create 1000 separate realizations.

First, we create two random distributions, each matching the number of BBGs and photo-z sources. We obtain a mean (median) p-value of 0.30 (0.27) with a standard deviation (σ) of 0.21. Second, we compare the photo-z sample with a randomly chosen subset of it consisting of 43 sources (matching the number of BBGs), which results in a p-value of 0.37 (0.36, σ = 0.21). These tests suggest that we can reliably rule out the possibility that the LAEs and BBGs—having 3 × 10⁻⁴—are drawn from the same parent distribution.

The relationship between the BBGs and photo-z candidates, however, is less clear with p = 0.05. Comparison of the photo-z distribution with 43 randomly distributed sources yields a p-value of 0.11 (0.07, σ = 0.12), comfortably bracketing the measured p-value. Thus, we cannot statistically rule out the possibility that the BBG positions are not correlated with those of the photo-z members. Spectroscopic redshifts are necessary for progress.

We assess the significance of the structure by estimating the range of the true galaxy overdensity given the observed level of surface density enhancement. The transverse size of the photo-z overdensity is computed by interpolating the $1.5\bar{{\rm{\Sigma }}}$ isodensity contour, which yields 139 arcmin² or 26.4 Mpc² (physical) at z = 3.78. Given that the physical scale remains constant within 2% at z = 3.65–3.85, our subsequent estimate of the overdensity and masses should be relatively insensitive to the precise redshift of the structure. We assume that the line-of-sight distance from the front to the back of the structure is 15 Mpc; this is motivated by the fact that the effective diameter of the progenitors of massive present-day clusters lies in this range (Chiang et al. 2013). The redshift distribution of the known members of PC 217.96+32.3, which ranges over z = 3.77–3.79, is consistent with this expectation (see the right panel of Figure 8).

We infer the range of the intrinsic galaxy overdensity by performing Monte Carlo simulations as follows. In each run, we create a mock field containing one protocluster with a galaxy overdensity δ_g in the middle by populating points randomly in the (α, δ, z) space. The "protocluster region" is defined as a rectangle. The overall number of sources and the transverse area of the protocluster match those of the data. An intrinsic galaxy overdensity δ_g is chosen at random in the range δ_g = 1–20. We divide the redshift range [3.4, 4.2] into 40 bins with a bin size of Δz = 0.02, corresponding to the step size of 15 Mpc in the comoving line-of-sight distance. Taking δ_g as an intrinsic overdensity, the number of true members is N_proto = (1+δ_g)N_phot/(40+δ_g), where N_phot is the total number of observed protocluster candidates in the field, and they are populated at random within the protocluster region. Setting δ_g = 10 (5) means that 58 (35) galaxies are part of the structure. The remainder (N_phot − N_proto) are assigned randomly assuming a uniform distribution in both transverse and line-of-sight positions. We construct the surface density map of the mock image using the identical procedure described previously and estimate the mean surface density enhancement within the protocluster region.

We repeat the above procedure 10,000 times and obtain the empirical relation between the true overdensity and the surface overdensity. The scaling relation is well behaved and nearly linear. Given the observed surface overdensity (the mean enhancement is 1.81 within the $1.5\bar{{\rm{\Sigma }}}$ isodensity contour), we estimate that the intrinsic overdensity of the structure is δ_g = 5.5–10.2 with a median value of 7.8. The value is comparable to the redshift overdensities found for several known protoclusters. Steidel et al. (2005) measured a redshift overdensity of δ_g ∼ 7 for a z = 2.30 structure. Based on the VIMOS Ultra Deep Survey (Le Fèvre et al. 2015), Lemaux et al. (2014) and Cucciati et al. (2014) reported inferred redshift overdensities of δ_g = 10.5 ± 2.8 and δ_g = 12 ± 2 for protoclusters at z = 3.28 and z = 2.90, respectively. These values are larger than that determined for the SSA 22 protocluster at z = 3.09, δ_g ∼ 3.5–4.0 (Steidel et al. 1998, 2000; Hayashino et al. 2004; Matsuda et al. 2005; Yamada et al. 2012; Topping et al. 2018).

To investigate how sensitive the inferred δ_g value is to the transverse area of the surface overdensity, we rerun the simulations using the isodensities of $1.3\bar{{\rm{\Sigma }}}$ and $1.6\bar{{\rm{\Sigma }}}$; lowering the density contrast effectively increases the effective area, while raising it has the opposite effect. We find that the estimate of the underlying overdensity is relatively insensitive to a specific choice of density contrast used to estimate the extent of the structure. However, our test does not include the possibility that the surface overdensity is systematically overestimated due to either Poisson shot noise or a superposition of another, unrelated group of galaxies. Given the lack of spectroscopy in the region, we currently have no way of quantifying this possibility. If the surface overdensity region is reduced by 20%, the δ_g value will decrease to 3.9–7.0.

Based on the inferred galaxy overdensity δ_g, we estimate the descendant mass of the underlying structure, i.e., the total mass enclosed within the overdense region which will be gravitationally bound and virialized by z = 0, which can be expressed as:

$$\begin{eqnarray}&&{M}_{z=0}=(1+{\delta }_{m})\langle \rho \rangle V\end{eqnarray} \tag{ 1 }$$

where $\langle \rho \rangle$ is the average matter density of the universe ($=[3{H}_{0}^{2}/8\pi G]{{\rm{\Omega }}}_{0}$), δ_m is the matter overdensity, and V is the comoving volume of the galaxy overdensity. With the adopted cosmology, Equation (1) is equivalent to ${M}_{z=0}=[3.67\,\times {10}^{10}\,{M}_{\odot ]}{(1+{\delta }_{m})[V/(1\mathrm{cMpc})}^{3}]$. The two overdensity parameters, δ_g and δ_m, are related through Equation (1)+bδ_m = C(1+δ_g), where C denotes a factor correcting for the effect of redshift-space distortions (Steidel et al. 1998) and b is galaxy bias. Given the lack of details to assume otherwise, we use C in the case of spherical collapse: $C{({\delta }_{m},z)=1+{{\rm{\Omega }}}_{m}^{4/7}(z)[1-(1+{\delta }_{m})}^{1/3}]$. As for galaxy bias, we adopt b = 3.5. Our choice is justified by the fact that the majority of our photo-z sources lie in an observed UV luminosity range comparable to that of $L\gtrsim {L}_{\mathrm{UV}}^{* }$ LBGs at z = 3–4. The bias value of the latter has been estimated through measurements of their clustering properties (e.g., Ouchi et al. 2004; Lee et al. 2006; Hildebrandt et al. 2009). We solve the above equations for δ_g and use Equation (1) to obtain the mass estimate.

The enclosed mass in the photo-z structure is $(7.9\,\pm 1.0)\times {10}^{14}\,{M}_{\odot }$ given the overdensity δ_g of 7.8 ± 2.4. The inferred dark matter overdensity is δ_m = 1.39 ± 0.3. Increasing the bias value to b = 4 would decrease the mass by 10%.

We evaluate how the number of massive quiescent galaxies ($\geqslant {10}^{11}\,{M}_{\odot }$) in our field compares with that expected in an average field. Based on K_S-selected galaxies in the 1.6 deg² COSMOS/UltraVISTA field, Muzzin et al. (2013) estimated that at z = 3–4, the cumulative number density of galaxies with ${M}_{\mathrm{star}}\geqslant {10}^{11}\,{M}_{\odot }$ is $({1.4}_{-0.5}^{+2.2})\times {10}^{-6}$ Mpc⁻³. In our survey field (28' × 35'), one expects to find ${2.5}_{-0.8}^{+3.9}$ BBG-selected quiescent galaxies. Similarly, Spitler et al. (2014) identified six quiescent galaxies above $M\geqslant {10}^{11}\,{M}_{\odot }$ in the ZFOURGE survey corresponding to the surface density of 0.015 ± 0.006 arcmin⁻², such that 3.7 ± 1.5 quiescent galaxies are expected in our field. We assume in the above calculations that the selection function takes the form of a top-hat filter in the range z = 3.6–4.2, where the H − K_S color samples the Balmer/4000 Å break. The relative change of angular diameter distance in this range is 6% and should result in 12% in the expected number depending on the redshift distribution of BBGs.

Taking the Muzzin et al. (2013) measurement as the field average, the implied overdensity of massive quiescent galaxies is δΣ_BBG ∼ 16. Excluding all of our post-starburst BBG candidates (assuming all are strong [O iii] emitters at z ∼ 3.4), the remaining BBGs correspond to δΣ_BBG ≈ 13. Using the Spitler et al. (2014) estimates, the overdensity is δΣ_BBG = 11 (9) with (without) the potential [O iii] emitters.

We also compare the observed abundance of quiescent galaxies with that measured in the SSA 22 protocluster at z = 3.09. Kubo et al. (2013) used color criteria tuned to z ∼ 3 (i' − K > 3, K − [4.5] < 0.5, and K < 23) and identified 11 massive galaxies ($\gtrsim {10}^{11}\,{M}_{\odot }$) concentrated near the overdensities of other types of galaxies with a surface density of 0.10 ± 0.03 arcmin⁻². In comparison, the overall surface density of BBGs in our field is 0.06 ± 0.01 arcmin⁻². Within a smaller rectangular region (15' × 16') in which the surface density of photo-z sources is enhanced by 50% (Figure 9, left), we find 21 quiescent BBGs, corresponding to the surface density of 0.09 ± 0.02 arcmin⁻². All errors are given assuming Poisson shot noise. Considering the change of angular diameter distance, the surface density per unit comoving transverse area is 0.027 ± 0.008 Mpc⁻² and 0.021 ± 0.004 Mpc⁻² for SSA 22 and the present structure, respectively. Similarly, Lemaux et al. (2014) estimated that the implied overdensity of massive ($\geqslant {10}^{10.8}\,{M}_{\odot }$) red galaxies in a z = 3.29 protocluster is δ_g = 25.1 ± 15.2.

A large population of massive quiescent galaxies found in our field implies that the formation of cluster galaxies occurred in shorter timescales and at earlier times than those of the field galaxies. Our results confirm an early onset of the cluster red sequence (e.g., Kodama et al. 2007; Lemaux et al. 2014). This is in broad agreement with the SFHs of present-day cluster ellipticals inferred from absorption line studies (Thomas et al. 2005). Little to no evolution of the cluster red sequence out to z ∼ 1.4 further strengthens this expectation (e.g., Blakeslee et al. 2003; Mei et al. 2006).

The sky distribution of BBGs appears to trace the full extent of the large-scale structure rather than being concentrated in the highest-density environments. Few are found in either LAE or photo-z overdensity peaks (see Section 4.2). We speculate that BBGs may be the central (and most massive) inhabitants of massive halos that are in the process of merging. The implication is that they were quenched long before the final coalescence of the structure, which occurred much later. Therefore, the quenching of massive cluster ellipticals is caused by the early onset of "mass quenching" rather than by any environmental effect suppressing their formation (Peng et al. 2010). This is in line with a study of intermediate-redshift galaxy clusters by Brodwin et al. (2013), who found that the level of star formation in cluster environments declines below that in the average field only at z ≲ 1.4 (see also Tran et al. 2010). Recent discoveries of compact galaxy groups in protocluster environments support this view, as the fraction of quiescent galaxies in such a group is observed to be low (e.g., Kubo et al. 2016; Wang et al. 2016).

In this work, we identify protocluster member candidates by employing two selection methods, namely photo-z-selected star-forming galaxies and BBGs. When combined with a population of LAEs in the same field (Lee et al. 2014; Dey et al. 2016a), these samples showcase diverse types of inhabitants residing in a very overdense cosmic structure.

In Figure 10, we show the SFR, stellar mass, and specific SFR (sSFR) values measured for our sample galaxies. The estimates for the photo-z candidates are made on individual galaxies. As for the quiescent BBG candidates, we fix the redshift to z = 3.8 for the SED fitting (see Section 3.2 for a discussion on redshift degeneracy); for the UV-bright BBGs with robust photo-z estimates, we fix the redshift to the best-fit value.

Zoom In Zoom Out Reset image size

As for the LAEs, while they have robust redshift estimates, they are too faint at infrared wavelengths to yield robust estimates of stellar population parameters on an individual basis. Instead, we perform image stacking on their positions and measure the parameters based on the aperture photometry on the stacked images. A total of 150 LAEs are used for stacking analysis after removing those too close to nearby bright sources. To estimate the range of their physical parameters, we randomly draw a subset of the LAEs and perform an image stacking, aperture photometry, and SED fitting procedure. Their distributions of stellar population parameters shown in Figure 10 are based on 2500 such realizations. Since median stacking is insensitive to significant outliers, the distribution of their physical parameters should be taken as a lower limit rather than the full range spanned by the LAEs.

The sample galaxies span a wide range of SFRs and stellar masses: the lack of overlap is at least in part driven by the selection effect. The lack of photo-z candidates at ${M}_{\mathrm{star}}\,\lesssim {10}^{10}\,{M}_{\odot }$ is tied to the sensitivity of our K_S-band data. A 10σ detection (K_S,AB = 24.0) corresponds to the rest-frame optical luminosity of a z = 3.8 galaxy with stellar mass $\approx {10}^{10.2}\,{M}_{\odot }$, assuming an exponentially decaying SFH with a τ value of 0.5 Gyr. The paucity of galaxies with $\mathrm{SFR}\lesssim 50\,{M}_{\odot }\,{\mathrm{yr}}^{-1}$ is also driven by the same mass limit, given the correlation between SFR and M_star. The large median mass of the BBGs is driven by the IRAC color selection as discussed in Section 3.1. The steep decline in the number of galaxies at $\mathrm{SFR}\gtrsim 150\,{M}_{\odot }\,{\mathrm{yr}}^{-1}$ (e.g., Smit et al. 2012) is likely further helped by the photo-z selection, which is biased against redder (dustier) galaxies than typical LBGs. The intrinsic distribution of these parameters spanned by different types of galaxies remains uncertain: such information will require careful analyses of deeper multiwavelength data and the modeling of their respective selection biases, which are outside the scope of this paper.

The measured overdensities of the different galaxy types highlight how they trace the same underlying large-scale structure(s). Such measures are more robust against any of the selection biases mentioned previously as any such bias should apply equally to field and cluster galaxies and thus should minimally impact their spatial distributions. The observed surface overdensity of photo-z galaxies is δΣ_phot ≈ 1.5, similar to that of the LAEs over the same general area. However, we show in Section 4.3 that the spatial overdensity of the photo-z galaxies is much larger, δ_g = 7.8 ± 2.4, than that of the LAEs. This is because the former are distributed over a much larger line-of-sight distance (i.e., larger Δz), and as a result, their surface overdensity is substantially diluted by interlopers. It is also possible that the narrowband Lyα filter "misses" the core of the protocluster and is only picking up the outer parts of the protocluster. In comparison, the surface overdensity of BBGs in the region is much higher at δΣ_BBG ≈ 9–16.

If all the types of galaxies we consider here (LAEs, BBGs, and photo-z candidates) trace the same underlying structure represented by a matter overdensity δ, the implication would be that more massive BBGs are far more biased tracers of the matter distribution than less massive star-forming galaxies. Our findings are consistent with the expectation from existing clustering studies that more luminous/massive galaxies have larger biases (e.g., Giavalisco & Dickinson 2001; Ouchi et al. 2004; Adelberger et al. 2005; Lee et al. 2006; Gawiser et al. 2007; Guaita et al. 2010; Kusakabe et al. 2018).

One corollary to the luminosity/mass-dependent bias is that, everything being equal, low-mass low-bias galaxies such as LAEs are the least biased (and thus most reliable) tracers of the density distribution within the large-scale structure. Using LAEs to "map out" the protocluster environment has additional advantages, including the relative ease of redshift identification through the narrowband selection technique and the abundance of low-luminosity galaxies implied by the steep faint-end slope of the UV luminosity function at this redshift range (e.g., Bouwens et al. 2007; Reddy & Steidel 2009; Alavi et al. 2016; Malkan et al. 2017). Given the difficulties of obtaining spectroscopic redshifts for faint distant galaxies, LAEs offer the best practical means to survey the local environment of massive protoclusters, thereby allowing for the study of the impact of local environment on its galaxy constituents (e.g., Kubo et al. 2013; Umehata et al. 2014; Kubo et al. 2016). While large numbers of protocluster candidates are being identified from wide-field deep surveys (e.g., Toshikawa et al. 2016, 2018), the lack of narrowband observations targeting these structures will remain a main challenge in utilizing these structures to elucidate the physics in the main epoch of cluster formation.

The primary challenge in investigating the environmental effect on protocluster constituents is the lack of spectroscopic redshifts, which prevents unambiguous confirmation of cluster membership and inhibits a robust mapping of the density profile of the cluster. Because our selection methods target a relatively broad range of redshifts, all galaxy samples are expected to contain and may even be dominated by interlopers not associated with the structure we wish to probe. These considerations testify to the clear need of spectroscopic information in making progress.

One possible way to discern any environment trend is to compare the galaxy statistic measured in a protocluster field with that obtained in a field without any strong density enhancements. Provided that the environmental effects are strong and a substantial number of galaxies in the sample belong to the protocluster, a qualitative trend may be identified through this comparison (e.g., Cooke et al. 2014). However, a comparative study is only meaningful if the two data sets are well matched in depth, dynamic range, and wavelength coverage, which determine the precision with which the photometric redshifts and stellar population parameters of the galaxies can be measured.

With these caveats in mind, we compare the properties of protocluster candidates with those of a control sample. The control sample is constructed from the COSMOS15 catalog (Laigle et al. 2016), where sources whose best-fit photo-z solution lies in the range z_phot = 3.4–4.2 are selected. After removing galaxies with multiple peaks in the photo-z PDFs, the sample consists of 19,318 sources. We run the CIGALE software using the same setup as before, assuming CSFs for both samples. While it is unrealistic to expect that all galaxies have CSFs, we are interested in the comparison of the two samples and not in exploring the full behavior of galaxies. A different SFH choice would generally shift measured quantities in the same direction for most galaxies and thus would not change our conclusions. Finally, we note that the photo-z precision for the COSMOS galaxies is expected to be much better (σ/(1+z) ∼ 0.02–0.03) than that for our sample (σ/(1+z) ∼ 0.06) thanks to the better imaging depth and finer wavelength sampling in the optical/near-IR wavelengths. While the larger uncertainty can introduce a larger scatter in the overall distribution of a derived quantity, it will not impact our ability to discern any mean relation between two different quantities.

In the left panel of Figure 11, we show the locations of our photo-z sources and of LAEs on the SFR–M_star plane together with those of the control sample. A prediction from a semi-analytic model (Dutton et al. 2010) is also shown. Both our photo-z candidates and LAEs occupy the same region as the field galaxies, suggesting that they obey the same star formation "main-sequence" scaling relation, consistent with existing studies (e.g., Cucciati et al. 2014; Koyama et al. 2014; Erfanianfar et al. 2016). From the same figure, it is evident that the COSMOS data sets can probe galaxies down to much lower masses than the present data set. The mismatch of the sensitivities of the two data sets renders it challenging to compare how the number counts in bins of SFR or stellar mass differ in these samples.

Zoom In Zoom Out Reset image size

To investigate possible environmental trends, we divide the photo-z sample into several environmental bins and color-code them accordingly, where darker shades represent higher densities. Given the uncertainties in the extent and center of the structure, we define the local environment using the LAE and photo-z surface densities. The results are shown in the top middle and right panels. The overall correlation—measured for each subsample in mass bins of ${\rm{\Delta }}\mathrm{log}{M}_{\mathrm{star}}=0.25$—is shown in solid lines. The SFR–M_star scaling laws measured from these subsamples are generally similar to that measured in the COSMOS sample.

We detect a hint of enhanced star formation activity in the highest photo-z overdensity subsample. Four galaxies deviate from the field average by 0.3–0.4 dex (a factor of 2–3). The overall scaling relation in this bin has a slightly higher normalization (i.e., ∼0.1 dex higher SFR in a given stellar mass bin), although the scatter is substantial. Interestingly, the same bin also lacks massive galaxies above $\mathrm{log}{M}_{\mathrm{star}}=10.8$. The high-mass high-SFR end is well populated by galaxies residing in all environments. All in all, the environmental effects on star-forming galaxies appear to be minimal.

The lack of detectable environmental effects on the galaxy properties is puzzling. Uncertain cluster membership surely plays a role in diluting any existing trend by misplacing a subset of galaxies into a wrong density bin. However, should there be an excess of high-mass or high-SFR galaxies in dense environments, our analyses would have captured it as the regions most likely to be dense are counted as such in one or the other scenario. Hence, our analysis suggests that the environmental effect on star formation is likely a subtle one.

Alternatively, most of the enhanced star formation is perhaps obscured from our view by dust. Koyama et al. (2013) reported that while sSFRs are higher for galaxies in cluster environments than for those in the field, the trend emerges only when the mid-infrared budget of the SFR is properly accounted for. They argued that their result may be explained if a higher fraction of nucleated dusty starbursts exist in cluster environments, where dust properties are significantly different from those in normal star-forming galaxies, such that applying the same dust correction as for the field galaxies would underestimate the true SFRs.

The lack of extreme star formers in our sample (in both the field and the protocluster) is also in part a selection effect. Extremely dusty starbursting galaxies would not be included in our sample, as they would not have a strong enough spectral break for us to identify them robustly, or perhaps are entirely invisible in the optical or infrared wavelengths. We therefore cannot test for the prevalence of dusty starburst systems in dense environments reported by several studies (Casey 2016; Hung et al. 2016). Testing these hypotheses will require deeper infrared and submillimeter coverage of the field.

The presence of powerful radio galaxies has been used as a signpost of highly overdense regions (Venemans et al. 2005, 2007; Kajisawa et al. 2006; Miley et al. 2006; Overzier et al. 2008; Hatch et al. 2011; Kuiper et al. 2012; Cooke et al. 2014; Rigby et al. 2014). Likewise, highly overdense structures appear to harbor powerful AGNs observed as X-ray or submillimeter luminous sources or giant Lyα nebulae (e.g., Steidel et al. 2000; Prescott et al. 2008; Lehmer et al. 2009; Casey et al. 2015; Casey 2016; Hung et al. 2016).

Motivated by these findings, we search existing radio and X-ray source catalogs to look for a sign of enhanced AGN activity. We cross-match the Chandra X-ray point-source catalog of the Boötes field (XBoötes; Kenter et al. 2005) with our photo-z, BBG, and LAE positions and find no match. The XBoötes survey sensitivity of the full band (0.5–8 keV) is 7.8 × 10⁻¹⁵ erg cm⁻² s⁻¹. Lehmer et al. (2009) studied X-ray-detected sources in and around the SSA structure at z = 3.09 and found that the X-ray fluxes for the confirmed members range across (0.3–5.0) × 10⁻¹⁵ erg cm⁻² s⁻¹. Thus, nondetection merely suggests that even the brightest X-ray sources in SSA 22 would lie below the XBoötes detection limit.

We also search for radio counterparts of our protocluster candidates (both photo-z and BBG candidates) in the radio source catalog based on deep Low-Frequency Array (LOFAR) 150 MHz observations (Tasse et al., in preparation). The rms noise of the data is 59 μJy/beam. Using a matching radius of 2'', no counterpart is found. We also compare our source positions against a photometric redshift catalog of the same LOFAR-detected sources constructed following the method presented in Duncan et al. (2018a, 2018b), which covers roughly two thirds of our survey field and only half of the photo-z overdensity region. This is because the bottom one third of our survey field lies outside of the NDWFS field (Jannuzi & Dey 1999). Once again, no credible counterpart is identified. In addition, we cross-match our candidates with the deep Westerbork Synthesis Radio Telescope 1.4 GHz catalog covering the Boötes field (de Vries et al. 2002) and find no counterpart. Therefore, we can rule out the presence of any high-redshift radio source with a flux density ≳0.2 mJy in the probed redshift range.

Apart from the limited survey sensitivities, the nondetection of powerful AGNs in the protocluster member candidates is perhaps not surprising. As discussed previously, the majority of our photo-z candidates, by design, resemble LBGs with a clean spectral break. This requirement effectively removes all galaxies that are either dusty starbursts or AGNs with power law–like SEDs similar to those identified by Hung et al. (2016) in the COSMOS field; robust identification of such galaxies will require improved sensitivities and wavelength baselines.

We assess how likely it is to find both structures or one very large structure in our survey volume. We utilize a catalog containing 2731 simulated clusters identified from the Millennium I+II runs as described in Chiang et al. (2013). The minimum cluster mass is ${10}^{14}\,{h}^{-1}\,{M}_{\odot }$ at z = 0. The comoving volume of the simulation is (500/h)³ Mpc³ or 0.364 Gpc³. Our survey volume is estimated conservatively to be 2.13 × 10⁶ Mpc³ assuming a flat redshift distribution at z = 3.4–4.2 over a 28' × 28' area, which is 0.6% of the Millennium volume.

We randomly pick a region matching our survey volume and record the number of clusters therein and the position and the mass of each cluster. The procedure is repeated 500,000 times. The median (mean) number of clusters is found to be 16.0 (16.5) with a standard deviation of 5.2; i.e., our survey volume is large enough to contain multiple clusters.

If the LAE and photo-z overdensities are part of a single very large structure, its combined mass would be enormous. In Dey et al. (2016a), based on the level and extent of the LAE overdensity alone, we estimated that the enclosed mass is $\gtrsim {10}^{15}\,{M}_{\odot }$. As discussed in Section 4.3, the photo-z overdensity should contain a comparable mass. Given that the two overdensities only partially overlap (and the regions of the peak overdensities do not overlap), a conservative limit on the combined mass is in the range of $(1.5\mbox{--}2.0)\times {10}^{15}\,{M}_{\odot }$. We find the probability of these scenarios to be 4.4% and 0.8%, respectively. In the entire Millennium volume, eight structures and one structure exist with masses above $1.5\times {10}^{15}\,{M}_{\odot }$ and $2\times {10}^{15}\,{M}_{\odot }$, respectively, corresponding to a comoving number density of (2.2 ± 0.8) × 10⁻⁸ Mpc⁻³ and (2.7 ± 2.7) × 10⁻⁹ Mpc⁻³, respectively. The most massive structure in the Millennium simulations has a total mass of $2.4\,\times {10}^{15}\,{M}_{\odot }$. The observational counterpart of such an ultramassive cluster may be the El Gordo system, which is a merging pair of Coma analogs at z = 0.87 (Marriage et al. 2011).

To test the possibility that the two overdensities are unrelated structures, we search for cases in which there are two Coma-like clusters (i.e., each with mass $\geqslant {10}^{15}\,{M}_{\odot }$). This occurs only 3.6% of the time. Finally, we assess how unlikely it is that the photo-z overdensity lies at z = 3.72 (see discussion in Section 4), which would put the distance between the two at 47 Mpc or 10 proper Mpc. Only four distinct pairs of Coma analogs exist in the Millennium sample that are within 10 Mpc (physical) of each other. Two of those pairs have a physical separation of 5.2 and 5.3 Mpc from each other, and the other two have 9.3 Mpc and 9.8 Mpc. The separation for the latter is comparable to that between PC 217.96+32.3 and the putative overdensity at z = 3.72 (Lee et al. 2018). The likelihood of such a configuration falling into our survey is 0.8%. These considerations show that both scenarios are extremely unlikely to occur by chance, but also that they are not impossible.

The overall density of the regions in and around PC 217.96+32.3 is remarkably high. Apart from the two overdensities we discuss here, two other LAE overdensities lie within ∼10 Mpc (physical) north of PC 217.96+32.3 (Lee et al. 2014), one of which is spectroscopically confirmed and has an estimated descendant mass of $\approx 6\times {10}^{14}\,{M}_{\odot }$ (Dey et al. 2016a). Given the distances between these systems, it is unlikely they will coalesce into a single structure within the Hubble time, but rather, they will evolve separately and form structures similar to local superclusters (e.g., Einasto et al. 1997, 2014).