Machine Learning Applied to Star–Galaxy–QSO Classification and Stellar Effective Temperature Regression

SDSS is an international project that has made the most detailed three-dimensional maps of our universe. The fourth stage of the project (SDSS-IV) started in 2014 July, with plans to continue until mid-2020 (Blanton et al. 2017). The automated spectral classification of the SDSS-IV is determined with the chi-square (χ²) minimization method, in which the templates are constructed by performing a rest-frame principal-component analysis (Bolton et al. 2012; Blanton et al. 2017). The first data release in the SDSS-IV, DR13, includes over 4.4 million sources, in which galaxies comprise 59%, QSOs 23%, and stars 18% (Albareti et al. 2017).

Another ongoing spectroscopic survey is undertaken by LAMOST (Cui et al. 2012; Zhao et al. 2012), which mainly aims at understanding the structure of the Milky Way (Deng et al. 2012). In 2016.06.02, LAMOST finished its forth year of survey (the forth data release; DR4) and has obtained spectra of more than 7.6 million sources included 91.6% stars, 1.1% galaxies, and 0.2% QSOs. The LAMOST one-dimensional (1D) pipeline recognizes spectral classes by applying a cross-correlation with templates (Luo et al. 2015). An additional independent pipeline and visual inspection are carried out in order to double check the galaxies' and QSOs' identification. Here, we adopt SDSS DR13 plus LAMOST DR4, since they are matched on the time of the data releasing.

Figure 1 shows the comparison between the two spectroscopic surveys. The objects of LAMOST are dominated by stars, while over half of the objects in SDSS are galaxies. The combination of the two surveys can provide a more balanced and larger training sample for the classification. In order to add more QSO samples, we adopt the 13th edition of quasars and active nuclei catalogs (Veron13, Véron-Cetty & Véron 2010), which includes 23108 samples. The priority of the catalogs is Veron13, SDSS, and LAMOST, if some objects are included in more than one catalog.

Zoom In Zoom Out Reset image size

The combination of optical and infrared (IR) data on huge numbers has been proved to be valid in the star/galaxy classification (Baldry et al. 2010; Henrion et al. 2011) and stellar parameter determination (Allende Prieto 2016). SDSS has imaged over 31000 square degrees in five broad bands (ugriz). The DR13 includes photometry for over one billion objects. WISE (Wright et al. 2010) performed an all-sky survey with images in 3.4, 4.6, 12, and 22 μm and yielded more than half a billion objects.

In order to obtain the training sample, we extract objects with the available model magnitudes in g, r, i bands for the SDSS and LAMOST spectroscopic surveys, and cross identify them with the WISE All-Sky Data Release catalog with the help of the Catalog Archive Server Jobs (CasJobs).⁴ Similar to Krakowski et al. (2016), we use w1(2)mpro magnitudes. The J, H, and K magnitudes are also extracted in order to cover the near-IR bands. Our selection required the object with zWarning = 0 for the SDSS objects, and S/Ns higher than 2 in the W1 and W2 bands. We adopt w?mag13 as the indicator for the extended objects (Bilicki et al. 2014; Krakowski et al. 2016; Kurcz et al. 2016; Solarz et al. 2017), which is defined as

$$\begin{eqnarray}&&{\rm{w}}?\mathrm{mag}13={\rm{w}}?\mathrm{mag}\_1-{\rm{w}}?\mathrm{mag}\_3,\end{eqnarray} \tag{ 1 }$$

where w?mag_1 and w?mag_3 are magnitudes measured with circular apertures of radii of 55 and 11''. The question mark is the channel number in the catalog.

We use the CasJobs to cross identify the photometric data with the spectral catalogs of LAMOST, SDSS, and Veron13. The result has 2,973,855 objects, including 2,123,831 stars, 806,139 galaxies, and 43885 QSOs. We present the color–color diagram in Figure 2 that is often used for the star–galaxy separation (e.g., Jarrett et al. 2011; Goto et al. 2012; Ferraro et al. 2015). The contours of the three classes overlap in the color–color diagram. Neither the cut W1 − W2 = 0.8 (Stern et al. 2012; Yan et al. 2013), nor W1 − J = −1.7 (Goto et al. 2012) could provide a clear cut to classify the stars, galaxies, and QSOs.

Zoom In Zoom Out Reset image size

We build a nine-dimensional color space, g − r, r − i, i − J, J − H, H − K, K − W1, W1 − W2, w1mag13, and w2mag13. Each object is weighted with the quadratic sum of their photometric uncertainty. The holdout validation is applied to test the total accuracies of different machine-learning algorithms, in which a random partition of 20% is held out for the prediction and the rest is used to train the classifier.

Table 1 lists the accuracies and time costs of different algorithms for the 20% held out samples (80% of the samples are used for training and 20% of the samples are held out for testing). Since the validation is applied, the time costs are approximate. The RF algorithm (Breiman 2001) shows the best performance on the time cost (57 minutes⁵ ) and the total accuracy (99.2%). Other methods, for example, the k-nearest neighbor and the SVM, either take more time to build the classifiers or show lower total accuracies.

The working theory of the RF is that it builds an ensemble of unpruned decision trees and merges them together to obtain a more accurate and stable prediction. The algorithm consists of many decision trees, and it outputs the class that is the mode of the class output by individual trees (Breiman 2001; Gao et al. 2009). The RF is often used when we have very large training data sets and a very large number of input variables. One big advantage of RF is fast learning from a very large number of data. Gao et al. (2009) listed many other advantages of RF.

After selecting the best algorithm, we apply the holdout validation and test the RF classifier 10 times. The average accuracy is 99% and the result is shown in Table 2 and Figure 4. In the classifier, the contributions of the nine colors are different, which could be described by the predictor importance estimates (Figure 3). We find that the IR colors play an important role in our classifier, which is similar to the result of Krakowski et al. (2016).

Zoom In Zoom Out Reset image size

We adopt the measures defined by Soumagnac et al. (2015) to show the performance of the classifier. These measures have been used in other machine-learning studies (Kovács & Szapudi 2015; Krakowski et al. 2016): completeness (c), and purity (p) for star, galaxy and QSO samples. We use the following equations (here for galaxies):

$$\begin{eqnarray}&&{c}_{g}=\displaystyle \frac{\mathrm{TG}}{\mathrm{TG}+\mathrm{FGS}+\mathrm{FGQ}},\end{eqnarray} \tag{ 2 }$$

$$\begin{eqnarray}&&{p}_{g}=\displaystyle \frac{\mathrm{TG}}{\mathrm{TG}+\mathrm{FSG}+\mathrm{FQG}}.\end{eqnarray} \tag{ 3 }$$

The FGS and FGQ are the numbers of galaxies misclassified as stars and QSOs, and FSG and FQG are the numbers of stars and QSOs misclassified as galaxies, respectively. The QSO sample shows the lowest completeness and purity, probably due to its smallest sample size.

In order to test this effect, we normalize the sample sizes of the three classes and apply the 20% holdout validation again (about 8800 objects in each type for the testing). The average result is shown in Table 2 and the right panel of Figure 4. The three classes have similar percentages of the completeness and purity. This result implies that we could not judge the performance of the classifier only by these measures.

Zoom In Zoom Out Reset image size

We also apply the magnitude binnings suggested by Krakowski et al. (2016) to test the completeness, since different magnitudes stand for different stellar and galactic types and distances. The binnings are 12 < W1 < 13, 13 < W1 < 14, 14 < W1 < 15, and 15 < W1 < 16. The completenesses for stars, galaxies, and QSOs are similar to those calculated without binning. On the even samples, the low performance of the galaxy sample is probably due to the relatively high contamination from the QSO sample rather than the lost information of the galaxy sample.

This section describes various tests using the classifier made from the LAMOST, SDSS, and Veron13. These tests allow us to quantify the performance of the classification.

3.2.1. 6dF Galaxy Survey

The 6dF Galaxy Survey has (6dFGS) mapped the nearby universe over nearly half the sky (Jones et al. 2004, 2009). The final redshift release of the 6dFGS contains 124,647 spectrally identified galaxies. We match the galaxies with the SDSS and WISE archive data, which yields 12300 galaxies. We then remove the galaxies, which are used to build the classifier, and there are 8382 galaxies left. We feed the classifier with nine colors of these galaxies and obtain the predicted types. The classifier can output three scores for each entry, corresponding to the possibilities for star, the galaxy, and the QSO. The type with the largest score is adopted as the predicted type. The classifier also output the standard deviation (σ) for each score.

About 99.5% of the galaxies are classified correctly, and 40 galaxies are incorrectly classified as stars. Here all the predicted QSOs are treated as the galaxies, since there is no QSO subtype in the 6dFGS. The classification result is shown in Figure 5. The scores of the correctly classified galaxies are larger than those of the incorrectly classified. About 73% of the correctly classified galaxies have σ < 0.2, while only 55% for the wrongly classified galaxies have σ < 0.2. This indicates that the classifier is very uncertain about the types of incorrectly classified galaxies.

Zoom In Zoom Out Reset image size

3.2.2. RAdial Velocity Extension (RAVE)

RAVE is designed to provide stellar parameters to complement missions that focus on obtaining radial velocities to study the motions of stars in the Milky Way's thin and thick disk and stellar halo (Steinmetz et al. 2006). Its fifth data release (DR5) contains 457,588 stars in the south sky (Kunder et al. 2017). There are 935 stars also observed by SDSS and WISE. We remove the stars used to build the classifier, and end up with 737 remaining stars. We feed the classifier with nine colors and it yields 736 stars and one QSO with an accuracy of 99.9%. The incorrectly classified star, SDSS J154142.28-194513.1, is located in the bright halo of the kap Lib. Its colors are probably polluted by the bright star.

We also take advantage the g, r, and i magnitudes from APASS (Munari et al. 2014) that have been matched with RAVE stars (Kunder et al. 2017). Not all of the stars are detected in both WISE and APASS, and there are 435,012 stars with seven valid colors. The prediction contains 434,735 stars, 264 galaxies, and 13 QSOs with an accuracy of 99.9%. The classification result is shown in Figure 6. The incorrectly classified stars have smaller scores and larger σ, implying a high uncertainty of the types.

Zoom In Zoom Out Reset image size

3.2.3. UV-bright Quasar Survey (UVQS)

The data release one of all-sky UVQS contains 1055 QSOs selected from GALEX and WISE photometry and identified with optical spectra (Monroe et al. 2016). We cross identified the QSOs with SDSS and WISE, which yields 262 QSOs. We remove the QSOs used to build the classifier, which leaves 237 QSOs. The classifier yields 224 QSOs, 12 galaxies, and one star with an accuracy of 94.5%. Again, the incorrectly classified QSOs show smaller scores and larger σ (Figure 7). The accuracies of the blind tests are summarized in Table 3.

Zoom In Zoom Out Reset image size

Table 3.  The Accuracies of the Blind Test

Survey	Accuracy
6dFGS	99.5%
RAVE	99.9%
UVQS	94.5%

Download table as:  ASCIITypeset image

The LAMOSTs 1D pipeline only provides rough classification results, and the accuracy of the subclasses is still not robust (Jiang et al. 2013). Therefore, we instead adopt the effective temperatures (T_eff) from the A-, F-, G-, and K-type star catalog, which was produced by the LAMOST stellar parameter pipeline (LASP; Wu et al. 2014). We also extract the T_eff computed with the SEGUE Stellar Parameter Pipeline in the SDSS (SSPP; Allende Prieto et al. 2008; Lee et al. 2008a, 2008b). Both samples are dominated by G stars and have similar distributions of T_eff (Figure 8).

Zoom In Zoom Out Reset image size

In the classification (Section 3), RF exhibits the advantage in accuracy and training time cost. Here we also adopt the algorithm of RF to build the regression of stellar effective temperature.

These temperatures and seven colors, g − r, r − i, i − J, J − H, H − K, $K-W1$ and W1 − W2, of 1,327,071 stars are used to train the RF for regression. We apply the 10-fold cross validation in order to test the performance of the regression. The cross validation partitions the sample into ten randomly chosen folds of roughly equal size. One fold is used to validate the regression that is trained using the remaining folds. This process is repeated ten times such that each fold is used exactly once for validation.

We present the results of the cross validation in Figure 9. The one-to-one correlation is shown in the left panel. In order to estimate the uncertainty of the prediction, we bin the predicted T_eff with a step size of 100 K and fit the distribution of the corresponding test T_eff with a Gaussian function. We calculate the root-sum square of the standard deviation and the offset of the fit, which is adopted as the uncertainty of the prediction (the blue error bars in Figure 9). The Gaussian fit to the total residuals is shown in the right panel of Figure 9, and the fitted offset (μ) and σ are listed in Table 4. The red bars in Figure 3 are the importance estimates for the regression. The optical and 2MASS colors show much more importance than the WISE colors, which are different from those of the classification. This may be due to the fact that the majority of our sample is G- and K-type stars.

Zoom In Zoom Out Reset image size

In this subsection, we use the T_eff extracted from the spectrum-based methods to test the actual performance of the regression.

4.2.1. RAVE

The RAVE pipeline processes the RAVE spectra and derives estimates of T_eff, log g, and [Fe/H] (Kunder et al. 2017). The pipeline is based on the combination of the MATrix Inversion for Spectral SynthEsis (Recio-Blanco et al. 2006) algorithm and the DEcision tree alGorithm for AStrophysics (Bijaoui et al. 2012). This pipeline is valid for stars with temperatures between 4000 and 8000 K. The estimated errors in T_eff are approximately 250 K, and ∼100 K for spectra with S/N⁶ ∼ 50 (Kunder et al. 2017).

We adopt the photometry from APASS and WISE in the RAVE database to construct the input colors. The sample is restricted to have S/N ≳ 50 and the quality flag of ${Algo}\_{Conv}\ne 3$ or 4.⁷ There are 165,011 stars left. We present the prediction results in Figure 10 and list the parameters of the Gaussian fit to the total residuals in Table 4.

Zoom In Zoom Out Reset image size

4.2.2. APOGEE

The Apache Point Observatory Galactic Evolution Experiment (APOGEE), one of the programs in SDSS-III, has collected high-resolution (R ∼ 22500) high signal-to-noise (>100) near-infrared (1.51–1.71 μm) spectra of 146,000 stars across the Milky Way (Majewski et al. 2017). These stars are dominated by red giants selected from the 2MASS. Their stellar parameters and chemical abundances are estimated by the APOGEE Stellar Parameters and Chemical Abundances Pipeline (García Pérez et al. 2016). The typical error in T_eff is ∼100 K (Mészáros et al. 2013).

We extract the photometric data of SDSS and WISE with the help of the Casjob. We feed the RF regression with seven colors of 13685 stars. The prediction is shown in Figure 11, and the parameters of fit to the total residuals are listed in Table 4.

Zoom In Zoom Out Reset image size

We find that the offsets of the validation and the predictions are less than 100 K, and the standard deviations are less than 200 K (Figures 10 and 11). Lee et al. (2015) has applied the SSPP to LAMOST stars and compared the results to those from RAVE and APOGEE catalogs. The offsets of T_eff between different pipelines range from 36 to 73 K, and the standard deviations range from 79 to 172 K. This indicates that our RF regression can determine stellar temperatures with fair accuracy.

Ball et al. (2006) applied the supervised decision tree algorithm to classify the stars and galaxies in SDSS-DR3. They used the colors u − g, g − r, r − i and i − z of 477,068 objects with spectroscopic attributes to train the machine-learning classifier, and they performed cross validation to test the performance. The accuracy and completeness were over 90%. Except for the optical colors, the IR colors are included in our multi-color data set, since they have shown considerable importance in machine-learning methods (Henrion et al. 2011). Our larger training sample and the IR aided color set result in a better performance of our classifier, over 99% for the stars and galaxies classification. We also test the performance of some decision tree algorithms, for which the accuracies are ∼98%. Compared to the decision tree, random forest avoids overfitting to the training set and limits the error from the bias (Hastie et al. 2008).

Krakowski et al. (2016) used the SVM learning algorithm to classify WISE × SuperCOSMOS objects based on the SDSS spectroscopic sources. The training sample included over one million objects, 95% of which were galaxies, 2% were stars, and 3% were QSOs. They used six parameters, W1, W3, W1 − W2, R − W1, B − R, and w1mag13. The 10-fold cross validation was performed to test the classifier, and the total accuracy was 97.3%. Instead of magnitudes, we adopt colors that are independent of distance. Our training sample shows better compositional balance, and its size is three times larger than theirs. We also try some SVM algorithms, and the accuracies are from 70% to 99%. The time cost to build the SVM classifier is much longer than that for the RF classifier.⁸ For a classification problem, RF gives the probability of belonging to classes (Breiman 2001), while SVM relies on the concept of "distance" between points and needs more time to calculate. The RF algorithm also shows better performance than SVM in other fields, such as Liu et al. (2013).

Liu et al. (2015) employed an SVM-based classification algorithm to predict MK classes with 27 line indices measured from a small sample, 3134 LAMOST stellar spectra. The holdout validation of 50% was performed to test the accuracy of the classifier. The completeness of A and G stars reached 90%, while that of other stars was below 80%. Since the spectral features of different types can be very similar, clear cuts of these features probably lead to misclassification. Therefore, we adopt the regression of T_eff rather than the MK classification in order to avoid such an effect. Liu et al. (2015)'s research also implies that a large sample could cover a larger area of the parameter space and could further yield more reliable predictions.

Sarro et al. (2018) constructed regression models to predict T_eff of M stars with eight machine-learning algorithms. The training sample is built with the features extracted from the BT-Settl of synthetic spectra. Then, the models were applied to two sets of real spectra from the NASA Infrared Telescope Facility and Dwarf Archives collections. Sarro et al. (2018) used the root mean/median square errors (RMSE/RMDSE) to describe the prediction errors. The RMSEs were from 160 to 390 K, and the RMDSEs from 90 to 220 K, various with the different algorithms and S/Ns. Our prediction for A, F, G, K stars gives similar results: RMSE/RMDSE (RAVE) = 246/140 K, and RMSE/RMDSE (APOGEE) = 247/130 K, implying that our regression built with photometric data could achieve similar accuracy to the spectrum-based model.

Another way to determine the T_eff is the fit of stellar SEDs with synthetic templates. The theoretical study has concluded that the broad-band photometry from the UV to the mid-IR allows atmospheric parameters and interstellar extinction to be determined with good accuracy (Allende Prieto 2016). The study used the SEDs extracted from the ATLAS9 model atmospheres (Mészáros et al. 2012). They added interstellar extinction to these SEDs in order to construct the theoretical templates. The test SEDs were also extracted from the ATLAS9 model, but added some random noise. Then the test SEDs were fitted with the templates using the χ²-optimization method. The standard deviations of the total residuals were from 130 to 380 K depending on the different bands used for the fittings. We follow this procedure to fit 10⁵ simulated SEDs, extracted from the BT-Cond theoretical model (Baraffe et al. 2003; Barber et al. 2006; Allard & Freytag 2010). Since the simulated SEDs have random stellar parameters, about 70000 SEDs are located inside of the reasonable ranges. The result is shown in the upper panels in Figure 12. We only plot ∼64000 samples with χ² ≲ 5.88, which is one standard deviation for a χ² distribution with the five degrees of freedom (T_eff, log g, [Fe/H], $E(B-V)$ and the scaling factor). The residuals are fitted with a Gaussian-like function:

$$\begin{eqnarray}&&f={{Ae}}^{\tfrac{-{(x-\mu )}^{\delta }}{\delta {\sigma }^{\delta }}}.\end{eqnarray} \tag{ 4 }$$

The standard deviation of T_eff is 207 ± 15 K for 12 bands fit, F-NUV, ugriz, ${JHK},W1W2$. The standard deviations of other parameters are also similar to the result in Allende Prieto (2016), indicating that the multi-band SED fit can well constrain the atmospheric parameters and interstellar extinction theoretically.

Zoom In Zoom Out Reset image size

However, the result is worse than expected (the lower panels in Figure 12) when we apply the SED templates to fit the SEDs in the real observation, the stars in the LAMOST Spectroscopic Survey of the Galactic Anticentre (LSS-GAC; Liu et al. 2014; Yuan et al. 2015). The standard deviation of T_eff is 454 ± 5 K and the offset is −365 ± 5 K, larger than those of the machine-learning regression by a factor of three. We also try this technology in other ways, e.g., fitting the stars in RAVE, or using 10 bands fit.⁹ The standard deviations of T_eff are about 400 K, extremely worse than both the theoretical simulation and the machine-learning regression. This implies that the atmospheric parameters of the stars in the real observation can't be well estimated by the SED fit using the χ² minimization. Based on photometric data, machine learning shows better performance on the T_eff estimate.

The ESA space mission Gaia is performing an all-sky survey at optical wavelengths, and its primary objective is to survey more than one billion stars (Gaia Collaboration et al. 2016). Its second data release (Gaia DR2; Gaia Collaboration et al. 2018) includes ∼1.3 billion objects with valid parallaxes. These parallaxes are obtained with a complex iterative procedure, involving various assumptions (Lindegren et al. 2012). Such a procedure may produce parallaxes for galaxies and QSOs, which should present no significant parallaxes (Liao et al. 2018).

We have applied the classifier to 85,613,922 objects in the Gaia DR2 based on the multi-wavelength data from Pan-STARRS and WISE (Bai et al. 2018a). The result shows that the sample is dominated by stars (∼98%), and galaxies and QSOs make up 2%. For the objects with negative parallaxes, about 2.5% are galaxies and QSOs. About 99.9% of the sample comprises stars if the relative parallax uncertainties are smaller than 0.2, implying that using the threshold of 0 < σ_π/π < 0.2 could yield a very clean stellar sample (Bai et al. 2018a).