TGCat^*: THE CHANDRA TRANSMISSION GRATING DATA CATALOG AND ARCHIVE

The event filtering, event transformation, spectral extraction, and response generation are done with standard CIAO programs (Fruscione et al. 2006) in conjunction with the Chandra Calibration Database⁸ (CALDB). The methods are equivalent to those outlined in the CXC data analysis "Science Threads."⁹ All of the processing steps can also be reproduced by users with distributed CXC software. A top-level pipeline sequence is shown in Table 2 and the output products are listed in Table 3. Most of the files are standard FITS format products whose definitions can be found on CIAO web pages (e.g., event, PHA, response files, aspect histograms, and light curves). Others added by the TGCat reprocessing scripts to augment the standard CIAO tool products are text summary files, summary plots, and a spectral properties file.

The processing is managed by a suite of custom scripts written in ISIS, the "Interactive System for Interpretation of Spectra"¹⁰ (Houck & Denicola 2000), which determine instrumental and observational parameters from the data, configure the CIAO commands with defaults appropriate to grating spectral extraction or with observationally dependent parameters, and spawn the CIAO processes. ISIS uses the S-Lang scripting language¹¹ as its interpreter. S-Lang is an excellent choice for scientific analysis, being small, efficient, modular, and extensible (see Noble & Nowak 2008). For TGCat work with Chandra data, the cfitsio¹² module is particularly useful in providing a function-oriented interface to FITS-format (Hanisch et al. 2001) files. The primary advantage for using ISIS as the platform instead of the S-Lang shell is that ISIS provides much infrastructure for working with spectra and responses; it also includes a plotting package through the pgplot¹³ module. We use the spectral support of ISIS to compute fluxes, plot spectra, and generate flux properties tables. In addition, the fitting support allows us to run "findzo"(Nichols et al. 2008), a script which determines the zeroth-order centroid (the origin of the wavelength scale) to high accuracy even if the zeroth-order image is distorted by detector or telemetry saturation, or even excluded from telemetry (about 20% of the observations require the use of findzo). ISIS was designed especially for Chandra grating spectroscopic analysis and modeling and is actively used and supported for users by CXC grating scientists. Further details regarding the design and capabilities of ISIS may be found in Houck & Denicola (2000) and Noble & Nowak (2008).

At the top level, there is a single ISIS function which determines the observational configuration and then calls the appropriate script to sequence the CIAO tasks. Customization can be done for non-standard extractions by setting global state variables (e.g., to specify coordinates other than given in the event file header, or to use an alternate source detection method, or to define alternate sky or extraction regions). Lower level functions can be easily scripted to provide nearly arbitrary CIAO tool invocations. While the TGCat processing software is not intended to be fully general or to replace the UNIX command-line CIAO interface, it can be used for many common processing scenarios. The processing software used to produce the TGCat archive is documented and made available to users¹⁴ (see Section 4 for details).

The flux properties table is a FITS binary table of count rates and fluxes integrated over bandpasses. Details of the file contents are given by a partial listing for one observation in Table 4. There is no continuum fitting or subtraction or line-profile fitting. Statistical uncertainties are provided for each band. For LETGS, counts are background subtracted, but flux values have not been computed since they are model dependent due to overlapping higher-order contributions in the counts. Regions are assigned a label indicating the dominant element and ion appropriate for a typical coronal thermal plasma in the region. Near each such band is a "continuum" band, labeled with a c. To obtain an approximate line-region rate or flux, the continuum values can be interpolated and scaled to the line bandpass using the tabulated wlo and whi values giving the bandpass, and then subtracted from the line rate. The band wavelength midpoint, wmid is tabulated. (For continuum sources, the line labels are irrelevant.) Broadband rates have been given for the HEG, MEG, LETG/HRC, and LETG/ACIS spectral ranges. We provide zeroth-order rates, but it must be realized that for many sources observed with ACIS, the zeroth order is saturated by photon pileup (Davis 2001). Rates are supplied for the Chandra Source Catalog "science energy" bands (Evans et al. 2010) and are labeled with a csc_ prefix and a band mnemonic character (b, u, s, m, and h).

If a bandpass is truncated by the sensitivity limits, or if the effective area is zero-valued somewhere within the band, then a value in the flag column will be non-zero (e.g., letgs\_band for HETGS observations). These rates are intended to be used for qualitative characterization of spectra. Detailed modeling is required for determination of rigorous values, particularly for weak or heavily absorbed sources in the softer bands (>20 Å) where noise and background can be amplified in the flux estimate.

Archive and process management are performed by an automated system that collects and processes observations after their proprietary period has expired, organizes information for manual verification and validation, and then responds to the manual review by either releasing to the public Web site, reprocessing with adjusted extraction parameters, or holding the output indefinitely. The major components include a MySQL database for storage of metadata, a file-based data archive, and the processing software described in Section 2.1. The data flow is sketched in Figure 1.

Zoom In Zoom Out Reset image size

The information relevant to processing includes the extractions, source, and spectral properties tables, data product files (see Table 3), and queue tables. The extractions table has one entry per processed extraction. Any ObsID can have many sources and any source might be found in multiple ObsIDs. Any source in any ObsID could have multiple extractions (e.g., for time slices, or different cross-dispersion widths). The extractions table stores one entry for each combination thereof. In order to consolidate all extractions of a single source, there is a source table to associate sources by indexes on SIMBAD¹⁵ identifier, a TGCat identifier, and source celestial coordinates.

An automatic process runs at regular intervals (typically daily), downloads a list of gratings observations from the public Chandra archive, and compares it with the list of ObsIDs that have been submitted to TGCat. Any ObsID not in TGCat whose proprietary period has expired will be added to the queue for TGCat processing. Daemons (written in python) run on any number of the local network hosts and continuously request entries from the queue in a first-in, first-out manner. They then configure the queue entry (which can contain a number of custom processing parameters), setup work spaces and logging, fork off the processing to the ISIS pipeline scripts (described in Section 2.1), and then ingest products into the database and archive.

During the ingest step, several automatic checks are done to evaluate whether the processed data are complete. If so, metadata are added to the main extractions table which returns a unique identifier that is then used to tag the products and also link the source and extraction tables. For incomplete data, the queue entry is assigned an error and notification sent to TGCat administrators. TGCat operators have the choice to investigate the existing processing workspace or simply re-evaluate parameters and re-queue the extraction as new.

Verification and validation (V&V) is done by members of the TGCat science team (CXC scientists at MIT and SAO; HETG project scientists at MIT) in order to confirm the zeroth-order placement, proper sky mask and spectral binning regions, and to assess the presence of any confusing sources. Review is done through an internal administrative Web site similar to the public interface for browsing extractions, but with the addition of forms for changing extraction parameters and re-queueing processing (if necessary), and for marking the extraction with a "good," "bad," or "warning" label. The reviewer can edit comment fields which are available for internal adminstrative purposes only, and the ones which provide a V&V report for the end user (e.g., the "warning" category flags cases which cannot be mitigated by processing, but of which the user should be aware, such as double sources that cannot be separately extracted). The "bad" extractions are never shown on the public site; the administrators have the option of rejecting any extraction for any reason. Each extraction is tagged with the date of processing, the version of TGCat used for processing, and a group ID which is unique per group of extractions intended to be of the same object for a single ObsID, which allows us to keep track of accepted extractions and to avoid duplicate extractions. The V&V process uses the same summary graphical products that are presented to users in the catalog.

The processing parameters are saved for each extraction. This allows us to easily reprocess all or any subset of the archive in the event of software or calibration updates. Several such reprocessings have already occurred, motivated by a CALDB updates or CIAO software improvements. Reprocessing the entire archive takes about three days using one dedicated host and several others on the local network, as available.

To go beyond simply assessing the data quality, we have provided an interactive and very flexible custom plotting interface. It can plot a single order, combined orders for a single observation, or it can combine spectra for multiple observations. Spectra can be binned by signal to noise and number of channels (or both). Scaling can be log or linear with user-defined ranges. There are diverse choices for spectral coordinates (frequency-like or wavelength-like) and flux units.

For careful and rigorous spectral analysis, we recommend download of the individual spectral and response data products (see Section 3.2). But for some applications, the interactive plot result itself may be sufficient, such as for inclusion in multi-band plots to show the X-ray spectrum beside radio, infrared, optical, or radio results. For such use, the displayed plot can be downloaded as a plain-text table with embedded comments which document the contents and units of the columns.

Behind the scenes, plotting is implemented on the server by taking the request for plotting parameters along with a unique file name, creating a small ISIS script of the commands for loading and plotting data, and piping this script into an ISIS process. We prevent malicious use by checking all parameter inputs for appropriate values, running the ISIS process as an unprivileged user, and checking the temporary file name for validity. Since the file name is known at the time of the request, the page simply needs to be reloaded to show the new plot. This web-generated ISIS plotting script is also available for inspection or download. It can be used in conjunction with any downloaded data products and TGCat software as a template for user re-use or customization. Figure 4 shows an example plot page.

Zoom In Zoom Out Reset image size

There are several options for data download. Generally, users will want to download more than one file at a time for detailed analysis, such as spectra and responses for a single observation, products for multiple observations of the same target, or a selection of several similar objects. To this end, TGCat runs a packager process that parses a queue table. Requested packages are added to the package queue table, validated and read by the packager which then fetches data to a temporary space, placing them in a directory hierarchy tagged with ObsID and TGCat ID. The entire hierarchy is then tarred, compressed, and provided to the user along with file checksums via HTTP. Users have the opportunity to select the types of files to package, such as only spectra and responses, or to also include event lists. It is also possible to view and download individual files from a file-table list.

TGCat is a registered Virtual Observatory (VO) service providing both the "Simple Cone Search" and "Simple Image Access" protocols. These are both implemented as XML output plug-ins taking input from a PHP script that provides appropriate parameters, then creates a query object after parsing the input data, and finally runs the query in exactly the same way as for the web and ASCII interfaces. Error handling is done by the calling script. Query types are tracked so we can obtain statistics on requests coming from services such as the HEASARC Datascope¹⁶ VO search facility. The VO interface further allows searching and data retrieval through any VO-enabled program.

Walborn et al. (2009) used TGCat to compare X-ray spectra of O-stars with respect to spectral type and luminosity class. For each star in their comparison, they added all available spectra of each object to improve the signal. Using spectra available in TGCat ensured they were processed identically with the most appropriate calibration for each separate observation.

This technique has recently been used by J. S. Nichols et al. (2011, in preparation) to add the seven spectra of HD 93129A, an O2 I star. The addition of these seven separate exposures allowed modeling of the He-like and H-like emission lines in spite of the low signal to noise of the individual spectra. These authors found that the emission lines can be modeled as skewed, apparently shifted lines consistent with a dense, absorbed wind, much like that seen in the O4 I star, ζ Pup. W. L. Waldron et al. (2011, in preparation) have used spectral data for the B-type supergiant, κ Ori for a detailed analysis of the wind. This method of modeling combined spectra is a standard feature of ISIS.

TGCat is particularly useful for variability analysis of emission line regions. Assuming sufficient signal to noise, the researcher can filter an observation into approximately equal exposure times. Each time interval can then be processed with TGCat scripts to apply appropriate calibrations and to extract the spectra. Variability tools can then be used on the collection to look for spectral variability.

The fields of view of the entire set of Chandra HETG observations contain zeroth-order and dispersed spectra of a large number of non-targeted stars. While TGCat does not yet contain systematic extractions of these spectra, there are plans to provide them. These "bonus" spectra represent a large untapped resource of the Chandra project. One example of this type of project is the collection of HETG spectra of θ² Ori A from multiple HETG pointings on the Orion Nebular Cluster. When all of the θ² Ori A spectra that fell in any of these HETG field of view were extracted, the cumulative exposure was 520 ks, resulting in a high very high quality spectrum (Mitschang et al. 2011).

Early in the development and testing of TGCat, one of us (H.L.M.) did a selection on LETG/HRC-S observations and sorted by descending exposure time. At the top of the list was CMa (Adara, ObsID 6441). Browsing the images, we noticed that there was diffuse emission surrounding the spectrum. This was unexpected for a B-star—a point source. Further investigation of the TGCat archive found no other sources with extended emission near 30 Å, nor could we identify any instrumental effect particular to this observation. This led to a successful ACIS-S imaging spectroscopy proposal (ObsID 9926) to search for the spectral excess near 0.4 keV and any spatially resolved emission (since the ACIS PSF is simpler and better modeled than that of LETG zeroth order). The result was negative—the spatial/spectral feature was not physical; while not understood, it certainly was not due to terribly interesting and unprecedented astrophysics of Adara. We put the study aside as an unreproducible artifact. Some time later, another bright B-star was observed, and the PI contacted us (L. Oskinova 2010, private communication) to ask what could be the bump in the LETG/HRC-S spectrum of α Cru (ObsID 8937)? This rang a bell. We finally came up with a plausible instrumental effect which would only affect UV-bright B-stars (very high order diffraction by the LETG coarse-support structure), and the Calibration Group provided some supporting quantitative analysis (Drake 2010). Case closed.