Sequence analysis fqtools: an efficient software suite for modern FASTQ file manipulation

Summary: Many Next Generation Sequencing analyses involve the basic manipulation of input sequence data before downstream processing (e.g. searching for speciﬁc sequences, format conversion or basic ﬁle statistics). The rapidly increasing data volumes involved in NGS make any dataset manipulation a time-consuming and error-prone process. I have developed fqtools; a fast and reliable FASTQ ﬁle manipulation suite that can process the full set of valid FASTQ ﬁles, including those with multi-line sequences, whilst identifying invalid ﬁles. Fqtools is faster than similar tools, and is designed for use in automatic processing pipelines. Availability and implementation: fqtools is open source and is available at: https://github.com/alas tair-droop/fqtools. Supplementary information : Supplementary data are available at Bioinformatics online.


Introduction
The FASTQ format has become the de facto standard for storage of next-generation sequencing read data (Cock et al., 2010).Based originally upon the FASTA sequence format (Pearson and Lipman, 1988), FASTQ stores nucleotide sequences and associated base qualities (Ewing and Green, 1998) for multiple named reads in a fourfield human-readable ASCII format.Although there is no defined standard for FASTQ files, Cock et al. (2010) provide a good overview of the format, and provide as close to a 'standard' as is available.
Many analysis pipelines involve initial data manipulation (e.g.reformatting, viewing or overview statistics) before downstream processing (e.g.quality control, adapter removal and alignment).Seemingly simple tasks like viewing the first few reads in a file or checking the distribution of read lengths often require scripting or loading the data in tools that are quite slow for large datasets.These file manipulations are much more frequent when data are being reused in novel analyses.Frequently, individual researchers will write scripts (e.g. in Python, Perl or AWK) to perform these tasks.Many tools are available for FASTQ processing such as the fastx-toolkit, bio-awk, fastq-tools, fast, seqmagick and seq-tk (see the Supplementary Materials for the URLs of these tools).None of these provide a comprehensive set of common manipulations that would be required for most analyses.
Most FASTQ processing tools fail to process reads with sequence data split across multiple lines.As read lengths from modern sequencing technologies are constantly increasing (Schneider and Dekker, 2012), this is likely to become problematic as human readability is vastly reduced by extremely long lines.
Detecting invalid input is extremely important, as bioinformatics pipelines are often automated; thus significant computation and analysis time can be wasted if input errors are not detected early.For this reason, a trustworthy FASTQ manipulation tool should report invalid files.Similarly, tools should be able to correctly process the full range of valid inputs.
The fqtools suite was written to address this need for efficient and reliable viewing, manipulation and summarization of FASTQ data before it is pre-processed (with e.g.FastQC [http://www.bioinformatics.babraham.ac.uk/projects/fastqc/) or cutadapt (Martin, 2011)].Both compressed and plain FASTQ can be processed, as can SAM and BAM-formatted data.Paired-end sequence data is handled either as file pairs or in interleaved format (Table 1).

Implementation
I have developed a fast and memory-efficient state machine for parsing FASTQ files.The use of a state machine (as opposed to a linebased approach) obviates the difficulties with line breaks in sequence and quality data.As read order must be identical for both paired-end files, manipulations that re-order reads process both pairs simultaneously.
The fqtools suite has been written to allow input and output from either files or standard streams.Both files and streams can contain either plain or gzip-compressed data.By using streams, the fqtools suite can be easily incorporated into computational pipelines.
The commands contained in the fqtools suite are listed in Table 2, along with a brief description of their purpose.

Performance
I tested several common sequence manipulation tools against four criteria: i.The ability to process the full range of valid FASTQ files; ii.The ability to detect the full range of FASTQ errors; iii.The ability to read and write compressed data; and iv.The processing speed.
To evaluate the ability of the fqtools suite to correctly process valid files and to reject invalid ones, I used the test set provided by Cock et al. (2010).The performance of the fqtools suite was tested against several similar tools using a sample file containing 100 000 reads generated using ART (Huang et al., 2012).Table 2 shows these results.For all tools, the closest option to parsing the file without further processing was used.The lowest time score over 50 repeats was taken.Speed results for printing the files on the bash terminal are supplied for a 'maximum speed' reference, although these commands make no attempt to parse the FASTQ data within the file.Full data are available in the supplementary information.
Of the tools tested, only three (fqtools, fast and seqmagick) correctly processed the full test set.Of these three, fqtools is by far the fastest when processing both uncompressed and compressed files.
Although several of these tools are designed to be extendable, data processing speed for basic FASTQ manipulation is becoming increasingly important.For these tasks, the flexibility of interpreted tools (such as seqmagick and fast) is unnecessary.

Summary
Here, I describe fastqc, a suite of FASTQ manipulations tools that efficiently handles the full range of valid FASTQ, whilst detecting invalid files.The suite can process both compressed and uncompressed files.fqtools is freely available on Github at https://github.com/alastair-droop/fqtools.Benchmark data for various FASTQ processing tools.All tools were installed locally, and run against the complete test set (Cock et al., 2010).Valid shows if all the valid test set were processed correctly.Invalid shows if the tool identified all the invalid files.Process .gzshows if the tool can natively read (R) and write (W) gzip-compressed files.The speed columns show the speed in reads per second.

Table 2 .
FASTQ processing tools overview