BinAC2/Data Compression

Compressing data

Uncompressed data wastes a lot of space on the workspace file system. Most tools read compressed files directly, so there is rarely a reason to keep files uncompressed. On BinAC 2, most uncompressed data is bioinformatics data, so most file types below come from that field, but the general rules apply to all text based formats.

General rules:

• Write output compressed in the first place instead of compressing it afterwards.
• Run conversions in a batch job, not on the login nodes.
• Delete the uncompressed original only after the check shown for each file type succeeded.
• Use the default compression level. Higher levels (-9) take 2 to 7 times longer and save only 0.4 to 1.6 percentage points.

Overview

Format	Use instead	Tool	Size after compression	Tested tools reading it
SAM	sorted BAM	samtools sort	about 27 %	samtools
FASTQ	.fastq.gz	pigz	about 35 %	bowtie2, Bismark
FASTA	.fa.gz	bgzip	about 25 %	samtools faidx, bowtie2-build

File types

SAM

SAM is the uncompressed text form of BAM. It contains exactly the same information. Do not keep SAM files.

Expected results

Measured on BinAC 2 with bowtie2 2.5.5 alignments of NA12878 (2x101 bp) to GRCh38, samtools 1.24 defaults:

Data set	SAM	unsorted BAM	sorted BAM
20 M read pairs	13.33 GB	4.29 GB (32 %)	3.62 GB (27 %)

Sorted BAM compresses better than unsorted BAM.

CRAM, measured on BinAC 2 with the sorted BAM of the 20 M read pairs (3.62 GB), samtools 1.24, 8 threads, node local disk:

Format	Size	of BAM	Write time	Read time
sorted BAM	3.62 GB	100 %
CRAM (default)	1.47 GB	41 %	37 s	17 s
CRAM small profile	1.33 GB	37 %	55 s	33 s
CRAM archive profile	1.28 GB	35 %	81 s	31 s

samtools 1.24 writes CRAM version 3.1 by default. Relative to the original SAM, a default CRAM is about 11 % of the size.

All alignment data was preserved: flagstat and all fields of all 40 M records were identical, with one exception allowed by the SAM specification. For 22 read pairs where both mates start at the same position, the sign of the template length (TLEN) was swapped between the mates.

Compress

module load bio/samtools/1.24
samtools sort -@ 8 -m 1G -o sample.bam sample.sam
samtools quickcheck sample.bam \
  && samtools flagstat -@ 8 sample.sam > sam.flagstat \
  && samtools flagstat -@ 8 sample.bam > bam.flagstat \
  && cmp sam.flagstat bam.flagstat \
  && rm sample.sam
samtools index sample.bam

-m 1G is per thread, so request at least 8 GB memory for -@ 8. The SAM file is only deleted if the BAM is intact and contains the same reads.

For new analyses, avoid SAM files entirely and pipe the aligner output into samtools sort:

bowtie2 -p 8 -x GRCh38 -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz \
  | samtools sort -@ 4 -m 1G -o sample.bam -
samtools index sample.bam

Optional, for long term storage, CRAM (default profile; small or archive save another 10 to 13 % at 1.5 to 2 times the write time):

samtools view -@ 8 -C -T GRCh38.fa -o sample.cram sample.bam \
  && samtools quickcheck sample.cram \
  && cmp <(samtools flagstat sample.bam) <(samtools flagstat --input-fmt-option reference=GRCh38.fa sample.cram) \
  && rm sample.bam
samtools index sample.cram

• A CRAM file cannot be read without the exact reference FASTA used to create it. Keep the reference together with the CRAM files.
• Not all tools read CRAM. Check your downstream tools before converting.

Decompress

Rarely needed. To look into a BAM file:

samtools view -h sample.bam | less

Tools reading the compressed format

Tool	BAM	CRAM
samtools	yes	yes (needs reference)

Only tools tested on BinAC 2 are listed. More will be added.

FASTQ

Expected results

Measured on BinAC 2 with 8 threads on node local disk. Test data: 20 M reads of NA12878 (ERR194147, HiSeq 2000, 2x101 bp), 5.4 GB uncompressed, one file.

Method	Size after compression	Compression time	Decompression time
pigz (default level)	35.3 %	83 s	17 s
bgzip (default level)	35.6 %	28 s	2 s (8 threads), 7 s (1 thread)
bgzip -l 9	34.0 %	206 s	2 s

This data has full resolution quality scores. Data from newer sequencers with binned quality scores usually compresses better.

Both pigz and bgzip give the same compression ratio. Use pigz: it writes a plain gzip file that every tool reading .fastq.gz understands. bgzip files are also valid gzip files, so if you already have them, keep them.

Compress

module load system/pigz/2.8
pigz -p 8 -k sample.fastq \
  && pigz -t sample.fastq.gz \
  && rm sample.fastq

-k keeps the original until the integrity check (-t) succeeded. pigz is a parallel gzip; the output is a normal .gz file that gzip and zcat can read.

Write trimmed or filtered reads compressed directly, e.g. by giving the output file a .fastq.gz name if the tool supports it.

Decompress

zcat sample.fastq.gz | head     # look into the file without decompressing it
pigz -d sample.fastq.gz         # decompress (rarely needed)

If a tool does not read gzip, do not decompress the file. Use process substitution instead:

some_tool --in <(zcat sample.fastq.gz)

This only works if the tool reads its input once from start to end.

Tools reading the compressed format

Tool	Gzipped FASTQ	Notes
bowtie2	yes	native support
Bismark	yes

Only tools tested on BinAC 2 are listed. More will be added.

FASTA

Expected results

Measured on BinAC 2 with 8 threads on node local disk. Test data: GRCh38 no_alt analysis set, 3.1 GB uncompressed.

Method	Size after compression	Compression time	Decompression time
pigz (default level)	27.8 %	74 s	7 s
pigz -9	26.6 %	335 s	8 s
bgzip (default level)	25.2 %	67 s	1 s
bgzip -l 9	24.8 %	131 s	1 s

bgzip gives the smallest files and supports indexing. Use it for all FASTA files.

Compress

bgzip writes gzip compatible files in independent blocks. Every tool that reads .gz also reads bgzip files. In addition, samtools can index them, so tools can read a region without decompressing the whole file. Plain gzip files cannot be indexed:

[E::fai_build3_core] Cannot index files compressed with gzip, please use bgzip

module load bio/samtools/1.24
bgzip -@ 8 -k genome.fa \
  && bgzip -t genome.fa.gz \
  && samtools faidx genome.fa.gz \
  && rm genome.fa

samtools faidx creates genome.fa.gz.fai and genome.fa.gz.gzi. Keep both next to the .fa.gz file. The index can also be created later.

To check whether an existing .gz file is bgzip or plain gzip, and to convert it:

htsfile genome.fa.gz          # "BGZF-compressed" or "gzip-compressed"
zcat genome.fa.gz | bgzip -@ 8 > genome.bgz.fa.gz && samtools faidx genome.bgz.fa.gz

Decompress

samtools faidx genome.fa.gz chr1:1000000-1000100   # extract a region
bgzip -d genome.fa.gz                               # decompress (rarely needed)

Tools reading the compressed format

Tool	Compressed FASTA	Notes
samtools faidx	bgzip only	not plain gzip
bowtie2-build	yes	since bowtie2 2.3.3

Only tools tested on BinAC 2 are listed. More will be added.