F Bartusch: Created page with "= Compressing data = TOC Uncompressed data wastes a lot of space on the workspace file system. Most tools read compressed files directly, so there is rarely a reason to keep files uncompressed. On BinAC 2, most uncompressed data is bioinformatics data, so most file types below come from that field, but the general rules apply to all text based formats. General rules: * Write output compressed in the first place instead of compressing it afterwards. * Run conversio..."

2026-11-03T05:57:56Z

Created page with "= Compressing data = __TOC__ Uncompressed data wastes a lot of space on the workspace file system. Most tools read compressed files directly, so there is rarely a reason to keep files uncompressed. On BinAC 2, most uncompressed data is bioinformatics data, so most file types below come from that field, but the general rules apply to all text based formats. General rules: * Write output compressed in the first place instead of compressing it afterwards. * Run conversio..."

New page

= Compressing data =

__TOC__

Uncompressed data wastes a lot of space on the workspace file system. Most tools read compressed files directly, so there is rarely a reason to keep files uncompressed. On BinAC 2, most uncompressed data is bioinformatics data, so most file types below come from that field, but the general rules apply to all text based formats.

General rules:
* Write output compressed in the first place instead of compressing it afterwards.
* Run conversions in a batch job, not on the login nodes.
* Delete the uncompressed original only after the check shown for each file type succeeded.
* Use the default compression level. Higher levels (<code>-9</code>) take 2 to 7 times longer and save only 0.4 to 1.6 percentage points.

== Overview ==

{| class="wikitable"
! Format !! Use instead !! Tool !! Size after compression !! Tested tools reading it
|-
| SAM || sorted BAM || <code>samtools sort</code> || about 27 % || samtools
|-
| FASTQ || <code>.fastq.gz</code> || <code>pigz</code> || about 35 % || bowtie2, Bismark
|-
| FASTA || <code>.fa.gz</code> || <code>bgzip</code> || about 25 % || samtools faidx, bowtie2-build
|}

== File types ==

=== SAM ===

SAM is the uncompressed text form of BAM. It contains exactly the same information. Do not keep SAM files.

==== Expected results ====

Measured on BinAC 2 with bowtie2 2.5.5 alignments of NA12878 (2x101 bp) to GRCh38, samtools 1.24 defaults:

{| class="wikitable"
! Data set !! SAM !! unsorted BAM !! sorted BAM
|-
| 20 M read pairs || 13.33 GB || 4.29 GB (32 %) || 3.62 GB (27 %)
|}

Sorted BAM compresses better than unsorted BAM.

CRAM, measured on BinAC 2 with the sorted BAM of the 20 M read pairs (3.62 GB), samtools 1.24, 8 threads, node local disk:

{| class="wikitable"
! Format !! Size !! of BAM !! Write time !! Read time
|-
| sorted BAM || 3.62 GB || 100 % || ||
|-
| CRAM (default) || 1.47 GB || 41 % || 37 s || 17 s
|-
| CRAM <code>small</code> profile || 1.33 GB || 37 % || 55 s || 33 s
|-
| CRAM <code>archive</code> profile || 1.28 GB || 35 % || 81 s || 31 s
|}

samtools 1.24 writes CRAM version 3.1 by default. Relative to the original SAM, a default CRAM is about 11 % of the size.

All alignment data was preserved: flagstat and all fields of all 40 M records were identical, with one exception allowed by the SAM specification. For 22 read pairs where both mates start at the same position, the sign of the template length (TLEN) was swapped between the mates.

==== Compress ====

<pre>
module load bio/samtools/1.24
samtools sort -@ 8 -m 1G -o sample.bam sample.sam
samtools quickcheck sample.bam \
&& samtools flagstat -@ 8 sample.sam > sam.flagstat \
&& samtools flagstat -@ 8 sample.bam > bam.flagstat \
&& cmp sam.flagstat bam.flagstat \
&& rm sample.sam
samtools index sample.bam
</pre>

<code>-m 1G</code> is per thread, so request at least 8 GB memory for <code>-@ 8</code>. The SAM file is only deleted if the BAM is intact and contains the same reads.

For new analyses, avoid SAM files entirely and pipe the aligner output into <code>samtools sort</code>:

<pre>
bowtie2 -p 8 -x GRCh38 -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz \
| samtools sort -@ 4 -m 1G -o sample.bam -
samtools index sample.bam
</pre>

Optional, for long term storage, CRAM (default profile; <code>small</code> or <code>archive</code> save another 10 to 13 % at 1.5 to 2 times the write time):

<pre>
samtools view -@ 8 -C -T GRCh38.fa -o sample.cram sample.bam \
&& samtools quickcheck sample.cram \
&& cmp <(samtools flagstat sample.bam) <(samtools flagstat --input-fmt-option reference=GRCh38.fa sample.cram) \
&& rm sample.bam
samtools index sample.cram
</pre>

* A CRAM file cannot be read without the '''exact''' reference FASTA used to create it. Keep the reference together with the CRAM files.
* Not all tools read CRAM. Check your downstream tools before converting.

==== Decompress ====

Rarely needed. To look into a BAM file:

<pre>
samtools view -h sample.bam | less
</pre>

==== Tools reading the compressed format ====

{| class="wikitable"
! Tool !! BAM !! CRAM
|-
| samtools || yes || yes (needs reference)
|}

Only tools tested on BinAC 2 are listed. More will be added.

=== FASTQ ===

==== Expected results ====

Measured on BinAC 2 with 8 threads on node local disk. Test data: 20 M reads of NA12878 (ERR194147, HiSeq 2000, 2x101 bp), 5.4 GB uncompressed, one file.

{| class="wikitable"
! Method !! Size after compression !! Compression time !! Decompression time
|-
| <code>pigz</code> (default level) || 35.3 % || 83 s || 17 s
|-
| <code>bgzip</code> (default level) || 35.6 % || 28 s || 2 s (8 threads), 7 s (1 thread)
|-
| <code>bgzip -l 9</code> || 34.0 % || 206 s || 2 s
|}


This data has full resolution quality scores. Data from newer sequencers with binned quality scores usually compresses better.

Both <code>pigz</code> and <code>bgzip</code> give the same compression ratio. Use <code>pigz</code>: it writes a plain gzip file that every tool reading <code>.fastq.gz</code> understands. <code>bgzip</code> files are also valid gzip files, so if you already have them, keep them.

==== Compress ====

<pre>
module load system/pigz/2.8
pigz -p 8 -k sample.fastq \
&& pigz -t sample.fastq.gz \
&& rm sample.fastq
</pre>

<code>-k</code> keeps the original until the integrity check (<code>-t</code>) succeeded. <code>pigz</code> is a parallel <code>gzip</code>; the output is a normal <code>.gz</code> file that <code>gzip</code> and <code>zcat</code> can read.

Write trimmed or filtered reads compressed directly, e.g. by giving the output file a <code>.fastq.gz</code> name if the tool supports it.

==== Decompress ====

<pre>
zcat sample.fastq.gz | head # look into the file without decompressing it
pigz -d sample.fastq.gz # decompress (rarely needed)
</pre>

If a tool does not read gzip, do not decompress the file. Use process substitution instead:

<pre>
some_tool --in <(zcat sample.fastq.gz)
</pre>

This only works if the tool reads its input once from start to end.

==== Tools reading the compressed format ====

{| class="wikitable"
! Tool !! Gzipped FASTQ !! Notes
|-
| bowtie2 || yes || native support
|-
| Bismark || yes ||
|}

Only tools tested on BinAC 2 are listed. More will be added.

=== FASTA ===

==== Expected results ====

Measured on BinAC 2 with 8 threads on node local disk. Test data: GRCh38 no_alt analysis set, 3.1 GB uncompressed.

{| class="wikitable"
! Method !! Size after compression !! Compression time !! Decompression time
|-
| <code>pigz</code> (default level) || 27.8 % || 74 s || 7 s
|-
| <code>pigz -9</code> || 26.6 % || 335 s || 8 s
|-
| <code>bgzip</code> (default level) || 25.2 % || 67 s || 1 s
|-
| <code>bgzip -l 9</code> || 24.8 % || 131 s || 1 s
|}

<code>bgzip</code> gives the smallest files and supports indexing. Use it for all FASTA files.

==== Compress ====

<code>bgzip</code> writes gzip compatible files in independent blocks. Every tool that reads <code>.gz</code> also reads bgzip files. In addition, samtools can index them, so tools can read a region without decompressing the whole file. Plain gzip files cannot be indexed:

<pre>
[E::fai_build3_core] Cannot index files compressed with gzip, please use bgzip
</pre>

<pre>
module load bio/samtools/1.24
bgzip -@ 8 -k genome.fa \
&& bgzip -t genome.fa.gz \
&& samtools faidx genome.fa.gz \
&& rm genome.fa
</pre>

<code>samtools faidx</code> creates <code>genome.fa.gz.fai</code> and <code>genome.fa.gz.gzi</code>. Keep both next to the <code>.fa.gz</code> file. The index can also be created later.

To check whether an existing <code>.gz</code> file is bgzip or plain gzip, and to convert it:

<pre>
htsfile genome.fa.gz # "BGZF-compressed" or "gzip-compressed"
zcat genome.fa.gz | bgzip -@ 8 > genome.bgz.fa.gz && samtools faidx genome.bgz.fa.gz
</pre>

==== Decompress ====

<pre>
samtools faidx genome.fa.gz chr1:1000000-1000100 # extract a region
bgzip -d genome.fa.gz # decompress (rarely needed)
</pre>

==== Tools reading the compressed format ====

{| class="wikitable"
! Tool !! Compressed FASTA !! Notes
|-
| samtools faidx || bgzip only || not plain gzip
|-
| bowtie2-build || yes || since bowtie2 2.3.3
|}

Only tools tested on BinAC 2 are listed. More will be added.

BinAC2/Data Compression - Revision history