BinAC2/Software/Bowtie2

The main documentation is available on the cluster via module help bio/bowtie2/2.5.5. Most software modules for applications provide working example batch scripts.

Description	Content
module load	bio/bowtie2/2.5.5
License	GPL-3.0, see [1]
Citing	Langmead B, Salzberg SL. Fast gapped-read alignment with Bowtie 2. Nature Methods 9, 357 to 359 (2012), [2]
Links	Bowtie 2 Website: [3] Bowtie 2 Manual: [4] Bowtie 2 Repository: [5]

Description

Bowtie 2 is a fast and memory-efficient aligner for short sequencing reads against a reference genome.

This page describes the module bio/bowtie2/2.5.5. All options, recommendations and run times on this page were measured with this version. The older module bio/bowtie2/2.5.4 is also installed, but not covered here.

The module is built from source for BinAC 2:

• Built with GCC 14.2 and -march=x86-64-v3; it runs on all BinAC 2 compute nodes.
• Index building uses libsais, which is much faster than the classic algorithm.
• Compressed reads (.fastq.gz) are read with zlib-ng.
• No compiler module is needed: the C++ and OpenMP runtimes are part of the binaries.

Usage

Loading the module

Load the module in your job script; lib/zlib-ng-compat/2.3.3 is loaded with it automatically.

module load bio/bowtie2/2.5.5

Only one bowtie2 module can be loaded at a time.

Command	Purpose
bowtie2	align reads to an index
bowtie2-build	build an index from a reference FASTA
bowtie2-inspect	show the contents of an index

Variable	Content
BOWTIE2_VERSION	version, 2.5.5
BOWTIE2_HOME	installation directory
BOWTIE2_BIN_DIR	directory of the commands

Building an index

bowtie2-build chooses the index type itself: references up to about 2 GiB of uncompressed FASTA get a small index (.bt2), larger ones a large index (.bt2l). Human and most mammalian genomes therefore get a large index; you do not need --large-index. bowtie2 finds either type by the index name.

For Zstandard-compressed references (.fa.zst), bowtie2-build cannot check the size. For such references above 2 GiB, pass --large-index; otherwise the build stops with an error message.

Measured index builds, 32 CPUs on a Zen 3 node of the compute partition (EPYC 7543):

Reference	Size (FASTA, uncompressed)	Index type	Wall time	Peak memory	Recommended --mem
GRCh38, no-alt analysis set	about 3.2 GB	large (.bt2l)	9.6 min	24.6 GiB	32G

Most of the build uses only a few CPUs (4.2 on average), but one phase uses all 32.

Your reference genome is not in the table? Send us the genome (e.g. an NCBI or Ensembl accession) via hpcmaster [at] uni-tuebingen.de. We measure the index build on BinAC 2 and add it to this table.

#!/bin/bash
#SBATCH --job-name=bt2-index
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=32
#SBATCH --mem=32G
#SBATCH --time=01:00:00

module load bio/bowtie2/2.5.5
bowtie2-build --threads ${SLURM_CPUS_PER_TASK} GRCh38.fa GRCh38

This writes GRCh38.1.bt2l to GRCh38.rev.2.bt2l; use -x GRCh38 when aligning.

Aligning reads

This job aligns one paired-end sample in two steps and writes a sorted BAM file to a workspace on the fast work storage:

1. bowtie2 aligns with all CPUs and pipes its output into samtools view, which writes an unsorted BAM file. No SAM file is written.
2. samtools sort sorts the unsorted BAM with all CPUs into the final BAM file.

The job writes into one of your existing workspaces; replace <workspace> in the job script with its name. ws_list shows your workspaces. If you do not have one yet, allocate it with ws_allocate <workspace> <days>.

#!/bin/bash
#SBATCH --job-name=bt2-align
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=32
#SBATCH --mem=40G
#SBATCH --time=10:00:00
set -euo pipefail

module load bio/bowtie2/2.5.5 bio/samtools/1.24

WORKSPACE=<workspace>
OUTDIR=$(ws_find ${WORKSPACE})
[ -n "${OUTDIR}" ] || { echo "workspace ${WORKSPACE} not found, check with ws_list" >&2; exit 1; }

# Intermediate files (unsorted BAM, sort temporary files):
# small samples: node-local $TMPDIR; large samples: use TMP=${OUTDIR} instead
TMP=${TMPDIR}

# 1. align with all CPUs, write an unsorted BAM
bowtie2 -p ${SLURM_CPUS_PER_TASK} -x /path/to/GRCh38 \
        -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz \
  | samtools view -@ 4 -b -o ${TMP}/sample.unsorted.bam -

# 2. sort with all CPUs into the workspace
samtools sort -@ ${SLURM_CPUS_PER_TASK} -T ${TMP}/sample.tmp -o ${OUTDIR}/sample.bam ${TMP}/sample.unsorted.bam
rm -f ${TMP}/sample.unsorted.bam

• Small or large sample: $TMPDIR is the node-local disk. It holds about 450 GB and is shared by all jobs on the node, and Slurm does not reserve space per job. For small samples, keep the intermediate files there; it is the fastest choice. For large samples we cannot guarantee that $TMPDIR has enough space; set TMP=${OUTDIR} to keep them in the workspace. As a guideline (estimate): use $TMPDIR if the compressed FASTQ files of one sample are below about 50 GB in total, roughly a human genome at 25x.
• Space needed: a human genome at about 50x coverage (787M read pairs, 2 x 101 bp) gives an unsorted BAM of 115 GB and a sorted BAM of 106 GB. The sort temporary files come on top during step 2.
• Run time: 50x took 4.1 hours on a node with EPYC 75F3 not shared with other jobs: 3.8 hours for step 1, 13 to 16 minutes for step 2. On nodes with EPYC 7543 or on a node shared with other jobs it takes longer, about 4.5 to 5 hours (estimate). A 30x genome takes about 2.5 to 3 hours (estimate).
• Memory: step 1 needs about 5 GiB, step 2 with 32 sort threads 28 GiB. The job ran with --mem=40G.
• CPUs: samtools view -@ 4 runs on top of the 32 bowtie2 threads. It needs very little CPU, and taking CPUs away from bowtie2 for it was slower in our tests.
• After the analysis: keep the BAM in the workspace while you work with it. When you no longer need it there, move it to your project directory, since workspaces expire.

Why two steps?

We compared the two-step job above with a single pipe into samtools sort and with writing a SAM file first. Full 50x genome, output in a workspace on work, nodes not shared with other jobs:

Variant	Run time	Node	Intermediate file
Two steps, intermediate files in $TMPDIR (example job)	4.05 h	EPYC 75F3	unsorted BAM, 115 GB
Two steps, intermediate files in the workspace	4.09 h	EPYC 75F3	unsorted BAM, 115 GB
One pipe: bowtie2 -p 26 | samtools sort -@ 6	4.73 h	EPYC 75F3	none
SAM file, then samtools sort	4.66 h	EPYC 7543	SAM file, 525 GB

• Two steps are about 14 % faster than the single pipe. In the pipe, samtools sort slows bowtie2 down, even when bowtie2 keeps all 32 CPUs. samtools view does not.
• The unsorted BAM is only slightly larger than the sorted one (115 GB vs 106 GB), while a SAM file is about five times as large (525 GB). Do not write SAM files.
• The sort step is faster with its temporary files in $TMPDIR (13 vs 16 minutes); where the unsorted BAM is read from made no difference.

Resources and Efficiency

Request 16 to 32 CPUs per sample and give bowtie2 all of them with -p; for many samples, use a job array with one sample per job.

Measured with bowtie2 2.5.5 and 20M read pairs (NA12878, 2 x 101 bp) against GRCh38 on a Zen 3 node of the compute partition with AMD EPYC 7543 (nodes with EPYC 75F3 were about 8 % faster in one comparison):

--cpus-per-task	Physical cores	Wall time (min)	Core time relative to 16 CPUs	Peak memory (GiB)
16	8	12.9	1.00	4.4
32	16	6.9	1.07	4.7
64	32	4.2	1.30	5.5
128	64	3.1	1.92	6.9

Run time grows linearly with the number of reads: a full genome at about 50x took 26.7 s per million read pairs with 32 CPUs, the 20M pair run 24.6 s.

Setting	Recommendation	Basis
Threads	bowtie2 -p = --cpus-per-task; samtools view -@ 4 on top; samtools sort -@ = --cpus-per-task	measured: as fast as bowtie2 writing a file directly
CPUs per sample	16 to 32	measured: 32 CPUs cost 7 % more core time than 16, 64 CPUs 30 % more
Memory, alignment against a human index	--mem=8G for bowtie2 alone, --mem=40G for the two-step example job	measured: bowtie2 4.4 to 6.9 GiB for 16 to 128 CPUs; samtools sort -@ 32 28 GiB on a full genome, tested at --mem=40G
Memory, building a human index	--mem=32G	measured: peak 24.6 GiB, tested at --mem=32G
Run time	--time=10:00:00 for one genome at up to 50x with 32 CPUs	measured: 4.1 hours at 50x on EPYC 75F3, longer on EPYC 7543 or shared nodes
Disk space	unsorted BAM 115 GB plus sort temporary files (in $TMPDIR or the workspace), sorted BAM 106 GB in the workspace, per 50x genome	measured
Many samples	one job per sample, e.g. a job array	measured: core time per sample rises with the CPU count
Several bowtie2 on one node with the same index	add --mm so they share one copy of the index in memory	bowtie2 manual

• A Slurm CPU is a hardware thread. On the Zen 3 compute nodes, --cpus-per-task=32 gives 16 physical cores with two threads each. bowtie2 uses the second thread well (27 to 40 % faster on the same cores up to 32 cores), so request CPUs as usual and do not use --hint=nomultithread.
• Compressed reads work directly. Pass .fastq.gz files directly; bowtie2 decompresses them while aligning, so there is no need to unpack the reads first.
• Avoid --reorder unless you need the input order. It makes bowtie2 hold back finished reads until all earlier ones are written, which can cost time and memory.

Where to Keep Your Data

Reads and index can stay on project storage; there is no need to copy them to $TMPDIR first. bowtie2 reads its input at well under 10 MB/s, which every storage on BinAC 2 delivers.

Reads and index	Sort temporary files and BAM	Wall time (s)
project	$TMPDIR	452
copied to $TMPDIR first	$TMPDIR	453, plus 10 for the copy
work	work	456

Earlier single pipe (bowtie2 -p 28 | samtools sort -@ 4), 20M read pairs, one Zen 3 node of the compute partition (EPYC 75F3), one run each. The full 50x genome, read directly from project, took 26.7 s per million read pairs, close to the 24.6 s of the 20M pair run.

Where the sort temporary files and the BAM are written does matter. Measured with the earlier single pipe (bowtie2 -p 28 | samtools sort -@ 4), full 50x genome (787M read pairs, reads from project), one exclusive Zen 3 node of the compute partition (EPYC 75F3), one run each:

Sort temporary files and BAM	Wall time	Compared to $TMPDIR
$TMPDIR (local NVMe)	4 h 39 min
work (NVMe)	4 h 48 min	+3 %
project (HDD)	4 h 59 min	+7 %

Copying the finished 106 GB BAM from $TMPDIR takes about 1 to 2 minutes:

Destination	Default layout (1 stripe)	lfs setstripe -c 4
project (HDD)	78 to 120 s (0.9 to 1.4 GB/s)	55 to 57 s (1.9 GB/s)
work (NVMe)	59 s (1.8 GB/s)	53 s (2.0 GB/s)

• Reads and index: read them directly from project.
• Intermediate files (unsorted BAM, sort temporary files): for small samples in $TMPDIR, the fastest choice. For large samples in the workspace: the node-local disk holds about 450 GB and is shared by all jobs on the node, up to four jobs with 32 CPUs each, and Slurm does not reserve space per job. If several large jobs land on one node, they can fill the disk and fail hours into the run.
• Final BAM: in the workspace, as in the example job.
• When the analysis is done: move the BAM files from the workspace to your project directory. Workspaces expire, and data in an expired workspace is deleted.
• Why not project for the BAM and the intermediate files: writing them directly to project costs about 7 % (20 minutes for a 50x genome).

Benchmark Data

The reference genome and the read files used for the measurements on this page are in /pfs/10/project/benchmark_data:

Data	Path
Reference, GRCh38 no-alt analysis set	/pfs/10/project/benchmark_data/reference_genomes/GRCh38_GIAB_references/GCA_000001405.15_GRCh38_no_alt_analysis_set.fasta.gz
Reads, NA12878 full genome, about 50x (787M read pairs, 2 x 101 bp)	/pfs/10/project/benchmark_data/wgs/giab_NA12878_truseq_30x/ERR194147/ERR194147_1.fastq.gz /pfs/10/project/benchmark_data/wgs/giab_NA12878_truseq_30x/ERR194147/ERR194147_2.fastq.gz

Changes Compared to Earlier Builds

• The module no longer loads compiler/gnu/14.2. Job scripts that relied on it being loaded must now load it themselves.
• References above 2 GiB now always get a large index (.bt2l)

Support

If you have questions or problems, contact hpcmaster [at] uni-tuebingen.de..