A pipeline one person understands is not enough
An NGS pipeline is the sequence of computational steps that takes raw sequencer output and produces data a scientist can analyze. Every genomics team has one. The practical question is which of those two it is.
What the pipeline actually does
The raw output from a sequencer arrives as FASTQ files — millions of short reads with quality scores attached to each base. The pipeline works through those reads in stages.
Quality control
Reads are checked for adapter contamination, low quality tails, and overall run health before anything else happens — because a problem caught here is cheap and the same problem caught after alignment is not.
Alignment
Each read is mapped to its position on a reference genome. BWA is the standard tool for this with short reads, and its MEM algorithm handles the read lengths modern sequencers produce. Alignment output lands in SAM or BAM format, and SAMtools sorts, indexes, and inspects those files so downstream tools can work with them efficiently.
Post alignment cleanup
Duplicate reads from PCR amplification are marked so they do not inflate variant evidence, and base quality scores are recalibrated against known variation to correct systematic sequencer error. Both are GATK steps in a typical germline workflow.
From there the pipeline hands off to variant calling, and then to annotation and reporting.
Why pipelines break
Three failure modes account for most of the pain teams run into.
Environment drift
A pipeline that ran correctly on one machine fails on another because a tool version changed, a library was updated, or a reference file moved. Containerization, with Docker or Singularity, and strict version pinning are the standard defenses — but they have to be applied consistently across every step.
Scale
A pipeline written for ten samples does not automatically work for a thousand. Steps that ran serially need to run in parallel, intermediate files that fit on a laptop no longer fit anywhere, and cloud or HPC scheduling becomes part of the design rather than something bolted on at the end.
Reproducibility
A result a reviewer or regulator asks about eighteen months later has to be reproducible exactly, not approximately. For the team, that means the pipeline, its parameters, its tool versions, and its reference data are all captured and retrievable for every run.
Workflow managers
Workflow management systems exist to solve those three problems in a structured way. Nextflow is the most widely used in genomics, with Snakemake and WDL as the other common choices. They handle parallelization, resume on failure, container integration, and execution across local, HPC, and cloud environments from the same pipeline definition.
The community maintained pipelines built on Nextflow, particularly for variant calling and RNA sequencing, are a reasonable starting point for many teams and a reasonable benchmark for anything built custom.
We build NGS pipelines using BWA, GATK, and SAMtools for genomic research and biomarker analysis, engineered for reproducibility and scale from the start.
Where a team wants to build and iterate faster, GenXFlo generates the workflow code from a drag and drop design and runs it in a containerized environment with no dependency setup.
GenXFlo →One public example of the surrounding engineering: a modular mutation calling pipeline, alignment through reporting, with every stage validated in TypeScript and 98 percent automated test coverage on the web application that runs it. Releases got shorter and post release defects fell.
Read the case study →