Skill Tracer
Skills › Scientific Agent Skills › polars-bio › pileup_operations.md

pileup_operations.md

118 lines · skills/polars-bio/references/pileup_operations.md at 5f2bc03

Open on GitHub
1# Read depth (polars-bio 0.36.0)
2
3`pb.depth()` computes CIGAR-aware read depth for BAM, SAM and self-contained
4CRAM input. This differs from `pb.coverage(query, targets)`, which measures
5union-covered interval length. See the [depth API](https://biodatageeks.org/polars-bio/api/operations/)
6and [pinned native coverage implementation](https://github.com/biodatageeks/datafusion-bio-functions/blob/v0.22.2/datafusion/bio-function-pileup/src/coverage.rs).
7

Read depth (polars-bio 0.36.0)

pb.depth() computes CIGAR-aware read depth for BAM, SAM and self-contained CRAM input. This differs from pb.coverage(query, targets), which measures union-covered interval length. See the depth API and pinned native coverage implementation.

Raw lines 1 to 7
8## Arguments and output
9
10```python
11# Template: supply a validated alignment file.
12import polars_bio as pb
13blocks = pb.depth("sample.bam", use_zero_based=True,
14 filter_flag=1796, min_mapping_quality=20).collect()
15```
17| Argument | Default | Meaning |
18|---|---|---|
19| `filter_flag` | `1796` | Exclude reads carrying any bit in mask |
20| `min_mapping_quality` | `0` | Minimum MAPQ; no base-quality filtering |
21| `binary_cigar` | `True` | Binary CIGAR path |
22| `dense_mode` | `"auto"` | Dense when contig lengths are available; also `"force"` / `"disable"` |
23| `use_zero_based` | `None` | Session convention; True requests half-open/zero-based |
24| `per_base` | `False` | True requires dense accumulation with contig lengths |
25| `output_type` | `"polars.LazyFrame"` | Also `"polars.DataFrame"` or optional `"pandas.DataFrame"` |
27Block columns: `contig` String, `pos_start`/`pos_end` Int32, `coverage` Int16.
28Per-base columns: `contig` String, `pos` Int32, `coverage` Int16. The default
29block output omits zero-depth blocks in the exercised fixtures. Per-base dense
30output includes zeroes across represented contig lengths. Check which contigs
31were emitted, especially contigs with no reads, rather than assuming a whole
32reference denominator. Dense accumulation needs memory proportional to contig
33length even when the returned run-length encoding is small.
35Index files are auto-discovered but a sequential SAM/BAM scan works without
36one. The Python depth API has no `reference_path`; external-reference CRAM needs
37a suitable conversion to BAM, whereas `read_cram` can accept a local reference.
38Do not pass CRAM-reader or cloud-I/O keyword arguments that depth does not expose.

Arguments and output

# Template: supply a validated alignment file.
import polars_bio as pb
blocks = pb.depth("sample.bam", use_zero_based=True,
                   filter_flag=1796, min_mapping_quality=20).collect()
ArgumentDefaultMeaning
filter_flag1796Exclude reads carrying any bit in mask
min_mapping_quality0Minimum MAPQ; no base-quality filtering
binary_cigarTrueBinary CIGAR path
dense_mode"auto"Dense when contig lengths are available; also "force" / "disable"
use_zero_basedNoneSession convention; True requests half-open/zero-based
per_baseFalseTrue requires dense accumulation with contig lengths
output_type"polars.LazyFrame"Also "polars.DataFrame" or optional "pandas.DataFrame"

Block columns: contig String, pos_start/pos_end Int32, coverage Int16. Per-base columns: contig String, pos Int32, coverage Int16. The default block output omits zero-depth blocks in the exercised fixtures. Per-base dense output includes zeroes across represented contig lengths. Check which contigs were emitted, especially contigs with no reads, rather than assuming a whole reference denominator. Dense accumulation needs memory proportional to contig length even when the returned run-length encoding is small.

Index files are auto-discovered but a sequential SAM/BAM scan works without one. The Python depth API has no reference_path; external-reference CRAM needs a suitable conversion to BAM, whereas read_cram can accept a local reference. Do not pass CRAM-reader or cloud-I/O keyword arguments that depth does not expose.

Raw lines 8 to 39
40## Filters and CIGAR semantics
42Default 1796 = 4 + 256 + 512 + 1024: excludes unmapped, secondary, QC-failed and
43duplicate reads. Supplementary (2048) is **not** excluded. If the protocol calls
44for excluding it, use `filter_flag=1796 | 2048`. MAPQ 255 often means unavailable
45mapping quality; a lower-bound threshold does not automatically exclude it.
47Native CIGAR behavior, verified with a synthetic spliced/deletion-bearing read:
49| Operation | Coverage effect |
50|---|---|
51| `M`, `=`, `X` | Count aligned reference positions |
52| `D`, `N` | Advance reference position without coverage |
53| `I`, `S`, `H`, `P` | Do not add covered reference positions |
55For a read starting at SAM POS 2 with `3M2D2M3N2M`, half-open covered blocks are
56`[1,4)`, `[6,8)`, `[11,13)`. Deletions are not counted as aligned bases.
57The API does not expose minimum base quality or paired-mate overlap correction;
58report this as read coverage, not fragment coverage or a nucleotide pileup.
59Align filter semantics before comparing with samtools/mosdepth defaults.

Filters and CIGAR semantics

Default 1796 = 4 + 256 + 512 + 1024: excludes unmapped, secondary, QC-failed and duplicate reads. Supplementary (2048) is not excluded. If the protocol calls for excluding it, use filter_flag=1796 | 2048. MAPQ 255 often means unavailable mapping quality; a lower-bound threshold does not automatically exclude it.

Native CIGAR behavior, verified with a synthetic spliced/deletion-bearing read:

OperationCoverage effect
M, =, XCount aligned reference positions
D, NAdvance reference position without coverage
I, S, H, PDo not add covered reference positions

For a read starting at SAM POS 2 with 3M2D2M3N2M, half-open covered blocks are [1,4), [6,8), [11,13). Deletions are not counted as aligned bases. The API does not expose minimum base quality or paired-mate overlap correction; report this as read coverage, not fragment coverage or a nucleotide pileup. Align filter semantics before comparing with samtools/mosdepth defaults.

Raw lines 40 to 60
61## Native numerical limitation
63In 0.36.0, depth accumulation is emitted via a narrowing Int16 cast. A tiny
64fixture of **32,768 reads at one base returned -32,768**. This is silent overflow,
65not negative biological coverage. High positive counts may also wrap to plausible
66small values, so merely checking for negatives is insufficient. Choose an
67independent wider-depth implementation when such depths are possible; widening
68the already emitted result does not repair it. Cast to Int64 before subsequent
69multiplication/summation only when source depth is known to be within range.

Native numerical limitation

In 0.36.0, depth accumulation is emitted via a narrowing Int16 cast. A tiny fixture of 32,768 reads at one base returned -32,768. This is silent overflow, not negative biological coverage. High positive counts may also wrap to plausible small values, so merely checking for negatives is insufficient. Choose an independent wider-depth implementation when such depths are possible; widening the already emitted result does not repair it. Cast to Int64 before subsequent multiplication/summation only when source depth is known to be within range.

Raw lines 61 to 70
71## Correct summary denominators
73Do not average block rows: long and short blocks represent different numbers of
74bases. This executed synthetic example includes known zero-depth target space:
76```python
77import polars as pl
79# Half-open blocks entirely within a known 20-base assay target.
80blocks = pl.DataFrame({"pos_start": [0, 10], "pos_end": [10, 15],
81 "coverage": [2, 4]})
82target_bases = 20
83summary = blocks.select(
84 ((pl.col("pos_end").cast(pl.Int64) - pl.col("pos_start").cast(pl.Int64))
85 * pl.col("coverage").cast(pl.Int64)).sum().alias("aligned_base_total")
87mean_depth_over_target = summary["aligned_base_total"][0] / target_bases
88assert mean_depth_over_target == 2.0 # 10*2 + 5*4 over 20 bases
89```
91For a real panel, merge/normalize target intervals to a nonoverlapping union,
92intersect blocks with those targets, calculate **clipped intersection lengths**,
93then divide the weighted total by the entire union length, including uncovered
94bases. A left overlap join alone does not clip bounds; repeated overlapping
95targets can double count. For closed intervals the length formula adds one.
96A block-row median is not a base-weighted median; build a depth histogram weighted
97by represented bases and add the zero-depth remainder instead.

Correct summary denominators

Do not average block rows: long and short blocks represent different numbers of bases. This executed synthetic example includes known zero-depth target space:

import polars as pl

# Half-open blocks entirely within a known 20-base assay target.
blocks = pl.DataFrame({"pos_start": [0, 10], "pos_end": [10, 15],
                        "coverage": [2, 4]})
target_bases = 20
summary = blocks.select(
    ((pl.col("pos_end").cast(pl.Int64) - pl.col("pos_start").cast(pl.Int64))
      * pl.col("coverage").cast(pl.Int64)).sum().alias("aligned_base_total")
)
mean_depth_over_target = summary["aligned_base_total"][0] / target_bases
assert mean_depth_over_target == 2.0  # 10*2 + 5*4 over 20 bases

For a real panel, merge/normalize target intervals to a nonoverlapping union, intersect blocks with those targets, calculate clipped intersection lengths, then divide the weighted total by the entire union length, including uncovered bases. A left overlap join alone does not clip bounds; repeated overlapping targets can double count. For closed intervals the length formula adds one. A block-row median is not a base-weighted median; build a depth histogram weighted by represented bases and add the zero-depth remainder instead.

Raw lines 71 to 98
99## Adequately covered regions and gaps
101```python
102# Template: this call explicitly requests half-open coordinates.
103depth = pb.depth("sample.bam", use_zero_based=True).collect()
104intervals = depth.rename({"contig": "chrom", "pos_start": "start", "pos_end": "end"})
105intervals.config_meta.set(coordinate_system_zero_based=True)
106adequate = intervals.filter(pl.col("coverage") >= 30)
107# min_dist=1 merges adjacent integer half-open blocks in 0.36.0.
108merged = pb.merge(adequate, min_dist=1).collect()
110# Bounds must match the assembly or assay being assessed.
111genome = pl.DataFrame({"chrom": ["chr1"], "start": [0], "end": [1000]})
112genome.config_meta.set(coordinate_system_zero_based=True)
113gaps = pb.complement(adequate, view_df=genome).collect()
114```
116For panel QC use the panel bounds, not arbitrary whole-chromosome sizes. Report
117assembly, mask/quality filters, read versus fragment semantics, coordinate
118convention, low-coverage threshold and denominator together.

Adequately covered regions and gaps

# Template: this call explicitly requests half-open coordinates.
depth = pb.depth("sample.bam", use_zero_based=True).collect()
intervals = depth.rename({"contig": "chrom", "pos_start": "start", "pos_end": "end"})
intervals.config_meta.set(coordinate_system_zero_based=True)
adequate = intervals.filter(pl.col("coverage") >= 30)
# min_dist=1 merges adjacent integer half-open blocks in 0.36.0.
merged = pb.merge(adequate, min_dist=1).collect()

# Bounds must match the assembly or assay being assessed.
genome = pl.DataFrame({"chrom": ["chr1"], "start": [0], "end": [1000]})
genome.config_meta.set(coordinate_system_zero_based=True)
gaps = pb.complement(adequate, view_df=genome).collect()

For panel QC use the panel bounds, not arbitrary whole-chromosome sizes. Report assembly, mask/quality filters, read versus fragment semantics, coordinate convention, low-coverage threshold and denominator together.

Raw lines 99 to 118