Skill Tracer
Skills › Scientific Agent Skills › scanpy › upstream-review.md

upstream-review.md

114 lines · skills/scanpy/references/upstream-review.md at 5f2bc03

Open on GitHub
1# Scanpy 1.12.4 review and representation boundaries
2
3Reviewed 2026-10-01. Local tests use Python 3.13, Scanpy 1.12.4 and AnnData
40.13.4. Official stable docs and the installed release source were inspected
5for every Scanpy API family used by the toolkit, assets and references.
6There are no remote analysis/authentication/pagination endpoints in these scripts.
7Package installation and fetching example datasets can use the network; the tests
8use local synthetic inputs and no patient data or atlas downloads.
9

Scanpy 1.12.4 review and representation boundaries

Reviewed 2026-10-01. Local tests use Python 3.13, Scanpy 1.12.4 and AnnData 0.13.4. Official stable docs and the installed release source were inspected for every Scanpy API family used by the toolkit, assets and references. There are no remote analysis/authentication/pagination endpoints in these scripts. Package installation and fetching example datasets can use the network; the tests use local synthetic inputs and no patient data or atlas downloads.

Raw lines 1 to 9
10## Invariants to preserve
12| Operation | Input | Stored/returned result |
13| --- | --- | --- |
14| QC/Scrublet | Identified raw counts; capture/library batch for doublets | QC/doublet columns in `obs`; filtered cells remain counts |
15| `normalize_total`, `log1p` | Counts, then normalized values | X is log-normalized; `counts` is an independent original count copy |
16| `highly_variable_genes` | Log values for seurat/cell_ranger; counts for v3/v3_paper | Boolean `var.highly_variable`; v3 uses scikit-misc; batch tie ordering differs between v3 flavors |
17| PCA | Selected HVGs, optionally scaled | `obsm.X_pca`, `varm.PCs`, `uns.pca`; ARPACK components < min(cells, selected genes) |
18| Neighbors | Explicit representation and available component count | Pairwise distances/connectivities in `obsp`, parameters in `uns.neighbors` |
19| UMAP/Leiden | Neighbor graph | Visualization in `obsm.X_umap`; categorical clusters in `obs`; neither validates cell identity |
20| Marker ranking | Log-normalized X/layer/raw explicitly selected | `uns.rank_genes_groups`; `get.rank_genes_groups_df` returns scores and method-dependent statistics |
21| Aggregation | Raw-count layer, biological sample x cell type | `layers['sum']`, with `X is None`; grouped annotation contains contributing-cell counts |
23`.raw` snapshots X and var, not layers, and does not mean raw counts. Cell slices
24subset `.raw` cells; gene slices leave its original gene axis. The scripts use
25`adata.raw = adata.copy()` before scaling to prevent shared-array mutation. AnnData
260.13 includes the unnamed X key (`None`) when iterating layers; filter for named
27strings where listing extra layers. A view is not an independent copy. Backed `r+`
28persists supported X changes only, not arbitrary annotation edits; write a new
29file after deliberate `.to_memory()` conversion and close the backing file.
31The full pipeline computes on an HVG copy but transfers aligned PCA loadings,
32embeddings, graph and parameters to the full-gene object. `--subset-hvg` explicitly
33shrinks *all* layers: keep the full-gene checkpoint for pseudobulk. Never reconstruct
34missing raw counts from `.raw` or normalized values. CSV/Matrix Market readers need
35known orientation and cell/gene identifiers; bare MTX carries neither identifier
36list. `.h5` is assumed 10x HDF5, not arbitrary HDF5. Loom needs loompy.

Invariants to preserve

OperationInputStored/returned result
QC/ScrubletIdentified raw counts; capture/library batch for doubletsQC/doublet columns in obs; filtered cells remain counts
normalize_total, log1pCounts, then normalized valuesX is log-normalized; counts is an independent original count copy
highly_variable_genesLog values for seurat/cell_ranger; counts for v3/v3_paperBoolean var.highly_variable; v3 uses scikit-misc; batch tie ordering differs between v3 flavors
PCASelected HVGs, optionally scaledobsm.X_pca, varm.PCs, uns.pca; ARPACK components < min(cells, selected genes)
NeighborsExplicit representation and available component countPairwise distances/connectivities in obsp, parameters in uns.neighbors
UMAP/LeidenNeighbor graphVisualization in obsm.X_umap; categorical clusters in obs; neither validates cell identity
Marker rankingLog-normalized X/layer/raw explicitly selecteduns.rank_genes_groups; get.rank_genes_groups_df returns scores and method-dependent statistics
AggregationRaw-count layer, biological sample x cell typelayers['sum'], with X is None; grouped annotation contains contributing-cell counts

.raw snapshots X and var, not layers, and does not mean raw counts. Cell slices subset .raw cells; gene slices leave its original gene axis. The scripts use adata.raw = adata.copy() before scaling to prevent shared-array mutation. AnnData 0.13 includes the unnamed X key (None) when iterating layers; filter for named strings where listing extra layers. A view is not an independent copy. Backed r+ persists supported X changes only, not arbitrary annotation edits; write a new file after deliberate .to_memory() conversion and close the backing file.

The full pipeline computes on an HVG copy but transfers aligned PCA loadings, embeddings, graph and parameters to the full-gene object. --subset-hvg explicitly shrinks all layers: keep the full-gene checkpoint for pseudobulk. Never reconstruct missing raw counts from .raw or normalized values. CSV/Matrix Market readers need known orientation and cell/gene identifiers; bare MTX carries neither identifier list. .h5 is assumed 10x HDF5, not arbitrary HDF5. Loom needs loompy.

Raw lines 10 to 37
38## Statistical and plotting interpretation
40`rank_genes_groups` defaults to `reference='rest'`; named reference groups and
41`groups` control the comparison, not the donor dependence. BH adjustment is over
42tested genes per contrast. t/Wilcoxon p-values do not account for selection of
43clusters from the same data or donor-level dependence. Logistic regression is a
44marker ranking without equivalent p-values or log fold changes. Scanpy log-fold
45changes are approximations computed from mean log expression, not a pseudobulk
46negative-binomial model. Do not rank scaled residuals or corrected expression as
47if it were log-normalized expression.
49The pseudobulk CLI requires the count layer, rejects missing grouping values,
50keeps stable opaque profile IDs to prevent underscore collisions, and checks
51that transferred condition/donor covariates are constant within each profile.
52It exports genes x profiles; transpose for PyDESeq2's samples x genes API.
53Preserve identifiers as strings when reading CSV and align rows by IDs. Fit each
54cell type separately with adequate independent biological replicates and a
55full-rank design; a sample x cell-type table does not multiply the donor count.
56Missing cell types are not zero-count biological replicates. Cell-count and
57library-size filtering decisions need study-specific justification.
59Scanpy 1.12 deprecates plotting `save=` but still supports it; toolkit suffixes
60retain distinct filenames. Global autosave alone can overwrite repeated plots.
61For exact exported figures use `return_fig=True` for embedding plots, then
62`Figure.savefig`; composite DotPlot/MatrixPlot/StackedViolin instead return plot
63objects with their own `.savefig()` when `return_fig=True`. `show=False` alone
64has function-specific return types, not a universal Figure contract. Dot size
65is detection fraction; dot color is mean expression; `standard_scale` rescales
66for display and changes the interpretation. Neighbor edges are not a PAGA
67partition graph, and cluster heatmaps are not automatically ordered by pseudotime.

Statistical and plotting interpretation

rank_genes_groups defaults to reference='rest'; named reference groups and groups control the comparison, not the donor dependence. BH adjustment is over tested genes per contrast. t/Wilcoxon p-values do not account for selection of clusters from the same data or donor-level dependence. Logistic regression is a marker ranking without equivalent p-values or log fold changes. Scanpy log-fold changes are approximations computed from mean log expression, not a pseudobulk negative-binomial model. Do not rank scaled residuals or corrected expression as if it were log-normalized expression.

The pseudobulk CLI requires the count layer, rejects missing grouping values, keeps stable opaque profile IDs to prevent underscore collisions, and checks that transferred condition/donor covariates are constant within each profile. It exports genes x profiles; transpose for PyDESeq2's samples x genes API. Preserve identifiers as strings when reading CSV and align rows by IDs. Fit each cell type separately with adequate independent biological replicates and a full-rank design; a sample x cell-type table does not multiply the donor count. Missing cell types are not zero-count biological replicates. Cell-count and library-size filtering decisions need study-specific justification.

Scanpy 1.12 deprecates plotting save= but still supports it; toolkit suffixes retain distinct filenames. Global autosave alone can overwrite repeated plots. For exact exported figures use return_fig=True for embedding plots, then Figure.savefig; composite DotPlot/MatrixPlot/StackedViolin instead return plot objects with their own .savefig() when return_fig=True. show=False alone has function-specific return types, not a universal Figure contract. Dot size is detection fraction; dot color is mean expression; standard_scale rescales for display and changes the interpretation. Neighbor edges are not a PAGA partition graph, and cluster heatmaps are not automatically ordered by pseudotime.

Raw lines 38 to 68
69## Optional branches and current upstream references
71- [Release notes](https://scanpy.readthedocs.io/en/stable/release-notes/index.html):
72 1.12.4 released 2026-08-27; Python >=3.12; deprecated Louvain/plot `save` remain.
73- [HVG](https://scanpy.readthedocs.io/en/stable/generated/scanpy.pp.highly_variable_genes.html),
74 [PCA](https://scanpy.readthedocs.io/en/stable/generated/scanpy.pp.pca.html),
75 [neighbors](https://scanpy.readthedocs.io/en/stable/generated/scanpy.pp.neighbors.html),
76 [Leiden](https://scanpy.readthedocs.io/en/stable/generated/scanpy.tl.leiden.html).
77- [Marker ranking](https://scanpy.readthedocs.io/en/stable/generated/scanpy.tl.rank_genes_groups.html),
78 [aggregation](https://scanpy.readthedocs.io/en/stable/generated/scanpy.get.aggregate.html),
79 [gene scoring](https://scanpy.readthedocs.io/en/stable/generated/scanpy.tl.score_genes.html).
80- [AnnData raw](https://anndata.readthedocs.io/en/stable/generated/anndata.AnnData.raw.html),
81 [backed I/O](https://anndata.readthedocs.io/en/stable/generated/anndata.io.read_h5ad.html),
82 [released source](https://github.com/scverse/anndata/tree/0.13.4).
83- [Harmony](https://scanpy.readthedocs.io/en/stable/generated/scanpy.external.pp.harmony_integrate.html)
84 changes `X_pca_harmony`, not X. Scanpy 1.12.4 still transposes `Z_corr`,
85 but [harmonypy 2.0.2](https://github.com/slowkow/harmonypy) returns cells x PCs.
86 The wrapper failed a native shape check; toolkit integration calls
87 `harmonypy.run_harmony` directly, validates shape/finiteness, and stores
88 `Z_corr` without transposition. Rebuild neighbors explicitly on it.
89 [BBKNN](https://scanpy.readthedocs.io/en/stable/generated/scanpy.external.pp.bbknn.html)
90 replaces the neighbor graph; running ordinary neighbors afterward discards it.
91 The CLI explicitly selects exact cKDTree (`approx=False, use_faiss=False`):
92 native Annoy 1.17.3 returned too few neighbors on the tested macOS ARM/NumPy
93 2.5 fixture. This is an observed backend limitation, not a universal BBKNN failure.
94 ComBat changes X, may densify, and needs >=2 cells per batch; retain uncorrected
95 log expression/raw counts for markers/DE. Integration cannot resolve perfectly
96 confounded batch and condition. Optional Harmony/BBKNN runtime validation is
97 separately reported; ordinary native tests do not establish integration quality.
98- [DPT](https://scanpy.readthedocs.io/en/stable/generated/scanpy.tl.dpt.html) requires
99 neighbors, diffusion components and a justified biological root. An arbitrary
100 cluster zero is only an illustrative placeholder; pseudotime is not elapsed time.
101- [ingest](https://scanpy.readthedocs.io/en/stable/generated/scanpy.tl.ingest.html)
102 requires the same ordered gene axis and compatible preprocessing in query and
103 reference, with reference neighbors/PCA/UMAP as requested. No atlas mapping was
104 executed in this refresh.
105- [R conversion](r_interop.md): official signatures were checked, but R system
106 installation/Seurat/zellkonverter/SeuratDisk conversions were not run. zellkonverter
107 manages its own supported Python environment; do not assume the local Scanpy
108 environment is used. An unspecified X_name selects the first assay, not necessarily
109 counts, so the example now requires a counts assay explicitly.
111Dask, alternative file formats, optional integrations and trajectories each need
112representative-data checks before production use. Small synthetic native tests
113validate software contracts; they establish no biological annotation, batch
114correction quality, calibrated DE error rate or cross-tool statistical equivalence.

Optional branches and current upstream references

  • Release notes: 1.12.4 released 2026-08-27; Python >=3.12; deprecated Louvain/plot save remain.
  • HVG, PCA, neighbors, Leiden.
  • Marker ranking, aggregation, gene scoring.
  • AnnData raw, backed I/O, released source.
  • Harmony changes X_pca_harmony, not X. Scanpy 1.12.4 still transposes Z_corr, but harmonypy 2.0.2 returns cells x PCs. The wrapper failed a native shape check; toolkit integration calls harmonypy.run_harmony directly, validates shape/finiteness, and stores Z_corr without transposition. Rebuild neighbors explicitly on it. BBKNN replaces the neighbor graph; running ordinary neighbors afterward discards it. The CLI explicitly selects exact cKDTree (approx=False, use_faiss=False): native Annoy 1.17.3 returned too few neighbors on the tested macOS ARM/NumPy 2.5 fixture. This is an observed backend limitation, not a universal BBKNN failure. ComBat changes X, may densify, and needs >=2 cells per batch; retain uncorrected log expression/raw counts for markers/DE. Integration cannot resolve perfectly confounded batch and condition. Optional Harmony/BBKNN runtime validation is separately reported; ordinary native tests do not establish integration quality.
  • DPT requires neighbors, diffusion components and a justified biological root. An arbitrary cluster zero is only an illustrative placeholder; pseudotime is not elapsed time.
  • ingest requires the same ordered gene axis and compatible preprocessing in query and reference, with reference neighbors/PCA/UMAP as requested. No atlas mapping was executed in this refresh.
  • R conversion: official signatures were checked, but R system installation/Seurat/zellkonverter/SeuratDisk conversions were not run. zellkonverter manages its own supported Python environment; do not assume the local Scanpy environment is used. An unspecified X_name selects the first assay, not necessarily counts, so the example now requires a counts assay explicitly.

Dask, alternative file formats, optional integrations and trajectories each need representative-data checks before production use. Small synthetic native tests validate software contracts; they establish no biological annotation, batch correction quality, calibrated DE error rate or cross-tool statistical equivalence.

Raw lines 69 to 114