Skill Tracer
Skills › Scientific Agent Skills › anndata › data_structure.md

data_structure.md

324 lines · skills/anndata/references/data_structure.md at 5f2bc03

Open on GitHub
1# AnnData Object Structure
2
3The AnnData object stores a data matrix with associated annotations, providing a flexible framework for managing experimental data and metadata.
4

AnnData Object Structure

The AnnData object stores a data matrix with associated annotations, providing a flexible framework for managing experimental data and metadata.

Raw lines 1 to 4
5## Core Components
6

Core Components

Raw lines 5 to 6
7### X (Data Matrix)
8The primary data matrix has shape `(n_obs, n_vars)` (observations by variables). It may be `None` for metadata-only objects. Record whether values are counts, normalized expression, residuals, or another measurement; names alone do not establish units.
9
10```python
11import anndata as ad
12import numpy as np
14# Create with dense array
15adata = ad.AnnData(X=np.random.rand(100, 2000))
17# Create with sparse matrix (recommended for large, sparse data)
18from scipy.sparse import csr_matrix
19sparse_data = csr_matrix(np.random.default_rng(42).binomial(1, 0.01, (100, 2000)))
20adata = ad.AnnData(X=sparse_data)
21```
23Access data:
24```python
25# Full matrix (caution with large datasets)
26full_data = adata.X
28# Single observation
29obs_data = adata.X[0, :]
31# Single variable across all observations
32var_data = adata.X[:, 0]
33```

X (Data Matrix)

The primary data matrix has shape (n_obs, n_vars) (observations by variables). It may be None for metadata-only objects. Record whether values are counts, normalized expression, residuals, or another measurement; names alone do not establish units.

import anndata as ad
import numpy as np

# Create with dense array
adata = ad.AnnData(X=np.random.rand(100, 2000))

# Create with sparse matrix (recommended for large, sparse data)
from scipy.sparse import csr_matrix
sparse_data = csr_matrix(np.random.default_rng(42).binomial(1, 0.01, (100, 2000)))
adata = ad.AnnData(X=sparse_data)

Access data:

# Full matrix (caution with large datasets)
full_data = adata.X

# Single observation
obs_data = adata.X[0, :]

# Single variable across all observations
var_data = adata.X[:, 0]
Raw lines 7 to 34
35### obs (Observation Annotations)
36DataFrame storing metadata about observations (rows). Each row corresponds to one observation in X.
38```python
39import pandas as pd
41# Create AnnData with observation metadata
42obs_df = pd.DataFrame({
43 'cell_type': ['T cell', 'B cell', 'Monocyte'],
44 'treatment': ['control', 'treated', 'control'],
45 'timepoint': [0, 24, 24]
46}, index=['cell_1', 'cell_2', 'cell_3'])
48adata = ad.AnnData(X=np.random.rand(3, 100), obs=obs_df)
50# Access observation metadata
51print(adata.obs['cell_type'])
52print(adata.obs.loc['cell_1'])
53```

obs (Observation Annotations)

DataFrame storing metadata about observations (rows). Each row corresponds to one observation in X.

import pandas as pd

# Create AnnData with observation metadata
obs_df = pd.DataFrame({
    'cell_type': ['T cell', 'B cell', 'Monocyte'],
    'treatment': ['control', 'treated', 'control'],
    'timepoint': [0, 24, 24]
}, index=['cell_1', 'cell_2', 'cell_3'])

adata = ad.AnnData(X=np.random.rand(3, 100), obs=obs_df)

# Access observation metadata
print(adata.obs['cell_type'])
print(adata.obs.loc['cell_1'])
Raw lines 35 to 54
55### var (Variable Annotations)
56DataFrame storing metadata about variables (columns). Each row corresponds to one variable in X.
58```python
59# Create AnnData with variable metadata
60var_df = pd.DataFrame({
61 'gene_name': ['ACTB', 'GAPDH', 'TP53'],
62 'chromosome': ['7', '12', '17'],
63 'highly_variable': [True, False, True]
64}, index=['ENSG00001', 'ENSG00002', 'ENSG00003'])
66adata = ad.AnnData(X=np.random.rand(100, 3), var=var_df)
68# Access variable metadata
69print(adata.var['gene_name'])
70print(adata.var.loc['ENSG00001'])
71```

var (Variable Annotations)

DataFrame storing metadata about variables (columns). Each row corresponds to one variable in X.

# Create AnnData with variable metadata
var_df = pd.DataFrame({
    'gene_name': ['ACTB', 'GAPDH', 'TP53'],
    'chromosome': ['7', '12', '17'],
    'highly_variable': [True, False, True]
}, index=['ENSG00001', 'ENSG00002', 'ENSG00003'])

adata = ad.AnnData(X=np.random.rand(100, 3), var=var_df)

# Access variable metadata
print(adata.var['gene_name'])
print(adata.var.loc['ENSG00001'])
Raw lines 55 to 72
73### layers (Alternative Data Representations)
74Mapping of same-shape matrices. In **AnnData 0.13.4**, `layers[None]` aliases `X` and iteration includes this special key. Named layers remain strings; handle `None` explicitly in code that exports or enumerates layers. On disk, native formats still store X under the top-level `X` key.
76```python
77# Derive each representation from the SAME synthetic count matrix.
78counts = np.random.default_rng(42).poisson(1, (100, 2000)).astype('float32')
79adata = ad.AnnData(X=counts)
80adata.layers['raw_counts'] = adata.X.copy()
81totals = adata.X.sum(axis=1, keepdims=True)
82adata.layers['normalized'] = adata.X / np.where(totals == 0, 1, totals)
83std = adata.X.std(axis=0)
84adata.layers['scaled'] = (adata.X - adata.X.mean(axis=0)) / np.where(std == 0, 1, std)
86# Access layers
87raw_data = adata.layers['raw_counts']
88normalized_data = adata.layers['normalized']
89named_layers = {key: value for key, value in adata.layers.items() if key is not None}
90assert adata.layers[None] is adata.X
91```
93Common layer uses:
94- `raw_counts`: Original count data before normalization
95- `normalized`: Log-normalized or TPM values
96- `scaled`: Z-scored values for analysis
97- `imputed`: Data after imputation

layers (Alternative Data Representations)

Mapping of same-shape matrices. In AnnData 0.13.4, layers[None] aliases X and iteration includes this special key. Named layers remain strings; handle None explicitly in code that exports or enumerates layers. On disk, native formats still store X under the top-level X key.

# Derive each representation from the SAME synthetic count matrix.
counts = np.random.default_rng(42).poisson(1, (100, 2000)).astype('float32')
adata = ad.AnnData(X=counts)
adata.layers['raw_counts'] = adata.X.copy()
totals = adata.X.sum(axis=1, keepdims=True)
adata.layers['normalized'] = adata.X / np.where(totals == 0, 1, totals)
std = adata.X.std(axis=0)
adata.layers['scaled'] = (adata.X - adata.X.mean(axis=0)) / np.where(std == 0, 1, std)

# Access layers
raw_data = adata.layers['raw_counts']
normalized_data = adata.layers['normalized']
named_layers = {key: value for key, value in adata.layers.items() if key is not None}
assert adata.layers[None] is adata.X

Common layer uses:

  • raw_counts: Original count data before normalization
  • normalized: Log-normalized or TPM values
  • scaled: Z-scored values for analysis
  • imputed: Data after imputation
Raw lines 73 to 98
99### obsm (Multi-dimensional Observation Annotations)
100Dictionary storing multi-dimensional arrays aligned to observations.
102```python
103# Store PCA coordinates and UMAP embeddings
104adata.obsm['X_pca'] = np.random.rand(100, 50) # 50 principal components
105adata.obsm['X_umap'] = np.random.rand(100, 2) # 2D UMAP coordinates
106adata.obsm['X_tsne'] = np.random.rand(100, 2) # 2D t-SNE coordinates
108# Access embeddings
109pca_coords = adata.obsm['X_pca']
110umap_coords = adata.obsm['X_umap']
111```
113Common obsm uses:
114- `X_pca`: Principal component coordinates
115- `X_umap`: UMAP embedding coordinates
116- `X_tsne`: t-SNE embedding coordinates
117- `X_diffmap`: Diffusion map coordinates
118- `protein_expression`: Protein abundance measurements (CITE-seq)

obsm (Multi-dimensional Observation Annotations)

Dictionary storing multi-dimensional arrays aligned to observations.

# Store PCA coordinates and UMAP embeddings
adata.obsm['X_pca'] = np.random.rand(100, 50)  # 50 principal components
adata.obsm['X_umap'] = np.random.rand(100, 2)  # 2D UMAP coordinates
adata.obsm['X_tsne'] = np.random.rand(100, 2)  # 2D t-SNE coordinates

# Access embeddings
pca_coords = adata.obsm['X_pca']
umap_coords = adata.obsm['X_umap']

Common obsm uses:

  • X_pca: Principal component coordinates
  • X_umap: UMAP embedding coordinates
  • X_tsne: t-SNE embedding coordinates
  • X_diffmap: Diffusion map coordinates
  • protein_expression: Protein abundance measurements (CITE-seq)
Raw lines 99 to 119
120### varm (Multi-dimensional Variable Annotations)
121Dictionary storing multi-dimensional arrays aligned to variables.
123```python
124# Store PCA loadings
125adata.varm['PCs'] = np.random.rand(2000, 50) # Loadings for 50 components
126adata.varm['gene_modules'] = np.random.rand(2000, 10) # Gene module scores
128# Access loadings
129pc_loadings = adata.varm['PCs']
130```
132Common varm uses:
133- `PCs`: Principal component loadings
134- `gene_modules`: Gene co-expression module assignments

varm (Multi-dimensional Variable Annotations)

Dictionary storing multi-dimensional arrays aligned to variables.

# Store PCA loadings
adata.varm['PCs'] = np.random.rand(2000, 50)  # Loadings for 50 components
adata.varm['gene_modules'] = np.random.rand(2000, 10)  # Gene module scores

# Access loadings
pc_loadings = adata.varm['PCs']

Common varm uses:

  • PCs: Principal component loadings
  • gene_modules: Gene co-expression module assignments
Raw lines 120 to 135
136### obsp (Pairwise Observation Relationships)
137Mapping of dense or sparse arrays whose first two dimensions are `(n_obs, n_obs)`.
139```python
140from scipy.sparse import csr_matrix
142# Store k-nearest neighbor graph
143n_obs = 100
144knn_graph = csr_matrix(np.random.rand(n_obs, n_obs) > 0.95)
145adata.obsp['connectivities'] = knn_graph
146adata.obsp['distances'] = csr_matrix(np.random.rand(n_obs, n_obs))
148# Access graphs
149knn_connections = adata.obsp['connectivities']
150distances = adata.obsp['distances']
151```
153Common obsp uses:
154- `connectivities`: Cell-cell neighborhood graph
155- `distances`: Pairwise distances between cells

obsp (Pairwise Observation Relationships)

Mapping of dense or sparse arrays whose first two dimensions are (n_obs, n_obs).

from scipy.sparse import csr_matrix

# Store k-nearest neighbor graph
n_obs = 100
knn_graph = csr_matrix(np.random.rand(n_obs, n_obs) > 0.95)
adata.obsp['connectivities'] = knn_graph
adata.obsp['distances'] = csr_matrix(np.random.rand(n_obs, n_obs))

# Access graphs
knn_connections = adata.obsp['connectivities']
distances = adata.obsp['distances']

Common obsp uses:

  • connectivities: Cell-cell neighborhood graph
  • distances: Pairwise distances between cells
Raw lines 136 to 156
157### varp (Pairwise Variable Relationships)
158Mapping of dense or sparse arrays whose first two dimensions are `(n_vars, n_vars)`.
160```python
161# Store gene-gene correlation matrix
162n_vars = 2000
163gene_corr = csr_matrix(np.random.rand(n_vars, n_vars) > 0.99)
164adata.varp['correlations'] = gene_corr
166# Access correlations
167gene_correlations = adata.varp['correlations']
168```

varp (Pairwise Variable Relationships)

Mapping of dense or sparse arrays whose first two dimensions are (n_vars, n_vars).

# Store gene-gene correlation matrix
n_vars = 2000
gene_corr = csr_matrix(np.random.rand(n_vars, n_vars) > 0.99)
adata.varp['correlations'] = gene_corr

# Access correlations
gene_correlations = adata.varp['correlations']
Raw lines 157 to 169
170### uns (Unstructured Annotations)
171Mapping for unstructured metadata. In-memory Python objects are not all serializable; native output requires supported AnnData encodings. Avoid slashes in keys and validate by reopening the saved file.
173```python
174# Store analysis parameters and results
175adata.uns['experiment_date'] = '2025-11-03'
176adata.uns['pca'] = {
177 'variance_ratio': [0.15, 0.10, 0.08],
178 'params': {'n_comps': 50}
180adata.uns['neighbors'] = {
181 'params': {'n_neighbors': 15, 'method': 'umap'},
182 'connectivities_key': 'connectivities'
185# Access unstructured data
186exp_date = adata.uns['experiment_date']
187pca_params = adata.uns['pca']['params']
188```
190Common uns uses:
191- Analysis parameters and settings
192- Color palettes for plotting
193- Cluster information
194- Tool-specific metadata

uns (Unstructured Annotations)

Mapping for unstructured metadata. In-memory Python objects are not all serializable; native output requires supported AnnData encodings. Avoid slashes in keys and validate by reopening the saved file.

# Store analysis parameters and results
adata.uns['experiment_date'] = '2025-11-03'
adata.uns['pca'] = {
    'variance_ratio': [0.15, 0.10, 0.08],
    'params': {'n_comps': 50}
}
adata.uns['neighbors'] = {
    'params': {'n_neighbors': 15, 'method': 'umap'},
    'connectivities_key': 'connectivities'
}

# Access unstructured data
exp_date = adata.uns['experiment_date']
pca_params = adata.uns['pca']['params']

Common uns uses:

  • Analysis parameters and settings
  • Color palettes for plotting
  • Cluster information
  • Tool-specific metadata
Raw lines 170 to 195
196### raw (Original Data Snapshot)
197Optional snapshot of the current `X`, `var`, and `varm`; it is not a full backup and is not automatically raw counts. Observation slicing subsets `.raw`, while variable slicing preserves its original variable axis. Layers follow both axes, so a counts layer loses genes when genes are filtered. Record the snapshot transformation in `uns` and keep a separate full-gene counts file when required.
199```python
200# Create AnnData and store raw state
201adata = ad.AnnData(X=np.random.rand(100, 5000))
202adata.var['gene_name'] = [f'Gene_{i}' for i in range(5000)]
204# Store raw state before filtering
205adata.raw = adata.copy()
206adata.uns['raw_semantics'] = 'synthetic unnormalized values'
208# Filter to highly variable genes
209highly_variable_mask = np.random.rand(5000) > 0.5
210adata = adata[:, highly_variable_mask]
212# Access original data
213original_matrix = adata.raw.X
214original_var = adata.raw.var
215```

raw (Original Data Snapshot)

Optional snapshot of the current X, var, and varm; it is not a full backup and is not automatically raw counts. Observation slicing subsets .raw, while variable slicing preserves its original variable axis. Layers follow both axes, so a counts layer loses genes when genes are filtered. Record the snapshot transformation in uns and keep a separate full-gene counts file when required.

# Create AnnData and store raw state
adata = ad.AnnData(X=np.random.rand(100, 5000))
adata.var['gene_name'] = [f'Gene_{i}' for i in range(5000)]

# Store raw state before filtering
adata.raw = adata.copy()
adata.uns['raw_semantics'] = 'synthetic unnormalized values'

# Filter to highly variable genes
highly_variable_mask = np.random.rand(5000) > 0.5
adata = adata[:, highly_variable_mask]

# Access original data
original_matrix = adata.raw.X
original_var = adata.raw.var
Raw lines 196 to 216
217## Object Properties
219```python
220# Dimensions
221n_observations = adata.n_obs
222n_variables = adata.n_vars
223shape = adata.shape # (n_obs, n_vars)
225# Index information
226obs_names = adata.obs_names # Observation identifiers
227var_names = adata.var_names # Variable identifiers
229# Storage mode
230is_view = adata.is_view # True if this is a view of another object
231is_backed = adata.isbacked # True if backed by on-disk storage
232filename = adata.filename # Path to backing file (if backed)
233```

Object Properties

# Dimensions
n_observations = adata.n_obs
n_variables = adata.n_vars
shape = adata.shape  # (n_obs, n_vars)

# Index information
obs_names = adata.obs_names  # Observation identifiers
var_names = adata.var_names  # Variable identifiers

# Storage mode
is_view = adata.is_view  # True if this is a view of another object
is_backed = adata.isbacked  # True if backed by on-disk storage
filename = adata.filename  # Path to backing file (if backed)
Raw lines 217 to 234
235## Creating AnnData Objects

Creating AnnData Objects

Raw lines 235 to 236
237### From arrays and DataFrames
238```python
239import anndata as ad
240import numpy as np
241import pandas as pd
243# Minimal creation
244X = np.random.rand(100, 2000)
245adata = ad.AnnData(X)
247# With metadata
248obs = pd.DataFrame({'cell_type': ['A', 'B'] * 50}, index=[f'cell_{i}' for i in range(100)])
249var = pd.DataFrame({'gene_name': [f'Gene_{i}' for i in range(2000)]}, index=[f'ENSG{i:05d}' for i in range(2000)])
250adata = ad.AnnData(X=X, obs=obs, var=var)
252# With all components
253adata = ad.AnnData(
254 X=X,
255 obs=obs,
256 var=var,
257 layers={'raw': np.random.randint(0, 100, (100, 2000))},
258 obsm={'X_pca': np.random.rand(100, 50)},
259 uns={'experiment': 'test'}
261```

From arrays and DataFrames

import anndata as ad
import numpy as np
import pandas as pd

# Minimal creation
X = np.random.rand(100, 2000)
adata = ad.AnnData(X)

# With metadata
obs = pd.DataFrame({'cell_type': ['A', 'B'] * 50}, index=[f'cell_{i}' for i in range(100)])
var = pd.DataFrame({'gene_name': [f'Gene_{i}' for i in range(2000)]}, index=[f'ENSG{i:05d}' for i in range(2000)])
adata = ad.AnnData(X=X, obs=obs, var=var)

# With all components
adata = ad.AnnData(
    X=X,
    obs=obs,
    var=var,
    layers={'raw': np.random.randint(0, 100, (100, 2000))},
    obsm={'X_pca': np.random.rand(100, 50)},
    uns={'experiment': 'test'}
)
Raw lines 237 to 262
263### From DataFrame
264```python
265# Create from pandas DataFrame (genes as columns, cells as rows)
266df = pd.DataFrame(
267 np.random.rand(100, 50),
268 columns=[f'Gene_{i}' for i in range(50)],
269 index=[f'Cell_{i}' for i in range(100)]
271adata = ad.AnnData(df)
272```

From DataFrame

# Create from pandas DataFrame (genes as columns, cells as rows)
df = pd.DataFrame(
    np.random.rand(100, 50),
    columns=[f'Gene_{i}' for i in range(50)],
    index=[f'Cell_{i}' for i in range(100)]
)
adata = ad.AnnData(df)
Raw lines 263 to 273
274## Data Access Patterns

Data Access Patterns

Raw lines 274 to 275
276### Vector extraction
277```python
278# Get observation annotation as array
279cell_types = adata.obs_vector('cell_type')
281# Get variable values across observations
282gene_expression = adata.obs_vector('ACTB') # If ACTB is in var_names
284# Get variable annotation as array
285gene_names = adata.var_vector('gene_name')
286```

Vector extraction

# Get observation annotation as array
cell_types = adata.obs_vector('cell_type')

# Get variable values across observations
gene_expression = adata.obs_vector('ACTB')  # If ACTB is in var_names

# Get variable annotation as array
gene_names = adata.var_vector('gene_name')
Raw lines 276 to 287
288### Subsetting
289```python
290# By index
291subset = adata[0:10, 0:100] # First 10 obs, first 100 vars
293# By name
294subset = adata[['cell_1', 'cell_2'], ['ACTB', 'GAPDH']]
296# By boolean mask
297high_count_cells = adata.obs['total_counts'] > 1000
298subset = adata[high_count_cells, :]
300# By observation metadata
301t_cells = adata[adata.obs['cell_type'] == 'T cell']
302```

Subsetting

# By index
subset = adata[0:10, 0:100]  # First 10 obs, first 100 vars

# By name
subset = adata[['cell_1', 'cell_2'], ['ACTB', 'GAPDH']]

# By boolean mask
high_count_cells = adata.obs['total_counts'] > 1000
subset = adata[high_count_cells, :]

# By observation metadata
t_cells = adata[adata.obs['cell_type'] == 'T cell']
Raw lines 288 to 303
304## Memory Considerations
306The AnnData structure is designed for memory efficiency:
307- Sparse matrices reduce memory for sparse data
308- Views avoid copying data when possible
309- H5AD backed mode leaves X on disk; metadata and named layers can still consume memory
310- Categorical annotations reduce memory for discrete values
312```python
313# Convert strings to categoricals (more memory efficient)
314adata.obs['cell_type'] = adata.obs['cell_type'].astype('category')
315adata.strings_to_categoricals()
317# Check if object is a view (doesn't own data)
318if adata.is_view:
319 adata = adata.copy() # Create independent copy
320```

Memory Considerations

The AnnData structure is designed for memory efficiency:

  • Sparse matrices reduce memory for sparse data
  • Views avoid copying data when possible
  • H5AD backed mode leaves X on disk; metadata and named layers can still consume memory
  • Categorical annotations reduce memory for discrete values
# Convert strings to categoricals (more memory efficient)
adata.obs['cell_type'] = adata.obs['cell_type'].astype('category')
adata.strings_to_categoricals()

# Check if object is a view (doesn't own data)
if adata.is_view:
    adata = adata.copy()  # Create independent copy
Raw lines 304 to 321
322## Verified upstream contract
324Reviewed against AnnData 0.13.4 source and the official [object API](https://anndata.readthedocs.io/en/stable/generated/anndata.AnnData.html), [raw semantics](https://anndata.readthedocs.io/en/stable/generated/anndata.AnnData.raw.html), [on-disk encodings](https://anndata.readthedocs.io/en/stable/fileformat-prose.html), and [0.13 release changes](https://anndata.readthedocs.io/en/stable/release-notes/). Examples with invented metadata and random arrays illustrate structure, not validated biological embeddings or graphs.

Verified upstream contract

Reviewed against AnnData 0.13.4 source and the official object API, raw semantics, on-disk encodings, and 0.13 release changes. Examples with invented metadata and random arrays illustrate structure, not validated biological embeddings or graphs.

Raw lines 322 to 324