Skill Tracer
Skills › Scientific Agent Skills › lamindb › core-concepts.md

core-concepts.md

129 lines · skills/lamindb/references/core-concepts.md at 5f2bc03

Open on GitHub

Most marks come from matching words in the file, so treat them as pointers.

1# Core concepts and lineage
2
3Targets LaminDB 2.10.0. These examples assume an initialized instance and imports
4`import lamindb as ln` and `import pandas as pd`. User file paths and pre-existing
5keys below are illustrative; the equivalent local workflows have regression tests.
6

Core concepts and lineage

Targets LaminDB 2.10.0. These examples assume an initialized instance and imports import lamindb as ln and import pandas as pd. User file paths and pre-existing keys below are illustrative; the equivalent local workflows have regression tests.

Raw lines 1 to 6
7## Register and retrieve content
8
9```python
10artifact = ln.Artifact("sample.fasta", key="reference/sample.fasta").save()
11frame = ln.Artifact.from_dataframe(
12 pd.DataFrame({"count": [1, 2]}), key="experiments/counts.parquet"
13).save()
14# For a user-provided AnnData object:
15# annotated = ln.Artifact.from_anndata(adata, key="scrna/batch.h5ad").save()
16loaded = frame.load()
17local_path = frame.cache()
18```
20`Artifact` records carry `uid`, `key`, `suffix`, `size`, `hash`, `created_by`,
21`run`, and schema/provenance links. `describe()` displays metadata;
22`view_lineage()` renders lineage and needs Graphviz's system executable in
23addition to its Python package. Check whether a run exists before accessing it.

Register and retrieve content

artifact = ln.Artifact("sample.fasta", key="reference/sample.fasta").save()
frame = ln.Artifact.from_dataframe(
    pd.DataFrame({"count": [1, 2]}), key="experiments/counts.parquet"
).save()
# For a user-provided AnnData object:
# annotated = ln.Artifact.from_anndata(adata, key="scrna/batch.h5ad").save()
loaded = frame.load()
local_path = frame.cache()

Artifact records carry uid, key, suffix, size, hash, created_by, run, and schema/provenance links. describe() displays metadata; view_lineage() renders lineage and needs Graphviz's system executable in addition to its Python package. Check whether a run exists before accessing it.

Raw lines 7 to 24
25## Revisions
27```python
28changed = loaded.assign(count=[3, 4])
29new = ln.Artifact.from_dataframe(
30 changed,
31 key="experiments/counts.parquet",
32 revises=frame,
33 version_tag="2",
34).save()
35latest = ln.Artifact.get(key="experiments/counts.parquet")
36versions = new.versions.to_dataframe(limit=None)
37exact_revision = ln.Artifact.get(frame.uid)
38```
40A key is a logical name, whereas a full UID identifies a revision. `get(key=...)`
41preferentially selects a latest version; `.filter(key=...)` can include several
42revisions. Identical content can be deduplicated; re-saving is not a guarantee
43of a new revision. Use `version_tag`, not the retired `version` field, for labels.
44Overwrite-enabled storage formats have additional old-content access restrictions;
45review them before assuming all historical bytes remain accessible.

Revisions

changed = loaded.assign(count=[3, 4])
new = ln.Artifact.from_dataframe(
    changed,
    key="experiments/counts.parquet",
    revises=frame,
    version_tag="2",
).save()
latest = ln.Artifact.get(key="experiments/counts.parquet")
versions = new.versions.to_dataframe(limit=None)
exact_revision = ln.Artifact.get(frame.uid)

A key is a logical name, whereas a full UID identifies a revision. get(key=...) preferentially selects a latest version; .filter(key=...) can include several revisions. Identical content can be deduplicated; re-saving is not a guarantee of a new revision. Use version_tag, not the retired version field, for labels. Overwrite-enabled storage formats have additional old-content access restrictions; review them before assuming all historical bytes remain accessible.

Raw lines 25 to 46
47## Features and experimental entities
49```python
50from datetime import date
52score = ln.Feature(name="gc_content", dtype=float).save()
53observed = ln.Feature(name="experiment_date", dtype=date, coerce=True).save()
54frame.features.set_values({score: 0.55, observed: "2026-09-30"})
55ln.Artifact.filter(gc_content__gte=0.5).to_dataframe(include="features")
57sample_type = ln.Record(name="Sample", is_type=True).save()
58sample = ln.Record(name="control-1", type=sample_type).save()
59replicate = ln.Record(name="control-1-replicate", type=sample_type).save()
60replicate.parents.add(sample)
61frame.records.add(sample)
62label = ln.ULabel(name="reviewed").save()
63frame.ulabels.add(label)
64```
66`set_values()` adds typed values, not merely a feature-name link. A string dtype
67is free text; a categorical dtype backed by `Record`, `ULabel`, or Bionty enables
68controlled terms. Date strings need deliberate coercion; preserve units and
69original values when conversions matter scientifically.

Features and experimental entities

from datetime import date

score = ln.Feature(name="gc_content", dtype=float).save()
observed = ln.Feature(name="experiment_date", dtype=date, coerce=True).save()
frame.features.set_values({score: 0.55, observed: "2026-09-30"})
ln.Artifact.filter(gc_content__gte=0.5).to_dataframe(include="features")

sample_type = ln.Record(name="Sample", is_type=True).save()
sample = ln.Record(name="control-1", type=sample_type).save()
replicate = ln.Record(name="control-1-replicate", type=sample_type).save()
replicate.parents.add(sample)
frame.records.add(sample)
label = ln.ULabel(name="reviewed").save()
frame.ulabels.add(label)

set_values() adds typed values, not merely a feature-name link. A string dtype is free text; a categorical dtype backed by Record, ULabel, or Bionty enables controlled terms. Date strings need deliberate coercion; preserve units and original values when conversions matter scientifically.

Raw lines 47 to 70
71## Script and notebook tracking
73Run this pattern from a real script/notebook, with the input already registered:
75```python
76ln.Project(name="QC study").save()
77ln.track(project="QC study", params={"minimum_count": 2})
78inp = ln.Artifact.get(key="experiments/counts.parquet")
79data = inp.load()
80out = ln.Artifact.from_dataframe(
81 data[data["count"] >= 2], key="experiments/filtered.parquet"
82).save()
83ln.finish()
85run = out.run
86assert inp in run.input_artifacts.all()
87run.output_artifacts.to_dataframe()
88run.transform.describe()
89ln.Run.filter(params__minimum_count=2).to_dataframe()
90ln.Run.filter(projects__name="QC study").to_dataframe()
91ln.Artifact.filter(transform=run.transform).to_dataframe()
92```
94Tracking stores code, environment, parameters, and tracked input/output links.
95Unregistered reads outside LaminDB need explicit registration/tracking. Do not
96mark failed computation as successfully finished. The `params` JSON is distinct
97from validated run `features`; arbitrary parameter dictionaries do not create
98Feature records automatically.

Script and notebook tracking

Run this pattern from a real script/notebook, with the input already registered:

ln.Project(name="QC study").save()
ln.track(project="QC study", params={"minimum_count": 2})
inp = ln.Artifact.get(key="experiments/counts.parquet")
data = inp.load()
out = ln.Artifact.from_dataframe(
    data[data["count"] >= 2], key="experiments/filtered.parquet"
).save()
ln.finish()

run = out.run
assert inp in run.input_artifacts.all()
run.output_artifacts.to_dataframe()
run.transform.describe()
ln.Run.filter(params__minimum_count=2).to_dataframe()
ln.Run.filter(projects__name="QC study").to_dataframe()
ln.Artifact.filter(transform=run.transform).to_dataframe()

Tracking stores code, environment, parameters, and tracked input/output links. Unregistered reads outside LaminDB need explicit registration/tracking. Do not mark failed computation as successfully finished. The params JSON is distinct from validated run features; arbitrary parameter dictionaries do not create Feature records automatically.

Raw lines 71 to 99
Runs code script/notebook :73
100## Function tracking
102```python
103@ln.step()
104def filter_counts(input_key: str, output_key: str, threshold: int = 2):
105 data = ln.Artifact.get(key=input_key).load()
106 return ln.Artifact.from_dataframe(
107 data[data["count"] >= threshold], key=output_key
108 ).save()
110@ln.flow()
111def analysis():
112 return filter_counts("experiments/counts.parquet", "experiments/step.parquet")
113```
115Invoke `analysis()` from a saved Python script. Decorator parameters are tracked;
116function source discovery and code/environment capture depend on execution context.
117Do not claim a decorator tracks every external tool or scheduler transition.

Function tracking

@ln.step()
def filter_counts(input_key: str, output_key: str, threshold: int = 2):
    data = ln.Artifact.get(key=input_key).load()
    return ln.Artifact.from_dataframe(
        data[data["count"] >= threshold], key=output_key
    ).save()

@ln.flow()
def analysis():
    return filter_counts("experiments/counts.parquet", "experiments/step.parquet")

Invoke analysis() from a saved Python script. Decorator parameters are tracked; function source discovery and code/environment capture depend on execution context. Do not claim a decorator tracks every external tool or scheduler transition.

Raw lines 100 to 118
119## Projects, branches, and spaces
121Projects label data/runs; branch and space settings control which records are
122created or visible in the current context. Inspect `lamin info` before writes.
123The current CLI supports `lamin create project`, `lamin switch -c <branch>`, and
124`lamin switch --space <space>`. Branch merging moves metadata records and should
125be reviewed in the target instance; it is not a file-system merge or backup.
127Sources: [tracking](https://docs.lamin.ai/track),
128[change management](https://docs.lamin.ai/manage-changes),
129[2.10.0 artifact implementation](https://github.com/laminlabs/lamindb/blob/2.10.0/lamindb/models/artifact.py).

Projects, branches, and spaces

Projects label data/runs; branch and space settings control which records are created or visible in the current context. Inspect lamin info before writes. The current CLI supports lamin create project, lamin switch -c <branch>, and lamin switch --space <space>. Branch merging moves metadata records and should be reviewed in the target instance; it is not a file-system merge or backup.

Sources: tracking, change management, 2.10.0 artifact implementation.

Raw lines 119 to 129