Skill Tracer
Skills › Scientific Agent Skills › omero-integration › metadata.md

metadata.md

317 lines · skills/omero-integration/references/metadata.md at 5f2bc03

Open on GitHub

Most marks come from matching words in the file, so treat them as pointers.

1# Metadata and Annotations
2
3Annotations may contain participant identifiers, sample names, unpublished
4results, free text, remote filenames, or attached files. Read and export the
5minimum fields needed.
6

Metadata and Annotations

Annotations may contain participant identifiers, sample names, unpublished results, free text, remote filenames, or attached files. Read and export the minimum fields needed.

Raw lines 1 to 6
7## Current Annotation Types
8
9The OMERO structured annotation model includes:
11- `TagAnnotation`
12- `MapAnnotation`
13- `FileAnnotation`
14- `CommentAnnotation`
15- `BooleanAnnotation`
16- `LongAnnotation`
17- `DoubleAnnotation`
18- `TimestampAnnotation`
19- `TermAnnotation`
20- `XmlAnnotation`
21- annotation hierarchies through annotation-to-annotation links
23Current `omero.gateway` exports corresponding wrappers, including
24`TagAnnotationWrapper`, `MapAnnotationWrapper`,
25`FileAnnotationWrapper`, and `CommentAnnotationWrapper`. Do not import a
26historical `BaseAnnotationWrapper`; the current public wrapper is
27`AnnotationWrapper`.
29Annotations can be linked to multiple objects. Their ownership and the
30ownership of each link may differ. Deleting an annotation is not the same as
31deleting one link.

Current Annotation Types

The OMERO structured annotation model includes:

  • TagAnnotation
  • MapAnnotation
  • FileAnnotation
  • CommentAnnotation
  • BooleanAnnotation
  • LongAnnotation
  • DoubleAnnotation
  • TimestampAnnotation
  • TermAnnotation
  • XmlAnnotation
  • annotation hierarchies through annotation-to-annotation links

Current omero.gateway exports corresponding wrappers, including TagAnnotationWrapper, MapAnnotationWrapper, FileAnnotationWrapper, and CommentAnnotationWrapper. Do not import a historical BaseAnnotationWrapper; the current public wrapper is AnnotationWrapper.

Annotations can be linked to multiple objects. Their ownership and the ownership of each link may differ. Deleting an annotation is not the same as deleting one link.

Raw lines 7 to 32
33## Bounded Read
35`listAnnotations()` supports a namespace filter but not a page-size argument.
36Cap client iteration and report truncation:
38```python
39from itertools import islice
41image = conn.getObject("Image", image_id)
42if image is None:
43 raise LookupError("Image unavailable")
45max_annotations = 100
46items = list(islice(image.listAnnotations(), max_annotations + 1))
47truncated = len(items) > max_annotations
49for annotation in items[:max_annotations]:
50 print(annotation.getId(), type(annotation).__name__, annotation.getNs())
52print({"truncated": truncated})
53```
55Avoid `getValue()` when values are outside the export scope. However,
56`listAnnotations()` loads linked annotations, including their values, before
57yielding, and its namespace filter is applied locally. Thus redaction and
58iteration caps protect output; they do not prevent transfer into client memory
59or bound the server query. Annotation wrappers inherit `OMERO_CLASS=None`;
60use their concrete wrapper class for an annotation type name.
62For explicit parent IDs, annotation links can be queried:
64```python
65image_ids = [101, 102]
66for link in islice(
67 conn.getAnnotationLinks("Image", parent_ids=image_ids),
68 200,
69):
70 print(link.getParent().getId(), link.getChild().getId())
71```
73Keep both the parent-ID list and returned-link count bounded.

Bounded Read

listAnnotations() supports a namespace filter but not a page-size argument. Cap client iteration and report truncation:

from itertools import islice

image = conn.getObject("Image", image_id)
if image is None:
    raise LookupError("Image unavailable")

max_annotations = 100
items = list(islice(image.listAnnotations(), max_annotations + 1))
truncated = len(items) > max_annotations

for annotation in items[:max_annotations]:
    print(annotation.getId(), type(annotation).__name__, annotation.getNs())

print({"truncated": truncated})

Avoid getValue() when values are outside the export scope. However, listAnnotations() loads linked annotations, including their values, before yielding, and its namespace filter is applied locally. Thus redaction and iteration caps protect output; they do not prevent transfer into client memory or bound the server query. Annotation wrappers inherit OMERO_CLASS=None; use their concrete wrapper class for an annotation type name.

For explicit parent IDs, annotation links can be queried:

image_ids = [101, 102]
for link in islice(
    conn.getAnnotationLinks("Image", parent_ids=image_ids),
    200,
):
    print(link.getParent().getId(), link.getChild().getId())

Keep both the parent-ID list and returned-link count bounded.

Raw lines 33 to 74
75## Redacted Inventory
77A safe default record contains identifiers and type, not values:
79```python
80def annotation_summary(annotation):
81 details = annotation.getDetails()
82 owner = details.getOwner() if details is not None else None
83 return {
84 "id": annotation.getId(),
85 "type": type(annotation).__name__.removesuffix("Wrapper"),
86 "namespace": annotation.getNs(),
87 "owner_id": owner.getId() if owner is not None else None,
88 "value_redacted": True,
89 }
90```
92Owner names, annotation values, file names, descriptions, and link-owner names
93require separate inclusion decisions.
95The bundled exporter defaults to redaction:
97```bash
98python -B scripts/export_image_metadata.py \
99 --image-id 101 \
100 --max-annotations-per-image 100 \
101 --max-rois-per-image 100 \
102 --output ./image-101-metadata.json
104# Review, then connect. Add inclusion flags only when approved.
105python -B scripts/export_image_metadata.py \
106 --image-id 101 \
107 --max-annotations-per-image 100 \
108 --max-rois-per-image 100 \
109 --execute \
110 --output ./image-101-metadata.json
111```
113It never downloads FileAnnotation bytes or pixel data.

Redacted Inventory

A safe default record contains identifiers and type, not values:

def annotation_summary(annotation):
    details = annotation.getDetails()
    owner = details.getOwner() if details is not None else None
    return {
        "id": annotation.getId(),
        "type": type(annotation).__name__.removesuffix("Wrapper"),
        "namespace": annotation.getNs(),
        "owner_id": owner.getId() if owner is not None else None,
        "value_redacted": True,
    }

Owner names, annotation values, file names, descriptions, and link-owner names require separate inclusion decisions.

The bundled exporter defaults to redaction:

python -B scripts/export_image_metadata.py \
  --image-id 101 \
  --max-annotations-per-image 100 \
  --max-rois-per-image 100 \
  --output ./image-101-metadata.json

# Review, then connect. Add inclusion flags only when approved.
python -B scripts/export_image_metadata.py \
  --image-id 101 \
  --max-annotations-per-image 100 \
  --max-rois-per-image 100 \
  --execute \
  --output ./image-101-metadata.json

It never downloads FileAnnotation bytes or pixel data.

Raw lines 75 to 114
Runs code scripts/export_image_metadata.py :98
115## Namespaces
117Namespaces let tools assign semantics:
119```python
120for annotation in image.listAnnotations(ns="org.example.analysis.v1"):
121 print(annotation.getId())
122```
124Use an organization-controlled URI or reverse-domain pattern and document its
125schema/version. Do not claim a custom namespace is an OME standard.
127The current client constant for client map annotations is:
129```python
130from omero.constants.metadata import NSCLIENTMAPANNOTATION
131```
133OME's Python example warns that a client map annotation should be linked to
134only one object. Create a separate map annotation for each target when using
135that namespace.

Namespaces

Namespaces let tools assign semantics:

for annotation in image.listAnnotations(ns="org.example.analysis.v1"):
    print(annotation.getId())

Use an organization-controlled URI or reverse-domain pattern and document its schema/version. Do not claim a custom namespace is an OME standard.

The current client constant for client map annotations is:

from omero.constants.metadata import NSCLIENTMAPANNOTATION

OME's Python example warns that a client map annotation should be linked to only one object. Create a separate map annotation for each target when using that namespace.

Raw lines 115 to 136
137## Map Annotations
139Read key/value pairs only when approved:
141```python
142from omero.gateway import MapAnnotationWrapper
144for annotation in image.listAnnotations(ns="org.example.analysis.v1"):
145 if isinstance(annotation, MapAnnotationWrapper):
146 pairs = annotation.getValue()
147 for key, value in pairs[:50]:
148 print(key, value)
149```
151Apply independent limits to annotation count, pair count, key length, and
152value length. Keys can be sensitive too; a “values redacted” export that leaves
153participant IDs in keys is not redacted.
155Creating and linking is a write:
157```python
158from omero.constants.metadata import NSCLIENTMAPANNOTATION
159from omero.gateway import MapAnnotationWrapper
161image = conn.getObject("Image", image_id)
162if image is None:
163 raise LookupError("Image unavailable")
165pairs = [["Analysis version", "2.1"], ["Status", "reviewed"]]
166annotation = MapAnnotationWrapper(conn)
167annotation.setNs(NSCLIENTMAPANNOTATION)
168annotation.setValue(pairs)
169annotation.save()
170image.linkAnnotation(annotation)
171```
173Before running this:
175- confirm image ID and group;
176- confirm write permission and namespace;
177- validate pair count and string lengths;
178- decide rollback behavior if save succeeds but link creation fails;
179- never create duplicate metadata merely because an earlier query was scoped
180 to the wrong group.

Map Annotations

Read key/value pairs only when approved:

from omero.gateway import MapAnnotationWrapper

for annotation in image.listAnnotations(ns="org.example.analysis.v1"):
    if isinstance(annotation, MapAnnotationWrapper):
        pairs = annotation.getValue()
        for key, value in pairs[:50]:
            print(key, value)

Apply independent limits to annotation count, pair count, key length, and value length. Keys can be sensitive too; a “values redacted” export that leaves participant IDs in keys is not redacted.

Creating and linking is a write:

from omero.constants.metadata import NSCLIENTMAPANNOTATION
from omero.gateway import MapAnnotationWrapper

image = conn.getObject("Image", image_id)
if image is None:
    raise LookupError("Image unavailable")

pairs = [["Analysis version", "2.1"], ["Status", "reviewed"]]
annotation = MapAnnotationWrapper(conn)
annotation.setNs(NSCLIENTMAPANNOTATION)
annotation.setValue(pairs)
annotation.save()
image.linkAnnotation(annotation)

Before running this:

  • confirm image ID and group;
  • confirm write permission and namespace;
  • validate pair count and string lengths;
  • decide rollback behavior if save succeeds but link creation fails;
  • never create duplicate metadata merely because an earlier query was scoped to the wrong group.
Raw lines 137 to 181
182## Tags and Comments
184Tag creation and linking are separate writes:
186```python
187from omero.gateway import TagAnnotationWrapper
189tag = TagAnnotationWrapper(conn)
190tag.setValue("Reviewed")
191tag.setDescription("Reviewed under protocol v2")
192tag.save()
194image = conn.getObject("Image", image_id)
195if image is None:
196 raise LookupError("Image unavailable")
197image.linkAnnotation(tag)
198```
200Query for an existing controlled tag before creating another. Check group and
201owner semantics; do not reuse a same-named tag from an unintended group.
203Comments are free text and often the most sensitive annotation type. Do not
204include them in a general inventory. Never pass untrusted comments into shell,
205HTML, SQL/HQL, filenames, or dynamic code.

Tags and Comments

Tag creation and linking are separate writes:

from omero.gateway import TagAnnotationWrapper

tag = TagAnnotationWrapper(conn)
tag.setValue("Reviewed")
tag.setDescription("Reviewed under protocol v2")
tag.save()

image = conn.getObject("Image", image_id)
if image is None:
    raise LookupError("Image unavailable")
image.linkAnnotation(tag)

Query for an existing controlled tag before creating another. Check group and owner semantics; do not reuse a same-named tag from an unintended group.

Comments are free text and often the most sensitive annotation type. Do not include them in a general inventory. Never pass untrusted comments into shell, HTML, SQL/HQL, filenames, or dynamic code.

Raw lines 182 to 206
207## File Annotations
209Inspect metadata without downloading bytes:
211```python
212from omero.gateway import FileAnnotationWrapper
214for annotation in image.listAnnotations():
215 if isinstance(annotation, FileAnnotationWrapper):
216 original = annotation.getFile()
217 print(
218 {
219 "annotation_id": annotation.getId(),
220 "original_file_id": original.getId(),
221 "size": original.getSize(),
222 "mimetype": original.getMimetype(),
223 "name_redacted": True,
224 }
225 )
226```
228Remote filenames are untrusted input. Never join them directly to an output
229directory.
231For one explicitly approved file, check size and use a caller-chosen path:
233```python
234from pathlib import Path
236max_bytes = 50 * 1024 * 1024
237destination = Path("./approved-result.bin")
239original = file_annotation.getFile()
240if original.getSize() > max_bytes:
241 raise ValueError("File exceeds approved byte limit")
242if destination.exists() or destination.is_symlink():
243 raise FileExistsError(destination)
245written = 0
246with destination.open("xb") as handle:
247 for chunk in file_annotation.getFileInChunks():
248 written += len(chunk)
249 if written > max_bytes:
250 raise ValueError("Received more than approved byte limit")
251 handle.write(chunk)
252```
254On failure, delete the partial local file if policy permits. Do not download
255all file annotations attached to a project/dataset without an explicit list.
257Uploading is a mutation:
259```python
260source = "./approved-analysis.csv"
261annotation = conn.createFileAnnfromLocalFile(
262 source,
263 mimetype="text/csv",
264 ns="org.example.analysis.v1",
265 desc="Reviewed analysis results",
267dataset.linkAnnotation(annotation)
268```
270Check local file size, type, content classification, target dataset ID/group,
271and whether upload is permitted before execution.

File Annotations

Inspect metadata without downloading bytes:

from omero.gateway import FileAnnotationWrapper

for annotation in image.listAnnotations():
    if isinstance(annotation, FileAnnotationWrapper):
        original = annotation.getFile()
        print(
            {
                "annotation_id": annotation.getId(),
                "original_file_id": original.getId(),
                "size": original.getSize(),
                "mimetype": original.getMimetype(),
                "name_redacted": True,
            }
        )

Remote filenames are untrusted input. Never join them directly to an output directory.

For one explicitly approved file, check size and use a caller-chosen path:

from pathlib import Path

max_bytes = 50 * 1024 * 1024
destination = Path("./approved-result.bin")

original = file_annotation.getFile()
if original.getSize() > max_bytes:
    raise ValueError("File exceeds approved byte limit")
if destination.exists() or destination.is_symlink():
    raise FileExistsError(destination)

written = 0
with destination.open("xb") as handle:
    for chunk in file_annotation.getFileInChunks():
        written += len(chunk)
        if written > max_bytes:
            raise ValueError("Received more than approved byte limit")
        handle.write(chunk)

On failure, delete the partial local file if policy permits. Do not download all file annotations attached to a project/dataset without an explicit list.

Uploading is a mutation:

source = "./approved-analysis.csv"
annotation = conn.createFileAnnfromLocalFile(
    source,
    mimetype="text/csv",
    ns="org.example.analysis.v1",
    desc="Reviewed analysis results",
)
dataset.linkAnnotation(annotation)

Check local file size, type, content classification, target dataset ID/group, and whether upload is permitted before execution.

Raw lines 207 to 272
273## Numeric and Boolean Values
275Wrapper examples:
277```python
278from omero.gateway import (
279 BooleanAnnotationWrapper,
280 DoubleAnnotationWrapper,
281 LongAnnotationWrapper,
283```
285Numeric values still need units and semantics in the namespace/schema.
286Do not infer that `DoubleAnnotation` values are in micrometers or that a
287`LongAnnotation` is a count.

Numeric and Boolean Values

Wrapper examples:

from omero.gateway import (
    BooleanAnnotationWrapper,
    DoubleAnnotationWrapper,
    LongAnnotationWrapper,
)

Numeric values still need units and semantics in the namespace/schema. Do not infer that DoubleAnnotation values are in micrometers or that a LongAnnotation is a count.

Raw lines 273 to 288
289## Unlink Versus Delete
291- **Unlink** deletes an object-annotation link but keeps the annotation and
292 other links.
293- **Delete annotation** deletes the annotation and may affect every linked
294 object.
296Before either operation:
2981. retrieve and display exact link/annotation IDs;
2992. count other links;
3003. verify ownership and permission;
3014. obtain explicit approval for the exact operation;
3025. do not use a namespace-only bulk delete without an ID review;
3036. wait for and check command completion.
305Read-only export utilities must not include delete/unlink modes.

Unlink Versus Delete

  • Unlink deletes an object-annotation link but keeps the annotation and other links.
  • Delete annotation deletes the annotation and may affect every linked object.

Before either operation:

  1. retrieve and display exact link/annotation IDs;
  2. count other links;
  3. verify ownership and permission;
  4. obtain explicit approval for the exact operation;
  5. do not use a namespace-only bulk delete without an ID review;
  6. wait for and check command completion.

Read-only export utilities must not include delete/unlink modes.

Raw lines 289 to 306
307## Metadata Export Checklist
309- Explicit object type and IDs
310- One group context
311- Maximum objects, annotations, links, pairs, and string length
312- Values redacted by default
313- File names and owner names separately gated
314- No attachment bytes unless one file and byte cap are approved
315- Output path chosen by caller; no remote-derived path
316- Atomic write with owner-only permissions
317- Connection closed in `finally`

Metadata Export Checklist

  • Explicit object type and IDs
  • One group context
  • Maximum objects, annotations, links, pairs, and string length
  • Values redacted by default
  • File names and owner names separately gated
  • No attachment bytes unless one file and byte cap are approved
  • Output path chosen by caller; no remote-derived path
  • Atomic write with owner-only permissions
  • Connection closed in finally
Raw lines 307 to 317