Direct Parquet Access Guide for IDC

Reviewed 2026-09-30: idc-index-data 24.2.2 (IDC data version v24), DuckDB 1.5

All idc-index metadata tables are published as Parquet files to a public GCS bucket with unrestricted CORS access. This enables metadata queries with DuckDB or pandas without installing idc-index.

Limitation: download helpers (download_from_selection()), viewer URLs (get_viewer_URL()), and citation generation require the idc-index client and are not available from raw Parquet files.

This is not the first no-install option to reach for. It still needs DuckDB installed. This GCS artifact set contains only the clinical_index dictionary; full clinical and BigQuery tables are exported separately to public S3 (below). For ad-hoc metadata with nothing installed, the REST API (rest_api_guide.md) needs no install at all and reaches clinical.<table> through POST /sql.

When to Use This Guide

Load this guide when you need to:

• Pin queries to a specific IDC data version (see Pinning to a Specific Version below) rather than whatever the hosted API currently serves
• Return more rows than the REST /sql ceiling of 10 000
• Run heavy or repeated local DuckDB analysis without driving the hosted API
• Query IDC metadata where DuckDB is available but idc-index is not

For downloads, viewer URLs, and citations, use idc-index as documented in the main SKILL.md.

URL Pattern

https://storage.googleapis.com/idc-index-data-artifacts/current/release_artifacts/{filename}.parquet

current/ always resolves to the latest data release. To pin to a specific version, replace current with the data version number (e.g., 23.10.1).

Available Files

Note: the main index file is named idc_index.parquet, not index.parquet. Reference it with an alias in SQL queries (e.g., FROM read_parquet(...) AS index).

Prerequisites

Install the Python duckdb package, using whatever installer manages the environment you are running in.

DuckDB reads Parquet directly from HTTPS URLs using HTTP range requests — no GCS client library or authentication required.

Basic Queries

import duckdb

BASE = "https://storage.googleapis.com/idc-index-data-artifacts/current/release_artifacts"

# Discover modalities and series counts
duckdb.sql(f"""
    SELECT Modality, COUNT(*) as series_count, ROUND(SUM(series_size_MB)/1000, 1) as size_GB
    FROM read_parquet('{BASE}/idc_index.parquet')
    GROUP BY Modality
    ORDER BY series_count DESC
""").df()

# Collections with CT data, ordered by size
duckdb.sql(f"""
    SELECT collection_id,
           COUNT(DISTINCT PatientID) as patients,
           COUNT(*) as series,
           ROUND(SUM(series_size_MB)/1000, 1) as size_GB
    FROM read_parquet('{BASE}/idc_index.parquet')
    WHERE Modality = 'CT'
    GROUP BY collection_id
    ORDER BY size_GB DESC
    LIMIT 10
""").df()

Volume Geometry Validation

volume_geometry_index covers single-frame CT, MR, and PT series. Each row has boolean checks for orientation, spacing, dimensions, and slice positions, plus a composite regularly_spaced_3d_volume flag.

import duckdb

BASE = "https://storage.googleapis.com/idc-index-data-artifacts/current/release_artifacts"

# Candidate regularly spaced CT series; verify task-specific geometry before analysis
duckdb.sql(f"""
    SELECT i.collection_id, i.SeriesInstanceUID, i.BodyPartExamined,
           v.obliquity_degrees, v.regularly_spaced_3d_volume
    FROM read_parquet('{BASE}/idc_index.parquet') i
    JOIN read_parquet('{BASE}/volume_geometry_index.parquet') v
        ON i.SeriesInstanceUID = v.SeriesInstanceUID
    WHERE i.Modality = 'CT'
      AND v.regularly_spaced_3d_volume = TRUE
    LIMIT 10
""").df()

# Fraction of 3D-valid series per collection and modality
duckdb.sql(f"""
    SELECT i.collection_id, i.Modality,
           COUNT(*) as total,
           SUM(CASE WHEN v.regularly_spaced_3d_volume THEN 1 ELSE 0 END) as valid_3d,
           ROUND(100.0 * SUM(CASE WHEN v.regularly_spaced_3d_volume THEN 1 ELSE 0 END) / COUNT(*), 1) as pct_valid
    FROM read_parquet('{BASE}/idc_index.parquet') i
    JOIN read_parquet('{BASE}/volume_geometry_index.parquet') v
        ON i.SeriesInstanceUID = v.SeriesInstanceUID
    WHERE i.Modality IN ('CT', 'MR', 'PT')
    GROUP BY i.collection_id, i.Modality
    ORDER BY total DESC
    LIMIT 10
""").df()

Key columns in volume_geometry_index:

RT Structure Sets

rtstruct_index has one row per RTSTRUCT series with aggregated ROI metadata.

import duckdb

BASE = "https://storage.googleapis.com/idc-index-data-artifacts/current/release_artifacts"

# RTSTRUCT series with ROI details
duckdb.sql(f"""
    SELECT i.collection_id, i.SeriesInstanceUID,
           r.total_rois, r.ROINames, r.RTROIInterpretedTypes,
           r.referenced_SeriesInstanceUID
    FROM read_parquet('{BASE}/idc_index.parquet') i
    JOIN read_parquet('{BASE}/rtstruct_index.parquet') r
        ON i.SeriesInstanceUID = r.SeriesInstanceUID
    WHERE i.Modality = 'RTSTRUCT'
    LIMIT 5
""").df()

# Collections with the most RTSTRUCT series
duckdb.sql(f"""
    SELECT i.collection_id,
           COUNT(*) as rtstruct_series,
           ROUND(AVG(r.total_rois), 1) as avg_rois_per_struct
    FROM read_parquet('{BASE}/idc_index.parquet') i
    JOIN read_parquet('{BASE}/rtstruct_index.parquet') r
        ON i.SeriesInstanceUID = r.SeriesInstanceUID
    GROUP BY i.collection_id
    ORDER BY rtstruct_series DESC
    LIMIT 10
""").df()

Key columns in rtstruct_index:

Pinning to a Specific Version

import duckdb

# Use a specific data release instead of 'current'
VERSION = "24.2.2"
BASE = f"https://storage.googleapis.com/idc-index-data-artifacts/{VERSION}/release_artifacts"

duckdb.sql(f"SELECT COUNT(*) FROM read_parquet('{BASE}/idc_index.parquet')").df()

Full metadata and clinical exports

The separate public bucket idc-open-metadata exports BigQuery tables, including per- collection clinical rows, per-segment metadata, and SR measurements. No Google account is required. Use a pinned dataset such as idc_v24_clinical; discover object keys first because a table can be one file or a directory of shards. Do not append .parquet blindly.

s5cmd --no-sign-request ls s3://idc-open-metadata/bigquery_export/idc_v24_clinical/
s5cmd --no-sign-request ls s3://idc-open-metadata/bigquery_export/idc_v24/

For example, after confirming the key, the clinical table can be queried directly:

import duckdb
url = "https://idc-open-metadata.s3.amazonaws.com/bigquery_export/idc_v24_clinical/nlst_canc/000000000000.parquet"
clinical = duckdb.sql(f"SELECT dicom_patient_id, clinical_stag FROM read_parquet('{url}') LIMIT 5").df()

Full instance tables can be very large. Inspect their schema and estimate resources before querying all shards; GoogleSQL from the BigQuery guide needs adaptation to DuckDB syntax. See IDC files and metadata.

Resources

• idc-index-data releases: https://github.com/ImagingDataCommons/idc-index-data/releases
• idc-index documentation: https://idc-index.readthedocs.io/
• IDC Portal: https://portal.imaging.datacommons.cancer.gov/