Skill Tracer
Skills › Scientific Agent Skills › matlab › data-import-export.md

data-import-export.md

236 lines · skills/matlab/references/data-import-export.md at 5f2bc03

Open on GitHub

Most marks come from matching words in the file, so treat them as pointers.

1# Data Import, Tables, Timetables, and MAT Files
2
3This reference targets MATLAB R2026a. Treat every external file as untrusted
4until its provenance, size, structure, and parser risk are reviewed.
5

Data Import, Tables, Timetables, and MAT Files

This reference targets MATLAB R2026a. Treat every external file as untrusted until its provenance, size, structure, and parser risk are reviewed.

Raw lines 1 to 5
6## Safe import workflow
7
81. Accept one named local path under a confirmed root.
92. Reject URLs, traversal, symlinks, device files, and unexpected extensions.
103. Bound compressed and uncompressed size, rows, columns, variables, nesting,
11 strings, and HDF5 objects.
124. Inventory format and metadata before loading values.
135. Define schema, classes, units, encoding, missing sentinels, time zones, and
14 duplicate policy.
156. Import the narrowest columns/ranges needed.
167. Validate before computation.
178. Write to a new local output; refuse accidental overwrite.
19Do not use a broad directory scan or environment dump to find data. Remote
20imports add network, redirect, credential, and changing-content risks; download
21them through a separately approved, checksum-recorded workflow.

Safe import workflow

  1. Accept one named local path under a confirmed root.
  2. Reject URLs, traversal, symlinks, device files, and unexpected extensions.
  3. Bound compressed and uncompressed size, rows, columns, variables, nesting, strings, and HDF5 objects.
  4. Inventory format and metadata before loading values.
  5. Define schema, classes, units, encoding, missing sentinels, time zones, and duplicate policy.
  6. Import the narrowest columns/ranges needed.
  7. Validate before computation.
  8. Write to a new local output; refuse accidental overwrite.

Do not use a broad directory scan or environment dump to find data. Remote imports add network, redirect, credential, and changing-content risks; download them through a separately approved, checksum-recorded workflow.

Raw lines 6 to 22
23## High-level text and spreadsheet import
25Choose the output model intentionally:
27```matlab
28options = detectImportOptions("measurements.csv", ...
29 TextType="string");
30options.SelectedVariableNames = ...
31 ["SampleID" "Timestamp" "Value" "Quality"];
32options = setvartype(options, "SampleID", "string");
33T = readtable("measurements.csv", options);
34```
36- `readtable`: mixed, named column-oriented data.
37- `readmatrix`: homogeneous numeric data.
38- `readcell`: heterogeneous cells when a table schema is inappropriate.
39- `readlines`/`fileread`: bounded text, with explicit encoding expectations.
40- `readtimetable`: time-indexed data when row-time semantics are known.
42Use `writetable`, `writematrix`, `writecell`, `writelines`, or
43`writetimetable` for corresponding exports. Text/spreadsheet round trips can
44change formatting, precision, names, multidimensional variables, empty values,
45or types. If exact MATLAB structure matters and the file is trusted, a MAT file
46can preserve it—but MAT files have object/code risks and are not a universal
47interchange format.
49R2026a adds JSON read/write support for tables and timetables. Define the JSON
50orientation/schema and test consumers; "JSON" alone does not specify table
51shape, time representation, or missing semantics.

High-level text and spreadsheet import

Choose the output model intentionally:

options = detectImportOptions("measurements.csv", ...
    TextType="string");
options.SelectedVariableNames = ...
    ["SampleID" "Timestamp" "Value" "Quality"];
options = setvartype(options, "SampleID", "string");
T = readtable("measurements.csv", options);
  • readtable: mixed, named column-oriented data.
  • readmatrix: homogeneous numeric data.
  • readcell: heterogeneous cells when a table schema is inappropriate.
  • readlines/fileread: bounded text, with explicit encoding expectations.
  • readtimetable: time-indexed data when row-time semantics are known.

Use writetable, writematrix, writecell, writelines, or writetimetable for corresponding exports. Text/spreadsheet round trips can change formatting, precision, names, multidimensional variables, empty values, or types. If exact MATLAB structure matters and the file is trusted, a MAT file can preserve it—but MAT files have object/code risks and are not a universal interchange format.

R2026a adds JSON read/write support for tables and timetables. Define the JSON orientation/schema and test consumers; "JSON" alone does not specify table shape, time representation, or missing semantics.

Raw lines 23 to 52
53## Tables and timetables
55```matlab
56required = ["SampleID" "Timestamp" "Value"];
57assert(all(ismember(required, string(T.Properties.VariableNames))));
58assert(isstring(T.SampleID));
59assert(isdatetime(T.Timestamp));
60assert(isnumeric(T.Value));
61```
63Table rules:
65- all variables have the same row count;
66- variables may differ in class and width;
67- `T(rows,vars)` preserves a table;
68- `T{rows,vars}` extracts/concatenates contents;
69- `T.Var` extracts one variable;
70- properties can store units and descriptions but are not always preserved by
71 external formats.
73Timetable rules:
75- row times are distinct metadata, not an ordinary variable;
76- sort and validate row times;
77- preserve or normalize `TimeZone`;
78- define duplicates before `retime` or `synchronize`;
79- choose interpolation/aggregation and union/intersection deliberately;
80- validate missing row times separately from `ismissing(TT)`.

Tables and timetables

required = ["SampleID" "Timestamp" "Value"];
assert(all(ismember(required, string(T.Properties.VariableNames))));
assert(isstring(T.SampleID));
assert(isdatetime(T.Timestamp));
assert(isnumeric(T.Value));

Table rules:

  • all variables have the same row count;
  • variables may differ in class and width;
  • T(rows,vars) preserves a table;
  • T{rows,vars} extracts/concatenates contents;
  • T.Var extracts one variable;
  • properties can store units and descriptions but are not always preserved by external formats.

Timetable rules:

  • row times are distinct metadata, not an ordinary variable;
  • sort and validate row times;
  • preserve or normalize TimeZone;
  • define duplicates before retime or synchronize;
  • choose interpolation/aggregation and union/intersection deliberately;
  • validate missing row times separately from ismissing(TT).
Raw lines 53 to 81
82## Missing values
84Standard indicators:
86| Class | Standard missing |
87|---|---|
88| `double`, `single`, `duration`, `calendarDuration` | `NaN` |
89| `datetime` | `NaT` |
90| `string` | `<missing>` |
91| `categorical` | `<undefined>` |
92| cell array of character vectors | empty character vector |
93| integer/logical | none |
95Use `standardizeMissing` when source sentinels are documented. Include
96`missing` in a custom indicator list when you intend to preserve standard
97indicators too. `Inf` is not missing by default.
99Never call `rmmissing` as generic cleaning without reporting what rows,
100variables, groups, or time coverage were removed.

Missing values

Standard indicators:

ClassStandard missing
double, single, duration, calendarDurationNaN
datetimeNaT
string<missing>
categorical<undefined>
cell array of character vectorsempty character vector
integer/logicalnone

Use standardizeMissing when source sentinels are documented. Include missing in a custom indicator list when you intend to preserve standard indicators too. Inf is not missing by default.

Never call rmmissing as generic cleaning without reporting what rows, variables, groups, or time coverage were removed.

Raw lines 82 to 101
102## MAT file versions
104MAT files are MATLAB binary workspace containers:
106| Version | `save` option | Compression | Key capability/limit |
107|---|---|---|---|
108| 4 | `"-v4"` | no | 2-D double, character, sparse; legacy |
109| 6 | `"-v6"` | no | N-D, cell, structure; under 2 GiB per variable |
110| 7 | `"-v7"` | yes | Unicode and v6 features; under 2 GiB per variable |
111| 7.3 | `"-v7.3"` | yes/chunked | HDF5-based, partial access, variables at least 2 GiB on 64-bit |
113Factory-default `save` operations use version 7, but the user can change the
114default in MAT and FIG Files settings. Explicitly specify `-v7` or `-v7.3`
115when exchange compatibility matters. Creating a new file with
116`matfile` defaults to version 7.3. File-system limits still apply. Version 7.3
117adds HDF5 metadata/chunk overhead and can be larger for heterogeneous
118containers.
120Do not label arbitrary HDF5 as MATLAB v7.3. The format is HDF5-based but has
121MATLAB conventions, references, metadata, and type encodings. GNU Octave 11
122cannot save MATLAB v7.3 and has only limited HDF5-based read support.

MAT file versions

MAT files are MATLAB binary workspace containers:

Versionsave optionCompressionKey capability/limit
4"-v4"no2-D double, character, sparse; legacy
6"-v6"noN-D, cell, structure; under 2 GiB per variable
7"-v7"yesUnicode and v6 features; under 2 GiB per variable
7.3"-v7.3"yes/chunkedHDF5-based, partial access, variables at least 2 GiB on 64-bit

Factory-default save operations use version 7, but the user can change the default in MAT and FIG Files settings. Explicitly specify -v7 or -v7.3 when exchange compatibility matters. Creating a new file with matfile defaults to version 7.3. File-system limits still apply. Version 7.3 adds HDF5 metadata/chunk overhead and can be larger for heterogeneous containers.

Do not label arbitrary HDF5 as MATLAB v7.3. The format is HDF5-based but has MATLAB conventions, references, metadata, and type encodings. GNU Octave 11 cannot save MATLAB v7.3 and has only limited HDF5-based read support.

Raw lines 102 to 123
124## MAT safety
126Never `load` an untrusted MAT file, even if selecting one variable. A MAT file
127can contain:
129- MATLAB objects whose classes customize deserialization with `loadobj` or
130 custom element serialization;
131- constructors, listeners, or System object load hooks reachable from class
132 restoration;
133- function handles and opaque values;
134- Java objects and data interpreted by installed code;
135- deeply nested/compressed structures that exhaust resources.
137`whos("-file", path)` is useful inside an already approved MATLAB environment,
138but invoking MATLAB is itself execution. The bundled
139`scripts/inventory_mat_file.py` never launches MATLAB and:
141- identifies the header/version;
142- optionally uses `scipy.io.whosmat` for Level-5 metadata only;
143- optionally uses `h5py` for bounded HDF5 names, shapes, dtypes, links, and
144 attribute names;
145- never calls `scipy.io.loadmat`;
146- never reads dataset values or follows soft/external HDF5 links;
147- never deserializes objects or Python pickle.
149HDF5 inventory distinguishes null dataspaces (`shape: null`) from scalar
150datasets (`shape: []`) and zero-length arrays (for example, `shape: [0]`).
151All three are valid HDF5 metadata states, and none is proof of MATLAB v7.3.
153An inventory is triage, not a safety certificate. Object-like, opaque,
154function, external-link, malformed, or unsupported content requires
155quarantine and expert review.
157R2026b changes failure recovery for loading some scalar and non-heterogeneous
158object arrays: MATLAB can substitute default-constructed objects, or return
159an empty class array if replacement fails. In trusted migration tests, check
160load warnings, classes, dimensions, and field invariants; successful completion
161alone does not prove the saved scientific state was restored.

MAT safety

Never load an untrusted MAT file, even if selecting one variable. A MAT file can contain:

  • MATLAB objects whose classes customize deserialization with loadobj or custom element serialization;
  • constructors, listeners, or System object load hooks reachable from class restoration;
  • function handles and opaque values;
  • Java objects and data interpreted by installed code;
  • deeply nested/compressed structures that exhaust resources.

whos("-file", path) is useful inside an already approved MATLAB environment, but invoking MATLAB is itself execution. The bundled scripts/inventory_mat_file.py never launches MATLAB and:

  • identifies the header/version;
  • optionally uses scipy.io.whosmat for Level-5 metadata only;
  • optionally uses h5py for bounded HDF5 names, shapes, dtypes, links, and attribute names;
  • never calls scipy.io.loadmat;
  • never reads dataset values or follows soft/external HDF5 links;
  • never deserializes objects or Python pickle.

HDF5 inventory distinguishes null dataspaces (shape: null) from scalar datasets (shape: []) and zero-length arrays (for example, shape: [0]). All three are valid HDF5 metadata states, and none is proof of MATLAB v7.3.

An inventory is triage, not a safety certificate. Object-like, opaque, function, external-link, malformed, or unsupported content requires quarantine and expert review.

R2026b changes failure recovery for loading some scalar and non-heterogeneous object arrays: MATLAB can substitute default-constructed objects, or return an empty class array if replacement fails. In trusted migration tests, check load warnings, classes, dimensions, and field invariants; successful completion alone does not prove the saved scientific state was restored.

Raw lines 124 to 162
Runs code scripts/inventory_mat_file.py :139
163## Partial access
165For a trusted version 7.3 file:
167```matlab
168file = matfile("trusted-large.mat");
169shape = size(file, "measurements");
170block = file.measurements(1:1000, :);
171```
173`matfile` avoids loading an entire variable, but it still processes a MAT file
174and can expose class/content risks. Partial read performance depends on HDF5
175chunk layout. Do not use it as a security sandbox.

Partial access

For a trusted version 7.3 file:

file = matfile("trusted-large.mat");
shape = size(file, "measurements");
block = file.measurements(1:1000, :);

matfile avoids loading an entire variable, but it still processes a MAT file and can expose class/content risks. Partial read performance depends on HDF5 chunk layout. Do not use it as a security sandbox.

Raw lines 163 to 176
177## Low-level I/O
179Use `onCleanup` to close reviewed files:
181```matlab
182[fid, message] = fopen("trusted-input.bin", "rb");
183assert(fid >= 0, message);
184cleanup = onCleanup(@() fclose(fid));
185values = fread(fid, [4 1000], "single=>single");
186```
188Specify byte order, element type, dimensions, record framing, and maximum
189length. Validate `fread` counts and check arithmetic for overflow before
190allocating.
192HDF5, netCDF, CDF, FITS, Parquet, audio, video, images, databases, and
193spreadsheets each have format/library/product/platform constraints. Use their
194official current documentation and enforce parser-specific bounds.

Low-level I/O

Use onCleanup to close reviewed files:

[fid, message] = fopen("trusted-input.bin", "rb");
assert(fid >= 0, message);
cleanup = onCleanup(@() fclose(fid));
values = fread(fid, [4 1000], "single=>single");

Specify byte order, element type, dimensions, record framing, and maximum length. Validate fread counts and check arithmetic for overflow before allocating.

HDF5, netCDF, CDF, FITS, Parquet, audio, video, images, databases, and spreadsheets each have format/library/product/platform constraints. Use their official current documentation and enforce parser-specific bounds.

Raw lines 177 to 195
196## Export and provenance
198Record:
200- source and output checksums;
201- schema/version, encoding, delimiter, locale, and numeric precision;
202- variable names, classes, units, dimensions, missing rules;
203- timestamp/time-zone representation;
204- sort/group order;
205- MAT version or external format/library;
206- MATLAB release and required products.
208Prefer a documented language-neutral format for exchange:
210- CSV/TSV for simple rectangular values with a sidecar schema;
211- JSON for bounded structured data with an explicit schema;
212- Parquet for typed tabular interchange when all consumers agree;
213- HDF5/netCDF for scientific arrays with documented conventions;
214- MAT only for trusted MATLAB-oriented storage.
216Python pickle is executable deserialization, not a scientific interchange
217format. Never create, load, or recommend pickle for MATLAB exchange.

Export and provenance

Record:

  • source and output checksums;
  • schema/version, encoding, delimiter, locale, and numeric precision;
  • variable names, classes, units, dimensions, missing rules;
  • timestamp/time-zone representation;
  • sort/group order;
  • MAT version or external format/library;
  • MATLAB release and required products.

Prefer a documented language-neutral format for exchange:

  • CSV/TSV for simple rectangular values with a sidecar schema;
  • JSON for bounded structured data with an explicit schema;
  • Parquet for typed tabular interchange when all consumers agree;
  • HDF5/netCDF for scientific arrays with documented conventions;
  • MAT only for trusted MATLAB-oriented storage.

Python pickle is executable deserialization, not a scientific interchange format. Never create, load, or recommend pickle for MATLAB exchange.

Raw lines 196 to 218
219## Sources (reviewed 2026-10-01)
221- [Data Import and Export](https://www.mathworks.com/help/matlab/data-import-and-export.html)
222- [`detectImportOptions`](https://www.mathworks.com/help/matlab/ref/detectimportoptions.html)
223- [`readtable`](https://www.mathworks.com/help/matlab/ref/readtable.html)
224- [`writetable`](https://www.mathworks.com/help/matlab/ref/writetable.html)
225- [Tables](https://www.mathworks.com/help/matlab/tables.html)
226- [Timetables](https://www.mathworks.com/help/matlab/timetables.html)
227- [`ismissing`](https://www.mathworks.com/help/matlab/ref/ismissing.html)
228- [MAT File Versions](https://www.mathworks.com/help/matlab/import_export/mat-file-versions.html)
229- [`MatFile`](https://www.mathworks.com/help/matlab/ref/matlab.io.matfile.html)
230- [Object Save and Load](https://www.mathworks.com/help/matlab/save-and-load.html)
231- [`loadobj`](https://www.mathworks.com/help/matlab/ref/loadobj.html)
232- [HDF5 Files](https://www.mathworks.com/help/matlab/hdf5-files.html)
233- [MATLAB release notes](https://www.mathworks.com/help/matlab/release-notes.html)
235- [SciPy MAT metadata inventory](https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.whosmat.html)
236- [h5py null dataspaces](https://docs.h5py.org/en/stable/high/dataset.html#creating-and-reading-empty-or-null-datasets-and-attributes)

Sources (reviewed 2026-10-01)

  • Data Import and Export
  • [detectImportOptions](https://www.mathworks.com/help/matlab/ref/detectimportoptions.html)
  • [readtable](https://www.mathworks.com/help/matlab/ref/readtable.html)
  • [writetable](https://www.mathworks.com/help/matlab/ref/writetable.html)
  • Tables
  • Timetables
  • [ismissing](https://www.mathworks.com/help/matlab/ref/ismissing.html)
  • MAT File Versions
  • [MatFile](https://www.mathworks.com/help/matlab/ref/matlab.io.matfile.html)
  • Object Save and Load
  • [loadobj](https://www.mathworks.com/help/matlab/ref/loadobj.html)
  • HDF5 Files
  • MATLAB release notes
Raw lines 219 to 236