Skills › Scientific Agent Skills › dask › review.md

review.md

106 lines · skills/dask/references/review.md at 5f2bc03

Open on GitHub

1# Current contract and validation record

3Reviewed September 30, 2026 against official documentation and released packages.

4The recipes target Dask/distributed **2026.8.0**, Python **3.13.3**, pandas **3.0.6**,

5NumPy **2.5.1**, PyArrow **25.0.1**, SciPy **1.18.1**, Zarr **3.4.0**, h5py **3.16.0**,

6Xarray **2026.9.0**, and Dask-ML **2025.1.0**. Core Dask requires Python 3.10+;

7these current optional packages have stricter requirements (Zarr 3.4: Python 3.12+).

8Use an isolated environment instead of changing another scientific package's stack.

Current contract and validation record

Reviewed September 30, 2026 against official documentation and released packages. The recipes target Dask/distributed 2026.8.0, Python 3.13.3, pandas 3.0.6, NumPy 2.5.1, PyArrow 25.0.1, SciPy 1.18.1, Zarr 3.4.0, h5py 3.16.0, Xarray 2026.9.0, and Dask-ML 2025.1.0. Core Dask requires Python 3.10+; these current optional packages have stricter requirements (Zarr 3.4: Python 3.12+). Use an isolated environment instead of changing another scientific package's stack.

Raw lines 1 to 9

10## Contract corrections that affect results

12- `da.from_delayed` wraps one delayed array. Assemble a grid with `da.block` after

13 wrapping each block separately. Declare its actual shape and dtype.

14- Exact `svd` requires chunking along one dimension only; the QR recipe uses one

15 column of chunks. A randomized compressed SVD is a different, approximate method.

16- `map_blocks` processes blocks independently. Use `map_overlap` for Gaussian filters,

17 with halo depth matching the filter radius, floating output, and explicit spatial

18 axes. A per-block FFT differs from a global FFT, which requires a single chunk along

19 each transform axis. Chunk-local means require sample-count weighting for global means.

20- `drop_axis` combines all chunks on that axis before mapping; it can defeat memory

21 bounds. The block-mean example preserves rank instead.

22- `set_index(sorted=True)` asserts existing ordering. Use ordinary `set_index` to sort

23 arbitrary input before time resampling; parse timestamps and check timezone policy.

24- Name aggregated output columns before Parquet writing; MultiIndex/non-string column

25 names are rejected by the Dask writer. The ETL example uses named aggregations.

26- CSV blocksize is source bytes, not pandas memory. Inferred dtypes can fail on later

27 files; specify null-capable dtypes where needed and validate complete partitions.

28- `map_partitions(meta=...)` supplies a structural contract, not automatic dtype

29 conversion. Clear divisions when changing index order. Row-wise apply loops over

30 pandas rows within each partition; it does not create one task per row.

31- Bag `from_sequence` consumes its input locally: pass filenames and map a loader.

32 `read_text(...).map(json.loads)` assumes JSON Lines. `foldby` needs a binary combine

33 function such as addition, not Python's iterable `sum`.

34- `to_zarr` has `mode='a'` by default. Use `mode='w-'` for new outputs and explicitly

35 choose `mode='w'` only when replacing output is intended. Its `**zarr_array_kwargs`

36 are direct keywords (e.g. `zarr_format=3`), not a nested `zarr_array_kwargs=` mapping.

37- HDF5 handles must stay open through computation and are not serializable to process

38 workers. Local HDF5 examples explicitly use threads; remote workers must open their

39 own files and return concrete chunks.

Contract corrections that affect results

da.from_delayed wraps one delayed array. Assemble a grid with da.block after wrapping each block separately. Declare its actual shape and dtype.
Exact svd requires chunking along one dimension only; the QR recipe uses one column of chunks. A randomized compressed SVD is a different, approximate method.
map_blocks processes blocks independently. Use map_overlap for Gaussian filters, with halo depth matching the filter radius, floating output, and explicit spatial axes. A per-block FFT differs from a global FFT, which requires a single chunk along each transform axis. Chunk-local means require sample-count weighting for global means.
drop_axis combines all chunks on that axis before mapping; it can defeat memory bounds. The block-mean example preserves rank instead.
set_index(sorted=True) asserts existing ordering. Use ordinary set_index to sort arbitrary input before time resampling; parse timestamps and check timezone policy.
Name aggregated output columns before Parquet writing; MultiIndex/non-string column names are rejected by the Dask writer. The ETL example uses named aggregations.
CSV blocksize is source bytes, not pandas memory. Inferred dtypes can fail on later files; specify null-capable dtypes where needed and validate complete partitions.
map_partitions(meta=...) supplies a structural contract, not automatic dtype conversion. Clear divisions when changing index order. Row-wise apply loops over pandas rows within each partition; it does not create one task per row.
Bag from_sequence consumes its input locally: pass filenames and map a loader. read_text(...).map(json.loads) assumes JSON Lines. foldby needs a binary combine function such as addition, not Python's iterable sum.
to_zarr has mode='a' by default. Use mode='w-' for new outputs and explicitly choose mode='w' only when replacing output is intended. Its **zarr_array_kwargs are direct keywords (e.g. zarr_format=3), not a nested zarr_array_kwargs= mapping.
HDF5 handles must stay open through computation and are not serializable to process workers. Local HDF5 examples explicitly use threads; remote workers must open their own files and return concrete chunks.

Raw lines 10 to 40

41## Distributed execution

43`submit` schedules work immediately but execution waits for resources and dependencies.

44List/dict scatter returns corresponding collections of futures; wrap the object in a

45one-element list and unpack it to obtain one future for the whole object. Scattered

46objects still move between workers and are lost if their only replicas disappear.

47Load substantial data on workers where possible. `compute()` and `gather()` collect

48results into the client; partitioned writes execute without first collecting a pandas

49DataFrame or NumPy array. `persist()` retains partitions, requires a memory budget, and

50can prevent later DataFrame I/O projection/filter pushdown.

52Nested blocking worker tasks use `worker_client()` so child tasks can run even with

53one initial worker slot. Blocking queue/event patterns need spare runnable threads and

54finite timeouts in production. Stateful/side-effect submissions need `pure=False` and

55idempotent external effects; `fire_and_forget` provides no exactly-once guarantee.

56Actors are not automatically reconstructed after worker loss. Distributed lock leases

57are not a replacement for durable storage transactions.

59Use `Client.register_plugin`, and import Kubernetes `KubeCluster` from

60`dask_kubernetes.operator`. The reviewed optional releases are Dask Kubernetes

61**2026.3.0** and Dask-jobqueue **0.9.0**. Kubernetes requires its operator/CRDs,

62authorized kubeconfig, and a worker image matching the client environment. SLURM

63requires site-specific queue/account/resource policy. Close separately created clusters

64as well as clients. Adaptive scaling is a cluster method; a client connected only to an

65address need not own a cluster object.

Distributed execution

submit schedules work immediately but execution waits for resources and dependencies. List/dict scatter returns corresponding collections of futures; wrap the object in a one-element list and unpack it to obtain one future for the whole object. Scattered objects still move between workers and are lost if their only replicas disappear. Load substantial data on workers where possible. compute() and gather() collect results into the client; partitioned writes execute without first collecting a pandas DataFrame or NumPy array. persist() retains partitions, requires a memory budget, and can prevent later DataFrame I/O projection/filter pushdown.

Nested blocking worker tasks use worker_client() so child tasks can run even with one initial worker slot. Blocking queue/event patterns need spare runnable threads and finite timeouts in production. Stateful/side-effect submissions need pure=False and idempotent external effects; fire_and_forget provides no exactly-once guarantee. Actors are not automatically reconstructed after worker loss. Distributed lock leases are not a replacement for durable storage transactions.

Use Client.register_plugin, and import Kubernetes KubeCluster from dask_kubernetes.operator. The reviewed optional releases are Dask Kubernetes 2026.3.0 and Dask-jobqueue 0.9.0. Kubernetes requires its operator/CRDs, authorized kubeconfig, and a worker image matching the client environment. SLURM requires site-specific queue/account/resource policy. Close separately created clusters as well as clients. Adaptive scaling is a cluster method; a client connected only to an address need not own a cluster object.

Raw lines 41 to 66

67## Executed verification and limits

69`tests/dask/test_recipes.py` passed **19 tests** in the isolated skill environment. It executes selected Markdown recipes against small synthetic

70fixtures: delayed blocks; unequal block means; Gaussian image seams and boundaries;

71SVD/QR reconstruction; local Zarr/HDF5 roundtrips; constant-input normalization;

72unsorted time-series resampling; CSV/Parquet ETL; Bag loading and foldby; nested worker

73tasks; scatter; actors; queues/events/variables; plugin registration; diagnostic HTML;

74Dask-ML scaling and Xarray; guarded process scheduling and a process worker.

76The numerical checks compare against NumPy, pandas, or SciPy. Use tolerance-based

77comparisons for floating reductions because partition order can change rounding. Check

78units, missing-data policy, random seeds, coordinate order, and output dtypes before

79scaling scientific data. Small fixtures establish recipe correctness, not large-data

80throughput, peak memory, fault tolerance, or cluster capacity.

82Local clients used one or two threads with dashboards disabled; one guarded subprocess

83used the processes scheduler and a separate worker process. A local performance-report

84HTML file was generated; no browser/dashboard interaction or externally exposed service

85was tested. Bind personal dashboards to loopback (e.g. `dashboard_address='127.0.0.1:8787'`)

86and use the deployment's authenticated access/TLS policy for remote schedulers. A

87scheduler address is not an authenticated public web API.

89Cloud S3/GCS paths, SLURM, Kubernetes creation/scaling, production adaptive scaling,

90external database writes, and hardware-scale workloads are illustrative and were not

91executed. The skill implements no REST API client, authentication flow, or pagination.

92Graphviz rendering is optional and needs both its Python package and system executable.

Executed verification and limits

tests/dask/test_recipes.py passed 19 tests in the isolated skill environment. It executes selected Markdown recipes against small synthetic fixtures: delayed blocks; unequal block means; Gaussian image seams and boundaries; SVD/QR reconstruction; local Zarr/HDF5 roundtrips; constant-input normalization; unsorted time-series resampling; CSV/Parquet ETL; Bag loading and foldby; nested worker tasks; scatter; actors; queues/events/variables; plugin registration; diagnostic HTML; Dask-ML scaling and Xarray; guarded process scheduling and a process worker.

The numerical checks compare against NumPy, pandas, or SciPy. Use tolerance-based comparisons for floating reductions because partition order can change rounding. Check units, missing-data policy, random seeds, coordinate order, and output dtypes before scaling scientific data. Small fixtures establish recipe correctness, not large-data throughput, peak memory, fault tolerance, or cluster capacity.

Local clients used one or two threads with dashboards disabled; one guarded subprocess used the processes scheduler and a separate worker process. A local performance-report HTML file was generated; no browser/dashboard interaction or externally exposed service was tested. Bind personal dashboards to loopback (e.g. dashboard_address='127.0.0.1:8787') and use the deployment's authenticated access/TLS policy for remote schedulers. A scheduler address is not an authenticated public web API.

Cloud S3/GCS paths, SLURM, Kubernetes creation/scaling, production adaptive scaling, external database writes, and hardware-scale workloads are illustrative and were not executed. The skill implements no REST API client, authentication flow, or pagination. Graphviz rendering is optional and needs both its Python package and system executable.

Raw lines 67 to 93

94## Official sources

96- [Release metadata](https://pypi.org/pypi/dask/json) and [installation requirements](https://docs.dask.org/en/stable/install.html)

97- [Array API](https://docs.dask.org/en/stable/array-api.html), [array creation](https://docs.dask.org/en/stable/array-creation.html), [map_blocks](https://docs.dask.org/en/stable/generated/dask.array.map_blocks.html)

98- [Overlap](https://docs.dask.org/en/stable/array-overlap.html), [SVD restrictions](https://docs.dask.org/en/stable/generated/dask.array.linalg.svd.html), [FFT axes](https://docs.dask.org/en/stable/generated/dask.array.fft.fft2.html)

99- [Gaussian radius and dtype behavior](https://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.gaussian_filter.html), [chunking](https://docs.dask.org/en/stable/array-chunks.html)

100- [Zarr writer](https://docs.dask.org/en/stable/generated/dask.array.to_zarr.html)

101- [DataFrame API](https://docs.dask.org/en/stable/dataframe-api.html), [set_index](https://docs.dask.org/en/stable/generated/dask.dataframe.DataFrame.set_index.html), [map_partitions](https://docs.dask.org/en/stable/generated/dask.dataframe.DataFrame.map_partitions.html)

102- [CSV reader](https://docs.dask.org/en/stable/generated/dask.dataframe.read_csv.html), [Parquet writer](https://docs.dask.org/en/stable/generated/dask.dataframe.to_parquet.html), [DataFrame best practices](https://docs.dask.org/en/stable/dataframe-best-practices.html)

103- [Bag API](https://docs.dask.org/en/stable/bag-api.html), [Bag creation](https://docs.dask.org/en/stable/bag-creation.html), [foldby](https://docs.dask.org/en/stable/generated/dask.bag.Bag.foldby.html)

104- [Distributed API](https://distributed.dask.org/en/stable/api.html), [nested tasks](https://distributed.dask.org/en/stable/task-launch.html), [scheduling](https://docs.dask.org/en/stable/scheduling.html), [TLS](https://distributed.dask.org/en/stable/tls.html)

105- [Kubernetes operator client](https://kubernetes.dask.org/en/stable/operator_kubecluster.html), [SLURMCluster](https://jobqueue.dask.org/en/latest/generated/dask_jobqueue.SLURMCluster.html)

106- [StandardScaler](https://ml.dask.org/modules/generated/dask_ml.preprocessing.StandardScaler.html), [Xarray DataArray](https://docs.xarray.dev/en/stable/generated/xarray.DataArray.html)