Data API

KonfAI datasets are built from group definitions, transforms, augmentations, and optional patching strategies.

Dataset configuration objects

class konfai.data.data_manager.DataTrain(dataset_filenames=['default|./Dataset:mha'], groups_src={'default|Labels': {'default|Labels': {'transforms': [], 'patch_transforms': []}}}, augmentations={'DataAugmentation_0': <konfai.data.augmentation.DataAugmentationsList object>}, inline_augmentations=False, patch=<konfai.data.patching.DatasetPatch object>, memory_budget=None, subset=<konfai.data.data_manager.TrainSubset object>, batch_size=1, validation=0.2, validation_augmentations=True, num_workers=None, pin_memory=False, prefetch_factor=None, persistent_workers=None)

Bases: Data

Dataset configuration used by the training workflow.

class konfai.data.data_manager.DataPrediction(dataset_filenames=['default|./Dataset'], groups_src={'default': {'default|Labels': {'transforms': [], 'patch_transforms': []}}}, augmentations={'DataAugmentation_0': <konfai.data.augmentation.DataAugmentationsList object>}, patch=<konfai.data.patching.DatasetPatch object>, memory_budget=None, subset=<konfai.data.data_manager.PredictionSubset object>, batch_size=1, num_workers=None, pin_memory=False, prefetch_factor=None, persistent_workers=None)

Bases: Data

Dataset configuration used by the prediction workflow.

class konfai.data.data_manager.DataMetric(dataset_filenames=['default|./Dataset:mha'], groups_src={'default': {'default|group_dest': {'transforms': [], 'patch_transforms': []}}}, memory_budget=None, subset=<konfai.data.data_manager.PredictionSubset object>, validation=None, num_workers=None, pin_memory=False, prefetch_factor=None, persistent_workers=None)

Bases: Data

Dataset configuration used by the evaluation workflow.

Evaluation never exposes a patch: each run sizes its own from memory_budget (a missing key means "auto") – a case that fits the budget is evaluated whole (exact); one that does not is cut into the largest DISJOINT patches that fit (overlap 0, no padding) and the reducible metrics combine their running partials into the exact whole-case value. The evaluator disables this sizing when any of its metrics is not reducible, so a metric that needs the whole volume always gets it.

prepare()

Instantiate config-driven transforms and augmentations before runtime.

Return type:

None

class konfai.data.data_manager.DatasetIter(rank, data, mapping, groups_src, inline_augmentations, data_augmentations_list, patch_size, overlap, buffer_size, apply_augmentations=True, use_cache=True)

Bases: Dataset

Torch dataset view over KonfAI dataset managers and patch mappings.

Patching

class konfai.data.patching.DatasetPatch(patch_size=[128, 128, 128], overlap=None, pad_value=None, extend_slice=0)

Bases: Patch

Patch definition applied when sampling data from datasets.

class konfai.data.patching.ModelPatch(patch_size=[128, 128, 128], overlap=None, patch_combine=None, pad_value=None, extend_slice=0)

Bases: Patch

Patch definition applied inside model graphs during prediction or training.

class konfai.data.patching.DatasetManager(index, group_src, group_dest, name, dataset, patch, transforms, data_augmentations_list)

Bases: object

Cache-backed manager for one dataset case and one source/destination group.

class konfai.data.patching.Accumulator(patch_slices, patch_size, patch_combine=None, batch=True)

Bases: object

Accumulate patch predictions and reassemble them into a full tensor.

add_layer(index, layer)

Blend one patch in; returns the slabs this completes (none for the whole-volume base).

Return type:

list[tuple[slice, Tensor]]

is_empty()

True until the first patch has been blended in (no volume-sized buffer allocated yet).

Return type:

bool

property footprint_shape: list[int]

Spatial shape held in memory at once — the whole volume for the base accumulator (overridden by StreamingAccumulator, which keeps only a window). Used to size the blend device.

Transforms and metadata

class konfai.data.transform.Transform

Bases: NeedDevice, ABC

Base class for transforms operating on tensors and cached attributes.

patch_locality(cache_attribute)

Declare how this transform’s output depends on its input, for patch streaming.

Answered from the transform’s own __init__ config and, where the honest answer depends on the image, from cache_attribute – the case’s SOURCE metadata, as the volume is stored. The dispatcher reads the header before any voxel, so a transform whose contract the image decides (a reorientation that is only a flip when the direction cosines are axis-aligned, a resample whose halo is the case’s own scale) can still declare it up front.

The default WHOLE_VOLUME is the safety net: any transform (including third-party custom ones) that does not override this falls to the whole-volume path, so nothing silently breaks.

An override is bound by three rules:

• READ-ONLY. Never write to cache_attribute. A declaration is made once, for the whole case, and what it wrote would be one patch’s answer imposed on every other – the first-patch-wins bug the streamed paths are built to avoid. The dispatcher hands over a private copy, so a write cannot reach the case; it is simply lost.

• NO I/O. Read the attribute already in hand, nothing else. Whether the outside world can honour the declaration (are the disk statistics readable, does a mask group exist) is the dispatcher’s call, and it already makes it.

• TOTAL. Answer for ANY case. The metadata may be absent – the config-time checks probe with an empty Attribute, and a group carries only what its writer stored – so a missing key must return WHOLE_VOLUME, never raise.

Return type:

PatchLocality

stream_region_source(target_slices, source_spatial_shape, cache_attribute)

Map a target-patch’s spatial slices to the source spatial region to read (region kinds).

Overridden by the kinds whose source region is an index remap of the target’s – ORIENTATION maps it and reorients what it reads, CROP maps it and is done. HALO and RESCALE are handled generically by the dispatcher, so the base raises for any transform that declares a region kind without providing the remap.

cache_attribute is the case’s SOURCE metadata, under the same rules as patch_locality(): a remap the image decides (a reorientation whose mirrored axes are the case’s own direction cosines) reads it here, and reads nothing else.

Return type:

list[slice]

stream_slab(name, tensor, region, spatial_shape, cache_attribute)

Run this transform on one finalized slab — rows region of a spatial_shape volume.

The streamed-write dispatcher calls this instead of __call__ for a SLAB declaration: the value map is per-voxel, so the default whole-volume call is exact on the slab, but the stage’s side effect needs to know where the slab sits — which is what a SLAB transform overrides this to read. Slabs arrive in order and tile the volume exactly once per case.

Return type:

Tensor

stream_abort(name)

Drop whatever stream_slab holds open for name after a mid-case failure.

Called by the streamed-write dispatcher when a case dies between slabs, so a SLAB stage’s region sink or buffer does not outlive the case. The base holds nothing.

Return type:

None

write_stream_cache_attribute(cache_attribute, source_spatial_shape)

Record the geometry a whole-volume __call__ would, given the FULL source shape.

Called once per case, on the persistent attribute, for the stage that owns a streamed region. A transform whose geometry rewrite depends on the volume’s EXTENT (a reorientation’s new origin is the corner it mirrors onto) cannot compute it from a patch, which is all its __call__ is handed while streaming: it writes the case-level answer here instead, and the patch-local one it wrote on the way is dropped rather than persisted. The base is a no-op – a transform that leaves geometry alone has nothing to record.

Return type:

None

class konfai.data.transform.TransformInverse(inverse)

Bases: Transform, ABC

Base class for transforms that can also invert their effect.

inverse_patch_locality(cache_attribute)

Declare how inverse’s output depends on its input, for the streamed-write dispatcher.

The write mirror of patch_locality(): a prediction’s finalize chain applies transforms INVERTED, so the streamed-write gate asks each one about its inverse. cache_attribute is the finalize-time state — the case’s attribute as inverse will receive it, with everything the forward pass pushed still stacked on it — under the same three rules (read-only, no I/O, total).

The default derives from the forward contract where the derivation is safe for any subclass: a per-voxel value map inverts to a per-voxel value map, and an index remap inverts to an index remap. Every other kind falls to WHOLE_VOLUME — an inverse that is streamable anyway (Padding’s crop, Resample’s rescale) declares itself.

Return type:

PatchLocality

inverse_transform_shape(shape, cache_attribute)

The spatial shape inverse produces from shape (write mirror of transform_shape).

cache_attribute is the finalize-time state, as in inverse_patch_locality(). The default is the identity, exactly as (in)exact as transform_shape’s: a shape-changing inverse must override it, and the streamed-write dispatcher only trusts it for the kinds inverse_patch_locality() declared streamable.

Return type:

list[int]

stream_region_target(target_slices, source_spatial_shape, cache_attribute)

Map a region of inverse’s OUTPUT to the region of its INPUT it is computed from.

The write mirror of stream_region_source(), with the same direction of travel: the slices are in the space being produced (here the written image), the shape is the space being consumed (here the finalized accumulator), and the answer is the consumed region. The streamed-write dispatcher holds a sliding window of finalized slabs and emits each output region once the region this returns has arrived. cache_attribute is the finalize-time state; a transform whose remap is read from what its own inverse pops accounts for those pops on a copy.

Return type:

list[slice]

class konfai.data.transform.TransformLoader

Bases: object

Resolve and instantiate transform classes from KonfAI configuration.

class konfai.data.transform.Clip(min_value=-1024, max_value=1024, save_clip_min=False, save_clip_max=False, mask=None)

Bases: Transform

Clip tensor intensities to a fixed or data-dependent value range.

patch_locality(cache_attribute)

Declare how this transform’s output depends on its input, for patch streaming.

Answered from the transform’s own __init__ config and, where the honest answer depends on the image, from cache_attribute – the case’s SOURCE metadata, as the volume is stored. The dispatcher reads the header before any voxel, so a transform whose contract the image decides (a reorientation that is only a flip when the direction cosines are axis-aligned, a resample whose halo is the case’s own scale) can still declare it up front.

The default WHOLE_VOLUME is the safety net: any transform (including third-party custom ones) that does not override this falls to the whole-volume path, so nothing silently breaks.

An override is bound by three rules:

• READ-ONLY. Never write to cache_attribute. A declaration is made once, for the whole case, and what it wrote would be one patch’s answer imposed on every other – the first-patch-wins bug the streamed paths are built to avoid. The dispatcher hands over a private copy, so a write cannot reach the case; it is simply lost.

• NO I/O. Read the attribute already in hand, nothing else. Whether the outside world can honour the declaration (are the disk statistics readable, does a mask group exist) is the dispatcher’s call, and it already makes it.

• TOTAL. Answer for ANY case. The metadata may be absent – the config-time checks probe with an empty Attribute, and a group carries only what its writer stored – so a missing key must return WHOLE_VOLUME, never raise.

Return type:

PatchLocality

class konfai.data.transform.Normalize(lazy=False, channels=None, min_value=-1, max_value=1, inverse=True)

Bases: TransformInverse

Map intensities to a target min/max interval and optionally invert it.

patch_locality(cache_attribute)

Declare how this transform’s output depends on its input, for patch streaming.

Answered from the transform’s own __init__ config and, where the honest answer depends on the image, from cache_attribute – the case’s SOURCE metadata, as the volume is stored. The dispatcher reads the header before any voxel, so a transform whose contract the image decides (a reorientation that is only a flip when the direction cosines are axis-aligned, a resample whose halo is the case’s own scale) can still declare it up front.

The default WHOLE_VOLUME is the safety net: any transform (including third-party custom ones) that does not override this falls to the whole-volume path, so nothing silently breaks.

An override is bound by three rules:

• READ-ONLY. Never write to cache_attribute. A declaration is made once, for the whole case, and what it wrote would be one patch’s answer imposed on every other – the first-patch-wins bug the streamed paths are built to avoid. The dispatcher hands over a private copy, so a write cannot reach the case; it is simply lost.

• NO I/O. Read the attribute already in hand, nothing else. Whether the outside world can honour the declaration (are the disk statistics readable, does a mask group exist) is the dispatcher’s call, and it already makes it.

• TOTAL. Answer for ANY case. The metadata may be absent – the config-time checks probe with an empty Attribute, and a group carries only what its writer stored – so a missing key must return WHOLE_VOLUME, never raise.

Return type:

PatchLocality

inverse_patch_locality(cache_attribute)

Declare how inverse’s output depends on its input, for the streamed-write dispatcher.

The write mirror of patch_locality(): a prediction’s finalize chain applies transforms INVERTED, so the streamed-write gate asks each one about its inverse. cache_attribute is the finalize-time state — the case’s attribute as inverse will receive it, with everything the forward pass pushed still stacked on it — under the same three rules (read-only, no I/O, total).

The default derives from the forward contract where the derivation is safe for any subclass: a per-voxel value map inverts to a per-voxel value map, and an index remap inverts to an index remap. Every other kind falls to WHOLE_VOLUME — an inverse that is streamable anyway (Padding’s crop, Resample’s rescale) declares itself.

Return type:

PatchLocality

class konfai.data.transform.Standardize(lazy=False, mean=None, std=None, mask=None, inverse=True)

Bases: TransformInverse

Standardize tensors using cached or computed mean and standard deviation.

patch_locality(cache_attribute)

Declare how this transform’s output depends on its input, for patch streaming.

Answered from the transform’s own __init__ config and, where the honest answer depends on the image, from cache_attribute – the case’s SOURCE metadata, as the volume is stored. The dispatcher reads the header before any voxel, so a transform whose contract the image decides (a reorientation that is only a flip when the direction cosines are axis-aligned, a resample whose halo is the case’s own scale) can still declare it up front.

The default WHOLE_VOLUME is the safety net: any transform (including third-party custom ones) that does not override this falls to the whole-volume path, so nothing silently breaks.

An override is bound by three rules:

• READ-ONLY. Never write to cache_attribute. A declaration is made once, for the whole case, and what it wrote would be one patch’s answer imposed on every other – the first-patch-wins bug the streamed paths are built to avoid. The dispatcher hands over a private copy, so a write cannot reach the case; it is simply lost.

• NO I/O. Read the attribute already in hand, nothing else. Whether the outside world can honour the declaration (are the disk statistics readable, does a mask group exist) is the dispatcher’s call, and it already makes it.

• TOTAL. Answer for ANY case. The metadata may be absent – the config-time checks probe with an empty Attribute, and a group carries only what its writer stored – so a missing key must return WHOLE_VOLUME, never raise.

Return type:

PatchLocality

inverse_patch_locality(cache_attribute)

Declare how inverse’s output depends on its input, for the streamed-write dispatcher.

The write mirror of patch_locality(): a prediction’s finalize chain applies transforms INVERTED, so the streamed-write gate asks each one about its inverse. cache_attribute is the finalize-time state — the case’s attribute as inverse will receive it, with everything the forward pass pushed still stacked on it — under the same three rules (read-only, no I/O, total).

The default derives from the forward contract where the derivation is safe for any subclass: a per-voxel value map inverts to a per-voxel value map, and an index remap inverts to an index remap. Every other kind falls to WHOLE_VOLUME — an inverse that is streamable anyway (Padding’s crop, Resample’s rescale) declares itself.

Return type:

PatchLocality

Dataset utilities

class konfai.utils.dataset.Attribute(attributes=None)

Bases: dict[str, Any]

Metadata container storing repeated values with a stack-like naming scheme.

pop(key, default=None)

If the key is not found, return the default if given; otherwise, raise a KeyError.

Return type:

Any

class konfai.utils.dataset.Dataset(filename, file_format)

Bases: object

Filesystem or HDF5-backed dataset abstraction used across KonfAI.

class AbstractFile

Bases: ABC

open_data_stream(name, shape, dtype, attributes)

Open name for incremental region writes; None when this backend cannot.

Return type:

DataStream | None

class H5File(filename, read)

Bases: AbstractFile

open_data_stream(name, shape, dtype, attributes)

Open name for incremental region writes; None when this backend cannot.

Return type:

DataStream | None

class SitkFile(filename, read, file_format)

Bases: AbstractFile

open_data_stream(name, shape, dtype, attributes)

Open name for incremental region writes; None when this backend cannot.

Return type:

DataStream | None

class OmeZarrFile(filename, read, level=0)

Bases: AbstractFile

OME-NGFF backend using chunked Zarr reads for KonfAI patches.

level selects the multiscale pyramid resolution to read (0 = full resolution, higher = coarser); it comes from the omezarr@<level> dataset-spec suffix.

open_data_stream(name, shape, dtype, attributes)

Open name for incremental region writes; None when this backend cannot.

Return type:

DataStream | None

class DicomFile(filename, read)

Bases: AbstractFile

DICOM series backend with header-only metadata and slice-level reads.

rebase(prefix)

Prepend prefix to this dataset’s path, re-deriving is_directory from the format.

Return type:

None

concurrent_write_safe()

Whether writes to different entries land in disjoint files, so a background writer may flush one entry while another thread writes elsewhere in the dataset.

Mirrors the backend dispatch in File.__enter__: everything that is not a single-store backend is a SitkFile directory, one image file per (group, name). A single store (one HDF5 file, one zarr hierarchy, a DICOM series) shares handles and metadata across entries and must stay serial.

Return type:

bool

can_stream_data(attributes)

Whether open_data_stream can serve this dataset’s write format.

H5 and OME-Zarr always can; MetaImage mha needs image geometry to write its header up front; every other format only writes whole volumes (use write).

Return type:

bool

open_data_stream(group, name, shape, dtype, attributes=None)

Open one entry for incremental region writes.

Returns None when the write format cannot serve region writes; the caller then assembles the volume and uses write. The returned stream is a context manager: a clean exit finalizes the entry, an exception removes the partial one.

Return type:

DataStream | None

konfai.utils.dataset.data_to_image(data, attributes)

Convert a NumPy array and KonfAI attributes into a SimpleITK image.

Return type:

Image

konfai.utils.dataset.image_to_data(image)

Convert a SimpleITK image into a channel-first NumPy array and attributes.

Return type:

tuple[ndarray, Attribute]

konfai.utils.dataset.get_infos(filename)

Read shape and metadata from an image file without loading its full pixel data.

Return type:

tuple[list[int], Attribute]

See also

• Datasets and groups

• Segmentation Example

• Synthesis Example