Data API

KonfAI datasets are built from group definitions, transforms, augmentations, and optional patching strategies.

Dataset configuration objects

class konfai.data.data_manager.DataTrain(dataset_filenames=['default|./Dataset:mha'], groups_src={'default|Labels': {'default|Labels': {'transforms': [], 'patch_transforms': []}}}, augmentations={'DataAugmentation_0': <konfai.data.augmentation.DataAugmentationsList object>}, inline_augmentations=False, patch=<konfai.data.patching.DatasetPatch object>, use_cache=True, memory_budget=None, subset=<konfai.data.data_manager.TrainSubset object>, batch_size=1, validation=0.2, validation_augmentations=True, num_workers=None, pin_memory=False, prefetch_factor=None, persistent_workers=None)

Bases: Data

Dataset configuration used by the training workflow.

class konfai.data.data_manager.DataPrediction(dataset_filenames=['default|./Dataset'], groups_src={'default': {'default|Labels': {'transforms': [], 'patch_transforms': []}}}, augmentations={'DataAugmentation_0': <konfai.data.augmentation.DataAugmentationsList object>}, patch=<konfai.data.patching.DatasetPatch object>, memory_budget=None, subset=<konfai.data.data_manager.PredictionSubset object>, batch_size=1, num_workers=None, pin_memory=False, prefetch_factor=None, persistent_workers=None)

Bases: Data

Dataset configuration used by the prediction workflow.

class konfai.data.data_manager.DataMetric(dataset_filenames=['default|./Dataset:mha'], groups_src={'default': {'default|group_dest': {'transforms': [], 'patch_transforms': []}}}, memory_budget=None, subset=<konfai.data.data_manager.PredictionSubset object>, validation=None, num_workers=None, pin_memory=False, prefetch_factor=None, persistent_workers=None)

Bases: Data

Dataset configuration used by the evaluation workflow.

class konfai.data.data_manager.DatasetIter(rank, data, mapping, groups_src, inline_augmentations, data_augmentations_list, patch_size, overlap, buffer_size, apply_augmentations=True, use_cache=True)

Bases: Dataset

Torch dataset view over KonfAI dataset managers and patch mappings.

Patching

class konfai.data.patching.DatasetPatch(patch_size=[128, 128, 128], overlap=None, pad_value=None, extend_slice=0)

Bases: Patch

Patch definition applied when sampling data from datasets.

class konfai.data.patching.ModelPatch(patch_size=[128, 128, 128], overlap=None, patch_combine=None, pad_value=None, extend_slice=0)

Bases: Patch

Patch definition applied inside model graphs during prediction or training.

class konfai.data.patching.DatasetManager(index, group_src, group_dest, name, dataset, patch, transforms, data_augmentations_list)

Bases: object

Cache-backed manager for one dataset case and one source/destination group.

class konfai.data.patching.Accumulator(patch_slices, patch_size, patch_combine=None, batch=True)

Bases: object

Accumulate patch predictions and reassemble them into a full tensor.

is_empty()

True until the first patch has been blended in (no volume-sized buffer allocated yet).

Return type:

bool

Transforms and metadata

class konfai.data.transform.Transform

Bases: NeedDevice, ABC

Base class for transforms operating on tensors and cached attributes.

patch_locality(cache_attribute)

Declare how this transform’s output depends on its input, for patch streaming.

Answered from the transform’s own __init__ config and, where the honest answer depends on the image, from cache_attribute – the case’s SOURCE metadata, as the volume is stored. The dispatcher reads the header before any voxel, so a transform whose contract the image decides (a reorientation that is only a flip when the direction cosines are axis-aligned, a resample whose halo is the case’s own scale) can still declare it up front.

The default WHOLE_VOLUME is the safety net: any transform (including third-party custom ones) that does not override this falls to the whole-volume path, so nothing silently breaks.

An override is bound by three rules:

• READ-ONLY. Never write to cache_attribute. A declaration is made once, for the whole case, and what it wrote would be one patch’s answer imposed on every other – the first-patch-wins bug the streamed paths are built to avoid. The dispatcher hands over a private copy, so a write cannot reach the case; it is simply lost.

• NO I/O. Read the attribute already in hand, nothing else. Whether the outside world can honour the declaration (are the disk statistics readable, does a mask group exist) is the dispatcher’s call, and it already makes it.

• TOTAL. Answer for ANY case. The metadata may be absent – the config-time checks probe with an empty Attribute, and a group carries only what its writer stored – so a missing key must return WHOLE_VOLUME, never raise.

Return type:

PatchLocality

stream_region_source(target_slices, source_spatial_shape, cache_attribute)

Map a target-patch’s spatial slices to the source spatial region to read (region kinds).

Overridden by the kinds whose source region is an index remap of the target’s – ORIENTATION maps it and reorients what it reads, CROP maps it and is done. HALO and RESCALE are handled generically by the dispatcher, so the base raises for any transform that declares a region kind without providing the remap.

cache_attribute is the case’s SOURCE metadata, under the same rules as patch_locality(): a remap the image decides (a reorientation whose mirrored axes are the case’s own direction cosines) reads it here, and reads nothing else.

Return type:

list[slice]

write_stream_cache_attribute(cache_attribute, source_spatial_shape)

Record the geometry a whole-volume __call__ would, given the FULL source shape.

Called once per case, on the persistent attribute, for the stage that owns a streamed region. A transform whose geometry rewrite depends on the volume’s EXTENT (a reorientation’s new origin is the corner it mirrors onto) cannot compute it from a patch, which is all its __call__ is handed while streaming: it writes the case-level answer here instead, and the patch-local one it wrote on the way is dropped rather than persisted. The base is a no-op – a transform that leaves geometry alone has nothing to record.

Return type:

None

class konfai.data.transform.TransformInverse(inverse)

Bases: Transform, ABC

Base class for transforms that can also invert their effect.

class konfai.data.transform.TransformLoader

Bases: object

Resolve and instantiate transform classes from KonfAI configuration.

class konfai.data.transform.Clip(min_value=-1024, max_value=1024, save_clip_min=False, save_clip_max=False, mask=None)

Bases: Transform

Clip tensor intensities to a fixed or data-dependent value range.

patch_locality(cache_attribute)

Declare how this transform’s output depends on its input, for patch streaming.

Answered from the transform’s own __init__ config and, where the honest answer depends on the image, from cache_attribute – the case’s SOURCE metadata, as the volume is stored. The dispatcher reads the header before any voxel, so a transform whose contract the image decides (a reorientation that is only a flip when the direction cosines are axis-aligned, a resample whose halo is the case’s own scale) can still declare it up front.

The default WHOLE_VOLUME is the safety net: any transform (including third-party custom ones) that does not override this falls to the whole-volume path, so nothing silently breaks.

An override is bound by three rules:

• READ-ONLY. Never write to cache_attribute. A declaration is made once, for the whole case, and what it wrote would be one patch’s answer imposed on every other – the first-patch-wins bug the streamed paths are built to avoid. The dispatcher hands over a private copy, so a write cannot reach the case; it is simply lost.

• NO I/O. Read the attribute already in hand, nothing else. Whether the outside world can honour the declaration (are the disk statistics readable, does a mask group exist) is the dispatcher’s call, and it already makes it.

• TOTAL. Answer for ANY case. The metadata may be absent – the config-time checks probe with an empty Attribute, and a group carries only what its writer stored – so a missing key must return WHOLE_VOLUME, never raise.

Return type:

PatchLocality

class konfai.data.transform.Normalize(lazy=False, channels=None, min_value=-1, max_value=1, inverse=True)

Bases: TransformInverse

Map intensities to a target min/max interval and optionally invert it.

patch_locality(cache_attribute)

Declare how this transform’s output depends on its input, for patch streaming.

Answered from the transform’s own __init__ config and, where the honest answer depends on the image, from cache_attribute – the case’s SOURCE metadata, as the volume is stored. The dispatcher reads the header before any voxel, so a transform whose contract the image decides (a reorientation that is only a flip when the direction cosines are axis-aligned, a resample whose halo is the case’s own scale) can still declare it up front.

The default WHOLE_VOLUME is the safety net: any transform (including third-party custom ones) that does not override this falls to the whole-volume path, so nothing silently breaks.

An override is bound by three rules:

• READ-ONLY. Never write to cache_attribute. A declaration is made once, for the whole case, and what it wrote would be one patch’s answer imposed on every other – the first-patch-wins bug the streamed paths are built to avoid. The dispatcher hands over a private copy, so a write cannot reach the case; it is simply lost.

• NO I/O. Read the attribute already in hand, nothing else. Whether the outside world can honour the declaration (are the disk statistics readable, does a mask group exist) is the dispatcher’s call, and it already makes it.

• TOTAL. Answer for ANY case. The metadata may be absent – the config-time checks probe with an empty Attribute, and a group carries only what its writer stored – so a missing key must return WHOLE_VOLUME, never raise.

Return type:

PatchLocality

class konfai.data.transform.Standardize(lazy=False, mean=None, std=None, mask=None, inverse=True)

Bases: TransformInverse

Standardize tensors using cached or computed mean and standard deviation.

patch_locality(cache_attribute)

Declare how this transform’s output depends on its input, for patch streaming.

Answered from the transform’s own __init__ config and, where the honest answer depends on the image, from cache_attribute – the case’s SOURCE metadata, as the volume is stored. The dispatcher reads the header before any voxel, so a transform whose contract the image decides (a reorientation that is only a flip when the direction cosines are axis-aligned, a resample whose halo is the case’s own scale) can still declare it up front.

The default WHOLE_VOLUME is the safety net: any transform (including third-party custom ones) that does not override this falls to the whole-volume path, so nothing silently breaks.

An override is bound by three rules:

• READ-ONLY. Never write to cache_attribute. A declaration is made once, for the whole case, and what it wrote would be one patch’s answer imposed on every other – the first-patch-wins bug the streamed paths are built to avoid. The dispatcher hands over a private copy, so a write cannot reach the case; it is simply lost.

• NO I/O. Read the attribute already in hand, nothing else. Whether the outside world can honour the declaration (are the disk statistics readable, does a mask group exist) is the dispatcher’s call, and it already makes it.

• TOTAL. Answer for ANY case. The metadata may be absent – the config-time checks probe with an empty Attribute, and a group carries only what its writer stored – so a missing key must return WHOLE_VOLUME, never raise.

Return type:

PatchLocality

Dataset utilities

class konfai.utils.dataset.Attribute(attributes=None)

Bases: dict[str, Any]

Metadata container storing repeated values with a stack-like naming scheme.

pop(key, default=None)

If the key is not found, return the default if given; otherwise, raise a KeyError.

Return type:

Any

class konfai.utils.dataset.Dataset(filename, file_format)

Bases: object

Filesystem or HDF5-backed dataset abstraction used across KonfAI.

class AbstractFile

Bases: ABC

class H5File(filename, read)

Bases: AbstractFile

class SitkFile(filename, read, file_format)

Bases: AbstractFile

class OmeZarrFile(filename, read, level=0)

Bases: AbstractFile

OME-NGFF backend using chunked Zarr reads for KonfAI patches.

level selects the multiscale pyramid resolution to read (0 = full resolution, higher = coarser); it comes from the omezarr@<level> dataset-spec suffix.

class DicomFile(filename, read)

Bases: AbstractFile

DICOM series backend with header-only metadata and slice-level reads.

konfai.utils.dataset.data_to_image(data, attributes)

Convert a NumPy array and KonfAI attributes into a SimpleITK image.

Return type:

Image

konfai.utils.dataset.image_to_data(image)

Convert a SimpleITK image into a channel-first NumPy array and attributes.

Return type:

tuple[ndarray, Attribute]

konfai.utils.dataset.get_infos(filename)

Read shape and metadata from an image file without loading its full pixel data.

Return type:

tuple[list[int], Attribute]

See also

• Datasets and groups

• Segmentation Example

• Synthesis Example