Data Config
Data configuration.
Float = Annotated[float, PlainSerializer(np_float_to_scientific_str, return_type=str)]
module-attribute
Annotated float type, used to serialize floats to strings.
DataConfig
Bases: BaseModel
Data configuration.
If std is specified, mean must be specified as well. Note that setting the std first
and then the mean (if they were both None before) will raise a validation error.
Prefer instead set_mean_and_std to set both at once. Means and stds are expected
to be lists of floats, one for each channel. For supervised tasks, the mean and std
of the target could be different from the input data.
All supported transforms are defined in the SupportedTransform enum.
Examples:
Minimum example:
>>> data = DataConfig(
... data_type="array", # defined in SupportedData
... patch_size=[128, 128],
... batch_size=4,
... axes="YX"
... )
To change the image_means and image_stds of the data:
One can pass also a list of transformations, by keyword, using the SupportedTransform value:
>>> from careamics.config.support import SupportedTransform
>>> data = DataConfig(
... data_type="tiff",
... patch_size=[128, 128],
... batch_size=4,
... axes="YX",
... transforms=[
... {
... "name": "XYFlip",
... }
... ]
... )
axes
instance-attribute
Axes of the data, as defined in SupportedAxes.
batch_size = Field(default=1, ge=1, validate_default=True)
class-attribute
instance-attribute
Batch size for training.
data_type
instance-attribute
Type of input data, numpy.ndarray (array) or paths (tiff, czi, and custom), as defined in SupportedData.
image_means = Field(default=None, min_length=0, max_length=32)
class-attribute
instance-attribute
Means of the data across channels, used for normalization.
image_stds = Field(default=None, min_length=0, max_length=32)
class-attribute
instance-attribute
Standard deviations of the data across channels, used for normalization.
patch_size = Field(..., min_length=2, max_length=3)
class-attribute
instance-attribute
Patch size, as used during training.
target_means = Field(default=None, min_length=0, max_length=32)
class-attribute
instance-attribute
Means of the target data across channels, used for normalization.
target_stds = Field(default=None, min_length=0, max_length=32)
class-attribute
instance-attribute
Standard deviations of the target data across channels, used for normalization.
train_dataloader_params = Field(default={'shuffle': True}, validate_default=True)
class-attribute
instance-attribute
Dictionary of PyTorch training dataloader parameters. The dataloader parameters,
should include the shuffle key, which is set to True by default. We strongly
recommend to keep it as True to ensure the best training results.
transforms = Field(default=[XYFlipConfig(), XYRandomRotate90Config()], validate_default=True)
class-attribute
instance-attribute
List of transformations to apply to the data, available transforms are defined in SupportedTransform.
val_dataloader_params = Field(default={}, validate_default=True)
class-attribute
instance-attribute
Dictionary of PyTorch validation dataloader parameters.
__str__()
all_elements_power_of_2_minimum_8(patch_list)
classmethod
Validate patch size.
Patch size must be powers of 2 and minimum 8.
Parameters:
-
patch_list(list of int) –Patch size.
Returns:
-
list of int–Validated patch size.
Raises:
-
ValueError–If the patch size is smaller than 8.
-
ValueError–If the patch size is not a power of 2.
axes_valid(axes)
classmethod
Validate axes.
Axes must: - be a combination of 'STCZYX' - not contain duplicates - contain at least 2 contiguous axes: X and Y - contain at most 4 axes - not contain both S and T axes
Parameters:
-
axes(str) –Axes to validate.
Returns:
-
str–Validated axes.
Raises:
-
ValueError–If axes are not valid.
set_3D(axes, patch_size)
set_default_pin_memory(dataloader_params)
classmethod
Set default pin_memory for dataloader parameters if not provided.
- If 'pin_memory' is not set, it defaults to True if CUDA is available.
Parameters:
-
dataloader_params(dict of {str: Any}) –The dataloader parameters.
Returns:
-
dict of {str: Any}–The dataloader parameters with pin_memory default applied.
set_default_train_workers(dataloader_params)
classmethod
Set default num_workers for training dataloader if not provided.
- If 'num_workers' is not set, it defaults to the number of available CPU cores.
Parameters:
-
dataloader_params(dict of {str: Any}) –The training dataloader parameters.
Returns:
-
dict of {str: Any}–The dataloader parameters with num_workers default applied.
set_means_and_stds(image_means, image_stds, target_means=None, target_stds=None)
Set mean and standard deviation of the data across channels.
This method should be used instead setting the fields directly, as it would otherwise trigger a validation error.
Parameters:
-
image_means((ndarray, tuple or list)) –Mean values for normalization.
-
image_stds((ndarray, tuple or list)) –Standard deviation values for normalization.
-
target_means((ndarray, tuple or list), default:None) –Target mean values for normalization, by default ().
-
target_stds((ndarray, tuple or list), default:None) –Target standard deviation values for normalization, by default ().
set_val_workers_to_match_train()
Set validation dataloader num_workers to match training dataloader.
If num_workers is not specified in val_dataloader_params, it will be set to the same value as train_dataloader_params["num_workers"].
Returns:
-
Self–Validated data model with synchronized num_workers.
shuffle_train_dataloader(train_dataloader_params)
classmethod
Validate that "shuffle" is included in the training dataloader params.
A warning will be raised if shuffle=False.
Parameters:
-
train_dataloader_params(dict of {str: Any}) –The training dataloader parameters.
Returns:
-
dict of {str: Any}–The validated training dataloader parameters.
Raises:
-
ValueError–If "shuffle" is not included in the training dataloader params.
std_only_with_mean()
Check that mean and std are either both None, or both specified.
Returns:
-
Self–Validated data model.
Raises:
-
ValueError–If std is not None and mean is None.
validate_dimensions()
Validate 2D/3D dimensions between axes, patch size and transforms.
Returns:
-
Self–Validated data model.
Raises:
-
ValueError–If the transforms are not valid.
np_float_to_scientific_str(x)
Return a string scientific representation of a float.
In particular, this method is used to serialize floats to strings, allowing numpy.float32 to be passed in the Pydantic model and written to a yaml file as str.
Parameters:
-
x(float) –Input value.
Returns:
-
str–Scientific string representation of the input value.