vision_models_evaluation
Install
To install the library, just run:
pip install vision_models_evaluation
How to use
This library provides a method that can help you in the process of model evaluation. Using the scikit-learn validation techniques you can validate your deep learning models.
In order to validate your model, you will need to build and train various versions of it (for example, using a KFold validation, it is needed to build five different models).
For doing so, you need to provide: the DataBlock
hparams (hyperparameters), the DataLoader
hparams, the technique used to split the data, the Learner
construction hparams, the learning mode (whether to use a pretrained model or not: fit_one_cycle
or finetune
) and the Learner
training hparams. So, the first step is to define them all:
= {
db_hparams "blocks": (ImageBlock, MaskBlock(codes)),
"get_items": partial(get_image_files, folders=['train']),
"get_y": get_y_fn,
"item_tfms": [Resize((480,640)), TargetMaskConvertTransform(), transformPipeline],
"batch_tfms": Normalize.from_stats(*imagenet_stats)
}= {
dl_hparams "source": path_images,
"bs": 4
}= KFold(n_splits = 5)
technique = {
learner_hparams "arch": resnet18,
"pretrained": True,
"metrics": [DiceMulti()]
}= {
learning_hparams "epochs": 10,
"base_lr": 0.001,
"freeze_epochs": 1
}= "finetune" learning_mode
Then, you need to call the evaluate
method with those defined hparams. After the execution, the method will return a dictionary of results (for each metric used to test the model, the value obtained in each fold).
= evaluate(
r
db_hparams,
dl_hparams,
technique,
learner_hparams,
learning_hparams,
learning_mode )
Finally, you can plot the metrics using a boxplot from pandas, for example:
import pandas as pd
= pd.DataFrame(r)
df "DiceMulti");
df.boxplot(
print(
"DiceMulti"].mean(),
df["DiceMulti"].std()
df[ )
You can use this method to evaluate your model, but you can also use it to evaluate several models with distinct hparams: you can get the results for each of them and then plot the average of their metrics.