See our paper here: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0263656 See the kickstart video on YouTube here: https://youtu.be/1oV-V2v-JPY Deep learning increasingly accelerates biomedical research, deploying neural networks in a wide range of different tasks, not least semantic segmentation. Neural networks are commonly trained supervised on large-scale, labeled datasets. These prerequisites raise issues in biomedical image recognition, as datasets are in general small-scale, challenging to obtain, expensive to label, and frequently heterogeneously labeled. Heterogeneous labels are a challenge for supervised methods. If not all classes are labeled for an individual sample, supervised deep learning approaches can only learn on a subset of the dataset with shared labels for each sample; consequently, biomedical image recognition engineers need to be frugal concerning their label and ground truth requirements. This paper discusses the effects of frugal labeling and proposes to train neural networks for multi-class semantic segmentation on heterogeneously labeled data based on a novel objective function. The objective function combines a class asymmetric loss with the Dice loss. The approach is demonstrated for training on the sparse ground truth of a heterogeneous labeled dataset, training within a transfer learning setting, and the use-case of merging multiple heterogeneously labeled datasets. For this purpose, a biomedical small-scale, multi-class semantic segmentation dataset based on the biologic model system (medaka fish) cardiac system is provided. Our approach and analysis show competitive results in supervised training regimes and encourage frugal labeling within biomedical image recognition. ![Visualization of the segmentation of the bulbus, ventricle and atrium][1] [1]: https://osf.io/gp9nu/?direct&mode=render&action=download **Segmentation of the bulbus, ventricle and atrium.**