Deep eye-part segmentation can overfit easily since large scale and precise annotated dataset is not always available. One might question if we can provide real world concept based semantic knowledge to our model to aid the segmentation learning. We present our research progress on using batch-based deep feature factorization to provide concept oriented ground-truth for model pre-training in eye-part segmentation. We examine our method on the CASIA-distance dataset and the results initially demonstrate the effectiveness. Further work on using different datasets and adopting a more exible factorization technique are required.