In analyzing the geological processes, the segmentation of scanning electron microscopy (SEM) images of geological samples is crucial but time-consuming because it needs to distinguish the boundaries for various mineral objects in natural rocks. To automate the segmentation, prior research has adopted supervised learning approaches that train convolutional neural networks (CNNs) using datasets consisting of images and labels. Although supervised learning techniques can produce a high accuracy for a substantial amount of data, the label preparation process is expensive and prone to mistakes because human experts must annotate millions of pixels for each image. To lessen the needs for labeling, in this work we investigated both unsupervised and semi-supervised approaches for fine-grained shale by developing a semi-supervised learning model, SU-Net, based on the U-Net architecture. We also proposed a novel algorithm to speed up the semi-supervised loss function in SU-Net using caching, skip zeros, and batching optimizations. Evaluation results demonstrate that SU-Net can achieve a higher accuracy than U-Net in the case of few labeled data. In addition, SU-Net can be trained as fast as U-Net and is 33 × faster than the state-of-the-art unsupervised model. Overall, SU-Net can benefit the geoscience community by demonstrating its utilities that shale images can be segmented with similar quality even when only limited labels are available.