Manual dataset labeling is always at the expense of low efficiency and high cost in the deep learning model training of crack detection. The adaptability of existing automatic labeling algorithms to complex environment is weak as well. Aiming at these problems, the concept of fast labeling with low supervision is proposed based on computer vision and the probability statistics theory. In addition, a novel fast labeling algorithm for crack dataset is formed, composed of computer labeling and manual labeling. Compared with manual pixel-by-pixel labeling, the labeling accuracy of the proposed automatic method is more than 84%, and can save at least 85% of the time. Compared with traditional computer labeling, complex background can be better dealt with by labeling interference and simple manual labeling. Validated by U-Net deep learning model, the average intersection ratio of test sets can reach 0.90.