Cattle identification is crucial to be registered for breeding association, food quality tracing, disease prevention and control and fake insurance claims. Traditional non-biometrics methods for cattle identification is not really satisfactory in providing reliability due to theft, fraud, and duplication. In this study, a computer vision technique was proposed to facilitate precision animal management and improve livestock welfare. This paper presents a novel face identification framework by integrating light-weight RetinaFace-mobilenet with Additive Angular Margin Loss (ArcFace), namely CattleFaceNet. RetinaFace-mobilenet is designed for face detection and location, and ArcFace is adopted to strengthen the within-class compactness and also between-class discrepancy during training. Experiments on real-word scenarios dataset prove that RetinaFace-mobilenet achieves superior detection performance and significantly accelerates the computation time against RetinaNet. Three loss functions utilized in human face recognition combined with RetinaFace-mobilenet are compared and results indict that the proposed CattleFaceNet outperforms others with identification accuracy of 91.3% and processing time of 24 frames per second (FPS). This research work demonstrates the potential candidate of CattleFaceNet for livestock identification in real time in practical production scenarios.