Multi-source Heterogeneous Data Governance Based on Semi-supervised Learning

Abstract:

In order to realize the intercommunication between different data management systems， we proposed a framework of multi-source heterogeneous data governance based on semi-supervised learning. Then， we designed， implemented and tested an automatic alignment method of unstructured data and structured data. The named entity recognition （NER） technology was firstly employed in the framework to convert the unstructured data into the structured one， and the string-similarity-based method and supervised-learning-based method were respectively used for the schema matching of structured data. With the semi-supervised learning method， the structured data and its corresponding entity in database were matched and integrated. Finally， natural language processing （NLP） technology and deep learning methods were used to impute missing values in the integrated dataset. It is shown that the F1-scores for the alignment on the paper dataset and video metadata set are 89.70% and 96.50%， respectively； and that the accuracy of missing value imputation on different attributes is all above 78%， which is a great improvement compared with the baseline methods.