基于半监督学习的多源异构数据治理

Multi-source Heterogeneous Data Governance Based on Semi-supervised Learning

摘要:

为实现不同数据管理系统之间的互通，提出一种基于半监督学习算法的多源异构数据治理框架，并由此设计、实现和测试了一套非结构化数据与结构化数据的自动化对齐方法。利用命名实体识别（NER）技术，将非结构化数据转化为结构化数据，再分别利用基于字符串相似度的方法和基于监督学习的方法，对结构化数据进行模式匹配；通过半监督学习方法，在结构化数据与数据库记录实体之间进行实体匹配与融合；利用自然语言处理（NLP）技术及深度学习方法，对融合后的数据集进行缺失值填补。结果表明：在论文数据集和视频元数据集上进行对齐处理后，两者的F1值分别达到89.70%及96.50%；在不同属性上进行缺失值填补后，整体填补准确率达到78%以上，大大优于基线方法的准确率。

Abstract:

In order to realize the intercommunication between different data management systems， we proposed a framework of multi-source heterogeneous data governance based on semi-supervised learning. Then， we designed， implemented and tested an automatic alignment method of unstructured data and structured data. The named entity recognition （NER） technology was firstly employed in the framework to convert the unstructured data into the structured one， and the string-similarity-based method and supervised-learning-based method were respectively used for the schema matching of structured data. With the semi-supervised learning method， the structured data and its corresponding entity in database were matched and integrated. Finally， natural language processing （NLP） technology and deep learning methods were used to impute missing values in the integrated dataset. It is shown that the F1-scores for the alignment on the paper dataset and video metadata set are 89.70% and 96.50%， respectively； and that the accuracy of missing value imputation on different attributes is all above 78%， which is a great improvement compared with the baseline methods.