The Challenges of Data Quality and Data Quality Assessment in the Big Data Era Li Cai1,3 and Yangyong Zhu2

I

CODATA

S

U

Cai, L and Zhu, Y 2015 The Challenges of Data Quality and Data Quality

Assessment in the Big Data Era. Data Science Journal, 14: 2, pp. 1-10,

DOI: http://dx.doi.org/10.5334/dsj-2015-002

PROCEEDINGS PAPER

The Challenges of Data Quality and Data Quality

Assessment in the Big Data Era

Li Cai1,3 and Yangyong Zhu2

School of Computer and Science, Fudan University, No. 220, Han Dan Road, Shanghai, China

lcai@fudan.edu.cn

1

Shanghai Key Laboratory of Data Science, Fudan University, Shanghai, China

yyzhu@fudan.edu.cn

2

School of Software, Yunnan University, No. 2 North Road of Cui Hu, Kunming, China

3

High-quality data are the precondition for analyzing and using big data and for guaranteeing

the value of the data. Currently, comprehensive analysis and research of quality standards and

quality assessment methods for big data are lacking. First, this paper summarizes reviews of

data quality research. Second, this paper analyzes the data characteristics of the big data

environment, presents quality challenges faced by big data, and formulates a hierarchical data

quality framework from the perspective of data users. This framework consists of big data

quality dimensions, quality characteristics, and quality indexes. Finally, on the basis of this

framework, this paper constructs a dynamic assessment process for data quality. This process

has good expansibility and adaptability and can meet the needs of big data quality assessment.

The research results enrich the theoretical scope of big data and lay a solid foundation for the

future by establishing an assessment model and studying evaluation algorithms.

Keywords: Big data; Data quality; Quality assessment; Data science

1 Introduction

Many significant technological changes have occurred in the information technology industry since the

beginning of the 21st century, such as cloud computing, the Internet of Things, and social networking. The

development of...