Extract, transform, and load (ETL) processes organized as workflows play an important role in the future data integration for cloud services. ETL designers/administrators need testing data set that is aware of semantics of ETL workflow workloads to evaluate their developed ETL systems. Populating testing ETL systems with meaningful workload data is a difficult task. In this paper, we propose a semantic-aware data generator for ETL workflows. With given ETL workflow models and workload characterizations, the generator is able to generate synthetic data that capture the semantics of ETL activities. This is carried out by a three-staged approach. First, we derive expected cardinalities of all the source, intermediate, and target data sets involved in the ETL workflow model with some user-specified cardinality requirements. Then, with the concept of symbolic test, symbolic data instead of concrete data involved in ETL activities are generated, and semantics of the ETL workflow models are transformed to various constraints over these symbols. At last, concrete data are derived on the basis of resolving constraints. Our generator may facilitate ETL workload test case generation for ETL toolkit performance and function evaluations as well as ETL workflow solution benchmarking. Copyright © 2013 John Wiley & Sons, Ltd.

References

1R. Kimball, and J. Caserta, The Data Warehouse ETL Toolkit: Practical Techniques for Extracting, Cleaning, Conforming, and Delivering Data. Wiley, 2004.
Google Scholar
2J. Buggert, L. Wyatt, B. Caufield, and D. Pol. (2009, 10 March 2011). Preview of TPC-ETL: A Benchmark Under Development. Available: www.tpc.org/tpctc2009/tpctc2009-14.pdf
Google Scholar
3A. Simitsis, P. Vassiliadis, U. Dayal, A. Karagiannis, and V. Tziovara, Benchmarking ETL workflows, Performance Evaluation and Benchmarking, pp. 199–220, 2009.
Google Scholar
4M. T. Ozsu and P. Valduriez, Principles of Distributed Database Systems: Springer-Verlag New York Inc, 2011.
Google Scholar
5M. Poess and J. M. Stephens Jr, Generating thousand benchmark queries in seconds, in VLDB, 2004, pp. 1045–1053.
Google Scholar
6P. Vassiliadis, A. Simitsis, P. Georgantas, and M. Terrovitis, A Framework for the design of ETL scenarios, in CAiSE, 2003, pp. 1031–1031.
Google Scholar
7C. Olston, S. Chopra, and U. Srivastava, Generating example data for dataflow programs, in SIGMOD, 2009, pp. 245–256.
Google Scholar
8C. Binnig, D. Kossmann, E. Lo, and M. T. Özsu, QAGen: generating query-aware test databases, in SIGMOD, 2007, pp. 341–352.
Google Scholar
9C. Mishra, Query data and statistics: pick two, Ph.D., University of Toronto, 2010.
Google Scholar
10J. C. King, Symbolic execution and program testing, Communications of the ACM, vol. 19, pp. 385–394, 1976.
10.1145/360248.360252
Web of Science® Google Scholar
11S. Abdul Khalek and S. Khurshid, Automated sql query generation for systematic testing of database engines, in IEEE/ACM International Conference on Automated Software Engineering, 2010, pp. 329–332.
Google Scholar
12X. Y. Naiqiao Du, Jianmin Wang, SAGen: a semantic aware data generator for ETL workflow, presented at the Internetware, Guilin, Guangxi, 2011.
Google Scholar
13A. Arasu, R. Kaushik, and J. Li, Data generation using declarative constraints, in SIGMOD, 2011, pp. 685–696.
Google Scholar
14T. Syrjänen, Logic programming with cardinality constraints, Ph.D., Helsinki University of Technology, Espoo, Finland, 2003.
Google Scholar
15E. Lo, N. Cheng, and W. K. Hon, Generating databases for query workloads, Proceedings of the VLDB Endowment, vol. 3, pp. 848–859, 2010.
Google Scholar
16M. Bozkurt, M. Harman, and Y. Hassoun, Testing web services: a survey, Department of Computer Science, King's College London, Tech. Rep. TR-10-01, 2010.
Google Scholar
17H. M. Rusli, M. Puteh, S. Ibrahim, and S. G. H. Tabatabaei, A comparative evaluation of state-of-the-art web service composition testing approaches, in 6th International Workshop on Automation of Software Test, AST, 2011, p. 29.
Google Scholar
18P. Bernstein, J. Madhavan, and E. Rahm, Generic schema matching, ten years later, in VLDB Endowment, 2011, pp. 1195–1206.
Google Scholar

Volume28, Issue4

Combined Special issues on Cloud computing for data‐driven science and engineering (ScienceCloud 2012), Recent advances in security and privacy in large‐scale networks (LSNS 2014) and Intelligent Distributed Computing (IDC 2014)

25 March 2016

Pages 1016-1040

A semantic-aware data generator for ETL workflows

Summary

References

References

Information

About Wiley Online Library

Help & Support

Opportunities

Connect with Wiley

A semantic-aware data generator for ETL workflows

Summary

References

References

Related

Information