Data curation in the Internet of Things: A decision model approach

1 INTRODUCTION

Internet of Things (IoT) is more realistic than ever. It is common to find hundreds and thousands of devices connected to the Internet and to each other in different contexts. Nowadays, IoT is used as a service¹ in big data pipelines² as a mechanism to integrate large amounts of data-centric services. As a consequence, the data generated by sensors in IoT contexts³ can easily reach the three dimensions of big data. In this respect, IoT must tackle multiple challenges in a pipeline of big data concerning the activities related to data acquisition or data curation. Nevertheless, the use of poor-quality data (i.e., data with problems) can produce terrible consequences,^4-6 for example, incorrect decision-making, damaging the performance in the operations, increasing costs. Therefore, to use and reach data with an expected/desired level of usability has become crucial to achieving success.⁷

In general, the integration of the IoT with big data is carried out in the data acquisition activity in the context of a big data pipeline.⁸ Thus, the IoT requires the integration of numerous devices through networks. In this context, the communication protocols are essential.⁹ Nevertheless, not only the communications are important since there exist a huge variability of sensors¹⁰ that can produce data in multiple formats and stored with different data schemas and data storage typologies. An example is the application of IoT in the context of the agri-food sector by transforming farms into smart farms¹¹ as shown in Figure 1. Therefore, on the one hand, we need to aggregate the heterogeneous data provided for sensors (i.e., data curation) but ensuring the data quality. And, on the other hand, sensors can be affected by external o internal problems by compromising the quality of the data produced,¹² for example, missing values due to power interruptions. Thereby, we need to evaluate the quality of the data in the process of data curation in order to avoid poor-quality data in afterwards tasks.

The rest of the article is organized as follows: Section 2 briefly presents the proposed big data pipeline for IoT scenarios. Section 3 briefly introduces the data quality approach integrated in the pipeline. Section 4 introduces a particular case study for the application of the approach as well as the decision models are explained for the case study. The related work is discussed in Section 5. Finally, the article is summarized, and the conclusions and future work are presented in Section 6.

2 IOT-BASED DATA PIPELINE APPROACH

A big data pipeline is a process composed of a set of activities whose objective is to extract value from data.⁸ Following the approach presented in References 2, 8, 13-15 we have defined the next IoT-based data framework depicted in Figure 2.

3 DMN4DQ IN A NUTSHELL

DMN4DQ¹⁸ is a methodology that enables the automatic generation of recommendations on the potential usability of data in terms of its level of data quality. As shown in Figure 3, DMN4DQ relies on establishing a hierarchy of business rules for data quality which enables the validation of data attributes, the measurement of data quality dimensions, and the assessment of the level of data quality. This hierarchy of business rules is supported by the decision model and notation paradigm (DMN).¹⁹

DMN is the modeling language and standard notation defined by OMG to describe decision rules. These rules take the form of the “if-then” structure of traditional programming languages. From the definition of a data model that is supported by a set of engines, for example, Camunda–DMN Engine, we see in this combination a possibility for the development of the study and the assessment of data quality.

The DMN4Spark source code can be downloaded for free at https://github.com/IDEA-Research-Group/dmn4spark.

A complete example of decision model and those business rules are instantiated or a particular use case in the next section.

4 DECISION MODEL FOR A CASE STUDY: IOT AGRICULTURAL FIELD

In our study, we propose to use our approach for IoT-big data pipeline in a real case study. First, we propose to choose a case study base on the real dataset in Reference 20. This dataset includes data extracted from several sensors (with different frequency) in an agricultural farm. Subsequently, we want to apply DMN4DQ for the dataset to evaluate the usability of the data in terms of the data quality to discard those that do not provide sufficient information and include in the final catalog the dataset that has obtained an adequate evaluation according to the data quality requirements, carried out by means of the defined decision rules. To do this, we have to define the decision model to be used.

The decision model (i.e., DMN model) proposed for the case study is shown in Figure 4. This decision model is focused on two quality dimensions completeness and accuracy. Thus, incompleteness and inaccuracy of the data could cause confusion, leading to incorrect or unrealistic values being entered into the system and leading to inappropriate decisions being made when further processing the collected data.

In the next subsections, the different parts of the DMN4DQ methodology are explained in details for our particular case study.

4.3 Identify data quality dimensions and define business rules for data values (BR.DV)

As previously mentioned in the presentation, the objective is to evaluate the completeness and accuracy. In this respect, we have grouped the data validation for each of the business rules BR01 to BR13 into the two quality dimensions: completeness and accuracy.

Completeness. Missing some relevant data from the dataset may lead to undesirable results. In this case, we define business rules BR01 to BR08. The first three rules: BR01, BR02, and BR03 are in charge of ensuring that the location, Date and Time fields contain data other than null, empty or blank; thereby for these three rules, the returned result will be false if the data field is null, empty or blank, and true otherwise. In the case of BR04 to BR08 rules refer to the sensors of the different depths (30, 60, 90, 120, and 150 cm). Taking the 30 cm depth sensor as an example, as shown in Figure 5, the BR04 rule takes humidity ($VW\_30$ cm) and temperature ($T\_30$ cm) as inputs so that when the inputs are null, the result will be 0 (i.e, S30 null); if either of the two values is other than null or contains the empty field or the string “NA”, the result will be 1 (i.e., S30 not available); in any other case, the result value will be 2 (S30 Data), which indicates that it contains data that can be accepted. In the list of rules, which are numbered, at this level of business rules, the first one that meets the imposed conditions, that is, the hit policy first (F) shall be triggered, not evaluating the following ones.

Accuracy. In this case, we are interested in detecting those cases in which the values collected by the sensors are not reliable due to the extreme values. BR09 to BR13 rules deal with this case for sensors located at different depths. For instance, the BR09 given in Figure 6 takes as input values the humidity ($VW\_30$ cm) and temperature ($T\_30$ cm) data in such a way that: (1) if the humidity is in the range of 0.150 to 0.700 and the temperature between 1 and 45 degrees the result will be “realistic”; (2) in the case where the temperature is below 1 degree or above 45, it will take as a result “unusual”; (3) in the case where the humidity is above 0.700 it will also give a result “unusual”; (4) in any other case not covered, the result will be “unrealistic”.

The results obtained from the data validation of the business rules, that is, BR01 to BR13, will be delivered to the higher level where the data quality measurement is performed.

4.4 Define business rules for data quality measurement (BR.DQM)

The result of the completeness dimension (BR.DQM.Completeness) measurement depends on the output values from BR01 to BR08 rules. Each line of the decision table will be triggered if the content of the fields BR01, BR02, and BR03 plus a value greater than or equal to 2 for the values coming from the validation of the data of the readings of each of the depth sensors BR04 to BR08. In this way, we will know how many readings are considered complete. These rules are evaluated following the hit policy C#, which returns the number of conditions that are met as given in Figure 7.

In relation to the measurement of the accuracy dimension (BR.DQM.Accuracy) the following conditions are defined: given the inputs BR09 to BR13, a value of 20 is assigned as result to each entry containing the string “realistic”, so that the hit policy C+ sums up the output values of each condition line that is fulfilled. This result gives a readable accuracy in terms of percentage, with the value of 100 being the fact of having valid readings in all the sensors coming from a register as given in Figure 8.

4.5 Define business rules for data quality assessment (BR.DQA)

For the data quality assessment, we have defined the decision table shown in Figure 9. The possible outputs of the data quality assessment are set as follows: (1) “suitable” for those records that have a completeness measure greater than or equal to 5 and accuracy greater than or equal to 100, it means that the record contains complete and accurate data from all sensors; (2) “enough quality” when there is at least a value equal to or greater than 3 in the completeness dimension and greater than or equal to 60 in the accuracy dimension, it means that 3 or 4 sensors have provided complete and accurate readings; (3) “bad quality” result is given by complete readings from one or two sensors or with a precision below 60, and; (4) “nonusable” in any other case. In this decision table, the hit policy first (F) are applied, thus the first entry that match is returned, disregarding the rest.

4.6 Define business rules for the usability of data (BR.DUD)

In this final step, the quality level of the data is decided in consequence we can decide to include or not the data records in the final catalog. Figure 10 shows the decision table for BR.DUD. In our case, we have defined as usable data those that have reported as a result of the data quality assessment (BR.DQA) values of “suitable” or “enough quality”, returning a result value of “use”. The rest are discarded as not exceeding the preset minimum, labeled as “do not use".

This decision model enables us to powerful decision-making based on usability by means of measuring and assessing the data quality of every record in the dataset of new ones generated in the case study.

5 RELATED WORK

Several authors have put their efforts into studying IoT within the big data paradigm.^{1, 3, 21} Cecchinel et al.²¹ proposed an architecture based on a similar idea but at a lower level. It consists of the connection of different groups of sensors in sensor boards. They are connected to each other in bridges. These are responsible for aggregating data from the sensor boards and sending them to the cloud. Marhani et al.³ offered a higher-level view of IoT architectures in big data environments. It consists of four layers: (i) IoT devices, which includes the devices responsible for capturing data; (ii) network devices, responsible for the interconnection between sensors and other IoT devices; (iii) IoT gateway, responsible for storing data in the cloud; and (iv) big data analytics, where data is processed to extract value. The real-time data processing and the big data pipeline that is applied in these cases do not differ from the exposed until now. Taherkordi et al.¹ specified the components of a big data architecture that would enable to process of data from IoT devices. Regarding real-time processing architectures, some authors in the literature use the lambda architecture to process the data captured by the IoT devices.^{22, 23}

Regarding data curation, this concept is widely employed in the literature^{16, 24-32} to define the task which implies managing the data so that it keeps its usability during its life cycle, for example, by cleaning, enriching, transforming the data acquired, and integrating it with other data sources . They agree on the importance of the data cleaning process in data-curation workflows. Some of the latest data curation solutions proposed in the literature are reviewed next, focusing on how they face up the data cleaning challenge.

Yan et al.³⁰ devised a cloud-based IoT architecture to support data curation processes, pointing out the importance of data being accurate. For this reason, the architecture they propose contemplates the cleaning of the data through, among other things, the detection of errors. They point out the lack of new methods for this task. We think that our proposal can contribute favorably to the architecture that they propose.

The data curation process involves several tasks which must work together so that the desired result is achieved. Beheshti et al.²⁸ highlights the extraction, classification and enriching of data, among others. They proposed an integration of APIs which facilitate the use of features to assists users in achieving the desired curated data. In other studies, Behesti et al.^{33, 34} developed a curation pipeline for data from social networks. They also highlight the importance of data cleaning to support the curation process. Their proposal on data cleaning is based on repairing text data (generally correcting misspellings). They propose an automatic approach which is able to learn from the knowledge of the users and other sources.

Rehm et al.²⁹ proposed a platform to support data curation workflows in different scenarios. They incorporate data quality features in an effort to filter bad-quality data in text documents is threefold: (i) by ensuring that data follow a specific schema; (ii) by assessing the data quality by means of machine learning algorithms; and (iii) by improving the credibility of data by using reference sources.

Murray et al.³¹ presented a data curation tool applied to the field of COVID-19 symptoms in numerous patients. They highlighted the necessity of cleaning raw data, and the complexity this task involves. They proposed a data cleaning process based on: (i) defining the schema of the data; (ii) establishing the data type of the attributes, and (ii) repairing the format of categorical fields, numeric fields, among others. Their proposal can be applied and reproduced in different systems. The data quality analysis process that we propose can complement their repairing and reproducibility proposal by means of the use of systematic business rules for data quality, and by associating repairing techniques to data that fail to meet the rules.

To the best of our knowledge, our proposal is the first that proposes the use of a proven methodological framework in the field of data quality to monitor the quality of data during its life cycle through context-aware business rules, being systematically applicable to different use cases within the IoT paradigm.

6 CONCLUSIONS AND FUTURE WORK

Current IoT scenarios have to deal with many challenges especially when a large amount of heterogeneous data sources are integrated. In this respect, the use of poor-quality data (i.e., data with problems) can produce terrible consequence from incorrect decision-making to damaging the performance in the operations. Therefore, to use and reach data with an acceptable level of usability has become essential to achieve success.

To overcome this necessity, in this article, we have proposed an IoT big data pipeline architecture that enables data acquisition and data curation in any IoT context. We have customized the pipeline by including the DMN4DQ approach to enable us the measuring and evaluating data quality in the data produced by sensors. Further, we have chosen a real dataset with data from sensors in an agricultural IoT context and we have defined the decision models (DMN) to enable us the automatic measuring and assessing of the data quality with regard to the usability of the data in the context.

As future work, this is the first step toward so in the next steps, we propose to evaluate in depth the data curation by running the decision model using the DMN4Spark tool. This will enable us to do a statistical analysis of the dataset in terms of data quality.

CONFLICT OF INTEREST

All the authors are responsible for the concept of the article, the results presented and the writing. All the authors have approved the final content of the manuscript. No potential conflict of interest was reported by the authors.