Chapter 8
Crowdsourcing and Human-in-the-Loop for IoT
Luis-Daniel Ibáñez,
Neal Reeves,
Elena Simperl,
Luis-Daniel Ibáñez
Web and Internet Science Department, University of Southampton, Southampton, Hampshire, UK
Search for more papers by this authorNeal Reeves
Web and Internet Science Department, University of Southampton, Southampton, Hampshire, UK
Search for more papers by this authorElena Simperl
Web and Internet Science Department, University of Southampton, Southampton, Hampshire, UK
Search for more papers by this authorLuis-Daniel Ibáñez,
Neal Reeves,
Elena Simperl,
Luis-Daniel Ibáñez
Web and Internet Science Department, University of Southampton, Southampton, Hampshire, UK
Search for more papers by this authorNeal Reeves
Web and Internet Science Department, University of Southampton, Southampton, Hampshire, UK
Search for more papers by this authorElena Simperl
Web and Internet Science Department, University of Southampton, Southampton, Hampshire, UK
Search for more papers by this authorBook Editor(s):John Davies,
Carolina Fortuna,
John Davies
BT Research and Venturing, Head of Next Generation Web Research, United Kingdom
Search for more papers by this authorCarolina Fortuna
Josef Stefan Institute, Department of Communication Systems, Ljubljana, Slovenia
Search for more papers by this authorSummary
Most advanced Internet of Things solutions today are more than just technology. This chapter first introduces crowdsourcing and human-in-the-loop (HITL), two related approaches for realizing use cases and devises data science pipelines that seamlessly combine machine with human and collective intelligence. Broadly speaking, there are two main categories of crowdsourcing activity: microtask and macrotask crowdsourcing. These can be distinguished based on task granularity, or the amount of work required for – and by extension, the complexity of – the task assigned to individual workers. HITL and crowdsourcing are related, but there are important differences: crowdsourcing is a distributed problem-solving approach. An important part of a crowdsourcing project is how to allocate the tasks to participants and how to validate and aggregate their contributions. HITL does not necessarily involve decentralization. The chapter also discusses two instances of crowdsourcing for location data: spatial crowdsourcing and citizen sensing, which are particularly relevant in a smart city context.
References
- CCS Insight (2018). Global Wearables Forecast. Technical Report. CCS Insight.
- Mohanty, S.P., Choppali, U., and Kougianos, E. (2016). Everything you wanted to know about smart cities: the internet of things is the backbone. IEEE Consumer Electronics Magazine 5 (3): 60–70.
-
Nam, T. and Pardo, T.A. (2011). Conceptualizing smart city with dimensions of technology, people, and institutions. In: Proceedings of the 12th Annual International Digital Government Research Conference: Digital Government Innovation in Challenging Times, 282–291. ACM.
10.1145/2037556.2037602 Google Scholar
-
Difallah, D.E., Catasta, M., Demartini, G. et al. (2015). The dynamics of micro-task crowdsourcing: the case of amazon mturk. In: Proceedings of the 24th International Conference on World Wide Web, 238–247. International World Wide Web Conferences Steering Committee.
10.1145/2736277.2741685 Google Scholar
- Bridgwater, A. (2016). Machine learning needs a human-in-the-loop. https://www.forbes.com/sites/adrianbridgwater/2016/03/07/machine-learning-needs-a-human-in-the-loop (Accessed 28 February 2019).
-
Brabham, D.C. (2013). Crowdsourcing. The MIT Press Essential Knowledge Series. Cambridge, MAand London, England: The MIT Press.
10.7551/mitpress/9693.001.0001 Google Scholar
-
Malone, T., Laubacher, R., and Dellarocas, C. (2010). The collective intelligence genome. IEEE Engineering Management Review
38 (3): 38–52.
10.1109/EMR.2010.5559142 Google Scholar
- Simperl, E. (2015). How to use crowdsourcing effectively: guidelines and examples. Liber Quarterly: The Journal of European Research Libraries 25 (1): 18–39.
-
Franklin, M.J., Kossmann, D., Kraska, T. et al. (2011). CrowdDB: answering queries with crowdsourcing. In: Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data, 61–72. ACM.
10.1145/1989323.1989331 Google Scholar
- Haas, D., Ansel, J., Gu, L., and Marcus, A. (2015). Argonaut: macrotask crowdsourcing for complex data processing. Proceedings of the VLDB Endowment 8 (12): 1642–1653.
- Hienerth, C. and Riar, F. (2013). The wisdom of the crowd vs expert evaluation: a conceptualization of evaluation validity. In: 35th DRUID Celebration Conference, Barcelona, Spain, June, DRUID, Copenhagen, Denmark, 17–19.
- Reeves, N., West, P., and Simperl, E. (2018). A Game without Competition Is Hardly a Game: The Impact of Competitions on Player Activity in a Human Computation Game. AAAI.
-
Simperl, E., Reeves, N., Phethean, C. et al. (2018). Is virtual citizen science a game?
ACM Transactions on Social Computing
1 (2): 1–39.
10.1145/3209960 Google Scholar
-
Reeves, N., Tinati, R., Zerr, S. et al. (2017). From crowd to community: a survey of online community features in citizen science projects. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, 2137–2152. ACM.
10.1145/2998181.2998302 Google Scholar
-
Gadiraju, U., Kawase, R., and Dietze, S. (2014). A taxonomy of microtasks on the web. In: Proceedings of the 25th ACM Conference on Hypertext and Social Media, HT '14, 218–223. ACM, New York, USA: ACM.
10.1145/2631775.2631819 Google Scholar
-
Li, G., Zheng, Y., Fan, J. et al. (2017). Crowdsourced data management: Overview and challenges. In: Proceedings of the 2017 ACM International Conference on Management of Data, 1711–1716. ACM.
10.1145/3035918.3054776 Google Scholar
- Smart, P., Simperl, E., and Shadbolt, N. (2014). A taxonomic framework for social machines. In: Social Collective Intelligence: Combining the Powers of Humans and Machines to Build a Smarter Society, Computational Social Sciences (eds. D. Miorandi, V. Maltese, M. Rovatsos, et al.), 51–85. Cham: Springer International Publishing.
- Seaborn, K. and Fels, D.I. (2015). Gamification in theory and action: a survey. International Journal of Human-Computer Studies 74: 14–31.
-
Gadiraju, U., Yang, J., and Bozzon, A. (2017). Clarity is a worthwhile quality: on the role of task clarity in microtask crowdsourcing. In: Proceedings of the 28th ACM Conference on Hypertext and Social Media, 5–14. ACM.
10.1145/3078714.3078715 Google Scholar
- Yang, J., Redi, J., Demartini, G., and Bozzon, A. (2016). Modeling task complexity in crowdsourcing. 4th AAAI Conference on Human Computation and Crowdsourcing. AAAI.
- Daniel, F., Kucherbaev, P., Cappiello, C. et al. (2018). Quality control in crowdsourcing: a survey of quality attributes, assessment techniques, and assurance actions. ACM Computing Surveys 51 (1): 1–40.
- Nunes, D.S., Zhang, P., and Silva, J.S. (2015). A survey on human-in-the-loop applications towards an internet of all. IEEE Communications Surveys Tutorials 17 (2): 944–965.
- Feng, L., Wiltsche, C., Humphrey, L., and Topcu, U. (April 2016). Synthesis of human-in-the-loop control protocols for autonomous systems. IEEE Transactions on Automation Science and Engineering 13 (2): 450–462.
-
Goodrich, M.A. and Schultz, A.C. (2008). Human-robot interaction: a survey. Foundations and Trends in Human Computer Interaction
1 (3): 203–275.
10.1561/1100000005 Google Scholar
- Schirner, G., Erdogmus, D., Chowdhury, K., and Padir, T. (2013). The future of human-in-the-loop cyber-physical systems. Computer 46 (1): 36–45.
- Amershi, S., Cakmak, M., Knox, W.B., and Kulesza, T. (2014). Power to the people: the role of humans in interactive machine learning. AI Magazine 35 (4): 105–120.
- Demartini, G. (2015). Hybrid human–machine information systems: challenges and opportunities. Computer Networks 90: 5–13.
- Zhao, Y. and Han, Q. (July 2016). Spatial crowdsourcing: current state and future directions. IEEE Communications Magazine 54 (7): 102–107.
-
Chen, Z., Fu, R., Zhao, Z. et al. (2014). gMission: a general spatial crowdsourcing platform. Proceedings of the VLDB Endowment
7 (13): 1629–1632.
10.14778/2733004.2733047 Google Scholar
-
Zeni, M., Zaihrayeu, I., and Giunchiglia, F. (2014). Multi-device activity logging. In: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication, UbiComp ‘14 Adjunct, 299–302. New York, NY, USA: ACM.
10.1145/2638728.2638756 Google Scholar
-
To, H., Shahabi, C., and Kazemi, L. (2015). A server-assigned spatial crowdsourcing framework. ACM Transactions on Spatial Algorithms and Systems
1 (1): 1–28.
10.1145/2729713 Google Scholar
- Tong, Y., Wang, L., Zhou, Z. et al. (2017). Flexible online task assignment in real-time spatial data. Proceedings of the VLDB Endowment 10 (11): 1334–1345.
- Klos nee Muller, S., Tekin, C., van der Schaar, M., and Klein, A. (2018). Context-aware hierarchical online learning for performance maximization in mobile crowdsourcing. IEEE/ACM Transactions on Networking 26 (3): 1334–1347.
- To, H., Ghinita, G., Fan, L., and Shahabi, C. (2017). Differentially private location protection for worker datasets in spatial crowdsourcing. IEEE Transactions on Mobile Computing 16 (4): 934–949.
-
Kandappu, T., Misra, A., Cheng, S.-F. et al. (2018). Obfuscation at-source: privacy in context-aware mobile crowd-sourcing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
2 (1): 16:1–16:24.
10.1145/3191748 Google Scholar
- See, L., Mooney, P., Foody, G. et al. (2016). Crowdsourcing, citizen science or volunteered geographic information? The current state of crowdsourced geographic information. ISPRS International Journal of Geo-Information 5 (5): 55.
-
Haklay, M.M., Mazumdar, S., and Wardlaw, J. (2018). Citizen science for observing and understanding the earth. In: Earth Observation Open Science and Innovation, ISSI Scientific Report Series (eds. P.-P. Mathieu and C. Aubrecht), 69–88. Cham: Springer International Publishing.
10.1007/978-3-319-65633-5_4 Google Scholar
- Gallo, D.S., Cardonha, C., Avegliano, P., and Carvalho, T.C. (2014). Taxonomy of citizen sensing for intelligent urban infrastructures. IEEE Sensors Journal 14 (12): 4154–4164.
-
Crowley, D.N., Curry, E., and Breslin, J.G. (2013). Closing the loop from citizen sensing to citizen actuation. In: 2013 7th IEEE International Conference on Digital Ecosystems and Technologies (DEST), 108–113.
10.1109/DEST.2013.6611338 Google Scholar
-
Wiggins, A. and Crowston, K. (2011). From conservation to crowdsourcing: a typology of citizen science. In: 2011 44th Hawaii International Conference on System Sciences, 1–10. New York: IEEE Computer Society.
10.1109/HICSS.2011.207 Google Scholar
-
Gao, L., Hou, F., and Huang, J. (2015). Providing long-term participation incentive in participatory sensing. In: 2015 IEEE Conference on Computer Communications (INFOCOM), 2803–2811.
10.1109/INFOCOM.2015.7218673 Google Scholar
- Gabrys, J. (2017). Citizen sensing, air pollution and fracking: from caring about your air to speculative practices of evidencing harm. The Sociological Review 65 (2_suppl): 172–192.
- Gubbi, J., Buyya, R., Marusic, S., and Palaniswami, M. (2013). Internet of Things (IoT): a vision, architectural elements, and future directions. Future Generation Computer Systems 29 (7): 1645–1660.
-
Kuznetsov, S. and Paulos, E. (2010). Participatory sensing in public spaces: activating urban surfaces with sensor probes. In: Proceedings of the 8th ACM Conference on Designing Interactive Systems, DIS ‘10, 21–30. New York, NY, USA: ACM.
10.1145/1858171.1858175 Google Scholar
-
Oscar Wang, X., Cheng, W., Mohapatra, P., and Abdelzaher, T. (2013). ARTSense: anonymous reputation and trust in participatory sensing. In: 2013 Proceedings IEEE INFOCOM, 2517–2525.
10.1109/INFCOM.2013.6567058 Google Scholar
- Tse, R., Zhang, L.F., Lei, P., and Pau, G. (2018). Social network based crowd sensing for intelligent transportation and climate applications. Mobile Networks and Applications 23 (1): 177–183.
- Jin, J., Gubbi, J., Marusic, S., and Palaniswami, M. (2014). An information framework for creating a smart city through internet of things. IEEE Internet of Things Journal 1 (2): 112–121.
- Guo, B., Zhang, D., Wang, Z. et al. (2013). Opportunistic IoT: exploring the harmonious interaction between human and the internet of things. Journal of Network and Computer Applications 36 (6): 1531–1539.
- Montori, F., Jayaraman, P.P., Yavari, A. et al. (2018). The curse of sensing: survey of techniques and challenges to cope with sparse and dense data in mobile crowd sensing for internet of things. Pervasive and Mobile Computing 49: 111–125.
- Khoi, N., Casteleyn, S., Moradi, M., and Pebesma, E. (2018). Do monetary incentives influence users behavior in participatory sensing? Sensors 18 (5): 1426.
- Sookhak, M., Tang, H., He, Y., and Yu, F.R. (2018). Security and privacy of smart cities: a survey, research issues and challenges. IEEE Communications Surveys and Tutorials 21 (2): 1718–1743.
- Mousa, H., Mokhtar, S.B., Hasan, O. et al. (2015). Trust management and reputation systems in mobile participatory sensing applications: a survey. Computer Networks 90: 49–73.
- Acosta, M., Zaveri, A., Simperl, E. et al. (2018). Detecting linked data quality issues via crowdsourcing: a DBpedia study. Semantic Web 9 (3): 303–335.
- Maddalena, E., Ibez, L.-D., and Simperl, E. (2019). On the mapping of points of interest through streetview imagery and paid crowdsourcing. Submitted.
