We wanted to apply and develop the statistical and computational expertise of college students to determine the scientific quality of data generated by museum patrons.

Specifically, our objectives are to determine the following:

1. Are the general public able to provide usable real data?

2. To what scale is real usable data generated (i.e. over 70%, for example)?

3. To what scale can different age groups (demographics) generate usable data?

4. Is there a difference in the setting of the kiosk - i.e. one where a facilitator is available (in the Science Hub) versus non-facilitated (in the Specimens exhibit)?

5. To explore the effectiveness of public participation in a museum setting.

We also seek to demonstrate how this exercise was driven by student work in a formal class setting. Although this was part of an industrial mathematics course at Roosevelt University, this type of university and museum collaboration could be replicated by partnerships between educators and museums seeking to work with community-generated data on a large scale. An underlying goal, as demonstrated below, was for students to develop a series of publicly accessible computational tools for data curation, validation and analysis.

Herbaria are reservoirs of both well‐documented specimens and undescribed diversity (Bebber et al. 2010). New species are described each year from specimens that have been housed in collections for decades, if not centuries. However, the pace of such discovery is slow, especially for non‐angiosperms and accelerating the process of discovery is expensive (Soltis et al. 2018). In order to help overcome this, Field Museum began partnering with educational institutions in the greater Chicago area in 2012, developing the MicroPlant project where students and the general public would generate data as community scientists aiding taxonomists (von Konrat 2012). There were many benefits, especially connecting natural history collections to education. Students, in particular, would have a hands-on experience where they could contribute to scientific discovery in a way that could be used in introductory biology courses as well as K-12 settings. Teachers who were not experts in botany or the life sciences could easily incorporate this into their courses. Our group provided an outline describing a model of a crowd-sourced data collection project that produces quality taxonomic datasets and empowers community scientists through real contributions to science (von Konrat et al. 2018). The project is an ongoing collaboration amongst taxonomists, community science experts, teachers and students from both universities and K–12. Scientists, who have more specimens than taxonomists can measure or observe, in this case, could use these student measurements to accelerate the pace of discovery. As a result of meetings between Field Museum scientists, faculties from many institutions, students and community scientist experts, an online web-based version was developed for the Zooniverse platform (Zooniverse 2021b). Classes could meet in a computer lab at their home institutions and generate measurement data contributing to authentic research. The project became surprisingly popular receiving media attention (e.g. Cimons 2018, Ruppenthal 2018) and, in 2017, had over 11,000 participants who generated almost 100,000 measurements. As data were collected, it became clear that due to the size and complexity of a real unculled dataset, it would need to be cleaned and analysed using more advanced techniques and semi-automated tools and so collaborations with the mathematics faculty occurred. An analysis of the web data showed excellent results (von Konrat et al. 2018) and an updated version of the project involving new images was created for a touchscreen kiosk for the Field Museum. This kiosk differed from the classroom setting not only because there was no instructor assistance for data collection, only an explanatory walk through via an optional onscreen tutorial, but also in its physical setting. The question of whether this new interface would lead to a usable dataset remained.

Experiential learning foregrounds the crucial role experience takes in the learning process (Kolb et al. 2014). According to Kolb (1984), learning involves four cyclical stages; concrete experience, reflective observation, abstract conceptualisation and active experimentation. A strong experiential learning experience in data analysis involves a set of real data, questions of interest to the partner and domain knowledge for the data. All of these elements require an ongoing communication between the mathematics students and scientists. Interactions between biologists and maths students occurred in the 2020 and 2021 Industrial Applications of Mathematics class at Roosevelt University. Biologists visited the class, at the start of the semester in person and regularly on Zoom, to present the overarching project, raw datasets of both measurements and demographics, the discoveries so far and background information. Students problem-solved to determine how to process the data, validate and improve upon initial processing and cleaning and analyse the data. The biologists joined the class every few weeks to both pose and answer questions about the data, kiosk set up and the biological underpinnings. Complimenting the experiential learning model, the students engaged in interdisciplinary collaborative community science experience where they worked along with the scientists to unpack raw data, reflect on the data, think about the data and act on how to apply these data in a meaningful way for stakeholders. For over a decade, the need to accelerate the adoption of interdisciplinary approaches has been recognised in an era of vast datasets (Derrick et al. 2011). The biologists were also able to help focus the research in a direction that was most meaningful for their needs. In a typical semester, students would visit the museum, interact with scientists and gain exposure to scientific natural history collections in order to put the science and industrial questions into context. However, due to the global pandemic, this was achieved remotely and virtually, helping the students to understand the experimental set up and to pose relevant data questions. At the end of the spring 2020 section of the course, many questions about selected images were answered, but others about the large-scale dataset remained. There was some student work on the project over the summer and then in spring 2021, the second group of mathematics students worked intensively with this dataset and biologists to expand upon the previous student work and complete the data analysis.

The MicroPlant project focuses on a group of early land plants often referred to as bryophytes. Bryophytes, including mosses, liverworts and hornworts, are the second largest group of land plants after flowering plants and are pivotal in our understanding of early land plant evolution (e.g. Ligrone et al. 2012, Zhang et al. 2020). Bryophytes play a significant ecological role including CO₂ exchange (DeLucia et al. 2003), plant succession (Cremer and Mount 1965), production and phytomass (Frahm 2008), nutrient cycling (Coxson 1991) and water retention (Pócs 1980). Bryophytes, together with lichens, serve as the “macrophytes,” providing a matrix where many microscopic organisms live, including tardigrades, mites, rotifers, micro-molluscs, microalgae, microfungi and prokaryotes (Gerson 1982, Huttunen et al. 2017). For the MicroPlants project, we focused on the liverwort genus Frullania (Fig. 1). This genus has a worldwide distribution and is one of the largest and taxonomically most complex genera of leafy liverworts with more than 2,000 published names (Hentschel et al. 2015). Specifically, participants were asked to measure a modified leaf or lobule (Fig. 1), from digitally rendered images.

Figure 1.  

An example of the liverwort genus Frullania a) Growing on bark; b) Ventral view of stem indicating modified leaves or lobules (L) that participants are asked to measure.

Given the success of the web-based MicroPlant project on Zooniverse, the Field Museum adapted the measurement tool to a touchscreen kiosk which was first used in the exhibit Specimens: Unlocking the Secrets of Life of the Field Museum. This was a special exhibit which ran from 10 March 2017, through to 7 January 2018 showcasing the museum's collection of over 30 million specimens and their ongoing scientific potential. Museum patrons, who had just viewed many of the typically hidden scientific specimens, were able to interact with digitised versions of liverwort specimens on the kiosk so that they too could contribute to scientific discovery (Fig. 2).

Figure 2.  

The online platform, based on Zooniverse, developed into touchscreen technology as part of an interactive kiosk in a high-profile exhibit at Field Museum: a) Instructions were mounted as well as available using the touchscreen; b) Students from Roosevelt University testing the platform; c) Depicting details of the interactive, including the workspace for measuring, instructions on what a pair of perpendicular lines looks like, map indicating the geographic locality and number of measurements.

After the Specimens exhibit closed, the kiosk was updated to include a survey question about the participant's age group. The kiosk then became one of the rotating exhibits in the Grainger Science Hub (Field Museum 2021) where it was used in 2018. After it was retired from the Science Hub, it made appearances at Field Museum Member Nights and at ad hoc events.

Initially, the kiosk was located in the Specimens: Unlocking the Secrets of Life exhibit at the Field Museum. For this phase of data collection, the general public interacted with the kiosk, viewing a brief instructional demonstration which was programmed into the kiosk that showed them an animation of how to measure the length and width of lobules along with the need for these line segments to be perpendicular. They then viewed a randomly displayed MicroPlant image. The image contained a stem of the plant with a various number of lobules, typically between 1 and 10 lobules (Fig. 1), which participants would measure using the touchscreen (Fig. 2). During the course of 24 days in 2017, staff, students and volunteers unobtrusively observed and collected demographics on who was using the touchscreen. Later, these were matched with the corresponding kiosk measurements in the data processing phase.

As these demographic observations were labour intensive, there was a desire for participants to self-classify their demographics; this would also increase accuracy. When the kiosk moved to the Science Hub in 2018, it included a brief demographic survey for participants to fill out. This survey allowed participants to pick one category or to skip the question. However, the survey did not automatically reset between each participant. Demographic information was collected over a 48-day period in 2018. The demographic results were saved separately from the kiosk lobule measurements and these files were matched in the data processing phase (Table 1).

Students in the 2020 and 2021 Industrial Applications of Mathematics course at Roosevelt University were tasked to clean and analyse data generated by museum participants and demographic data collected by interns (Fig. 3).

Figure 3.  

Data generation and processing. Blue round-edged rectangles indicate data from the public. Green round-edged rectangles indicate data processed by hand by students. Yellow rectangles indicate automated data processing. Red rectangles indicate data which have been filtered out.

Once the data was split into a csv file where each row corresponded to a unique pair of intersecting lines, it was necessary to determine which data were of sufficiently high quality to use. As the kiosk specified that pairs of line segments should intersect at 90 degree angles (see Fig. 2), an initial cut was made to all data based on the angle measured. Good data are any data with an intersecting pair of lines that have an angle 80 degrees or above. Note that the kiosk records the smaller angle that occurs between the line segments; thus, if a pair of line segments intersect with smaller angles of 85 degrees and larger angles of 95 degrees, the dataset only records the 85 degree angles. This means that the maximum angle measurement possible is 90 degrees. We analyzed one image in detail, comparing measurements created by the public to a set of measurements created by an expert. This led to a second set of cuts using the interquartile range (IQR) independently for each axis to remove outliers (Zwillinger and Kokoska 2000). To calculate the IQR cut for each axis, we first split the dataset into quartiles (Q₀ = min ,Q₁, Q₂ = median, Q₃, Q₄ = max) and set IQR = Q₃-Q₁. We keep all data that are in the range from Q₁-1.5*IQR to Q₃+1.5*IQR (Table 2).

Fig. 6 is an example of a pair of lines that do not intersect, known as invalid data. The image is an example of a data entry that would not meet our qualifications for good data or IQR range data. However, this is the kind of data that will be under the category of all data because all data accept any data entry regardless of quality.

Figure 6.  

A lobule with a pair of non-intersecting measurements.

Fig. 7 passes the qualifications for potentially valid data, a pair of intersecting lines. This is an example of a pair of lines that intersect with an angle of 33 degrees. Note that the smaller angle is recorded rather than the larger 147 degree angle. This image is an example of a data entry that would not meet our qualifications for good data or IQR range data. However, this is the kind of data that will be under the category of all data and valid data.

Figure 7.  

A lobule with a pair of line segments which intersect, but which do so at a small angle.

Fig. 8 passes the qualifications for good data, a pair of lines that intersect and form an angle of at least 80 degrees. This is an example of a pair of lines that intersect with an angle of 88.6 degrees. This is an example of a data entry that has the potential to pass the qualifications of the IQR range data. This is also the kind of data that will be under the category of all data, good data and potentially IQR range data.

Figure 8.  

A lobule with a pair of nearly perpendicular intersecting line segments whose smaller angle is above 80 degrees.

While the initial data analysis was performed on a 2018 dataset collected from the Science Hub, a secondary analysis was done on a dataset collected in the Specimens exhibit. This exhibit was focused on the large collection of scientific specimens at the museum and so the kiosk was only a small part of the larger exhibit. This differed from the Science Hub, which is a dedicated space where visitors can interact with scientists, as well as specimens from the Museum's collection. The fact that there was a smaller timeframe where interns collected demographic data from the Specimens exhibit (24 versus 48 days), meant that the amount of data collected from the Specimens exhibit was smaller. As some of the images had a very small amount of data associated with them, we combined the two datasets to perform the IQR cuts. Note that because the IQR cuts depend on the specific dataset used, the results may change when additional data are added in. This happened here; the IQR cut for the 2018 data alone had 3,125 pass the cuts. When we added in the 2017 data, there were 3,126 data out of the 2018 set within the IQR ranges. This suggests that combining the two is robust. When these combined cuts were used to examine the 2017 exhibits data, we found that, although there was a smaller amount of 2017 data collected, the quality was similar and the majority of data collected was usable, based on completeness, angle and IQR cuts. As the quality of data was good, the majority of images in the 2017 dataset had sufficient data to determine the lobule lengths and widths (Table 6).

To gain more insight into the image measurement data, an analysis was conducted on the images themselves. The classification of the MicroPlant images was based on the number of lobules present, complexity and clarity of the image. Standard deviations of the axis length measurements were used to determine the clustering of the axis measurements. The lower the standard deviation, the closer together the measurements are clustered. The image classifications were then compared to the standard deviations of the axis measurements, post IQR cut for each image. There were no notable trends present between the complexity of the images and the standard deviations of the measurements.

We then looked at images with large standard deviations, meaning the measurements were not very clustered together. Out of the 78 distinct subjects, we found only three images with very large standard deviations, one of them from the 2017 dataset. This was only 4% of the total images displayed on the kiosk. One similarity between these three different images is the number of observations counted, with each having between 6-14 observations total. This can be one possible explanation about the low quality of data collected since such data were very limited. For the image with only six measurements, there may have been confusion determining the difference between the lobules edge and the leaf behind the lobule. The shading of the pictures can leave room for confusion as well; and it might become unclear what is considered part of the lobule and what is not.

With the goal of comparing the accuracy between participant measurements and expert scientist measurements, students conducted the same statistical test used in von Konrat et al. (2018) on the IQR cut data for image ID. No. 8735435. This was the third distinct image which had expert measurements associated with it. The statistical test used was a t-test which measures the difference between two datasets and determines if they are significantly different. After performing the angle and IQR cuts, 45% of the original 855 participant measurements remained. After calculating the t-values for the minimum and maximum axis data with a confidence interval of 95%, we were able to conclude that the participant measurements after IQR cuts were not significantly different from expert measurements.

Though literally hundreds of citizen or community science platforms exist, to our knowledge, this was one of the first to be featured in a live interactive museum exhibit. Commonly, people-powered research projects engage participants online via platfoms like Zooniverse (Zooniverse 2022) or via targeting specific interests of their users- iNaturalist (California Academy of Sciences and National Geographic Society 2022) or WeDigBio (WeDigBio 2022). We wanted to know what impact placing an exploratory and unguided community science platform within a museum setting would have. What percentage of people who pass through the exhibit would stop to engage with the kiosk platform (Table 7)? Rough calculations, based both on our observations, ticketing information and dataset timestamps tell us that about 14-20% of individuals who passed through the Specimens exhibit interacted with the kiosk in some manner. We were able to tabulate kiosk interactions by occasionally placing interns who observed and recorded from a distance who was interacting with the exhibit following standard protocols for observing people in exhibitions. These observers noted approximate age and perceived gender of participants, as well engagement levels.

The Science Hub is designed for hands-on interactions and discussion with scientists. During the time that the kiosk was present in the Science Hub, 23,549 people visited the Science Hub, with 1,014 interactions with the kiosk. Based on this, we estimate that between 4.3% and 12% of the visitors to the Science Hub interacted with the kiosk; if the group sizes were similar to those directly observed in the Specimens Exhibit, approximately 8% of the Science Hub patrons interacted with the kiosk. As the kiosk was a stand-alone exhibit in the Science Hub, it is likely that museum patrons were more inclined to interact with the scientists present rather than a stand-alone computer exhibit.

From this study, the public is capable of producing a usable set of measurements. However, there are two limitations to the precision of these measurements. One is the touchscreen technology. As the smallest unit was a pixel, the difference of just a couple of pixels in the measurement corresponded to a 1% difference in length. This level of precision could not be improved upon with the technology used. The second is the variation in public measurements. Scientists who want to distinguish between species whose size differences are large (such as 10% difference) would be able to use work from the public; however, if the difference is very subtle (such as a 1% difference in length), it would not be possible.

For cases where multiple types of species may exist in an individual image, there could be multiple sizes of measurements in an individual image. This would make the IQR analysis ineffective at finding outliers and so more subtle methods would need to be employed. However, if one found clusters of different sizes, it may be possible to create a machine-learning algorithm to use for the data processing portion.

Interdisciplinary science entails the collaboration of scientists with largely non-overlapping training and core expertise to solve a problem that lies outside the grasp of the individual scientists (Cech and Rubin 2004). Yet interdisciplinary research (IDR) is more than collaboration: it is also applying concepts or methods from other fields or writing to make your research accessible to other types of scientists (Brigandt 2013). IDR is better suited for addressing critical “big picture problems” such as sustainability and conservation (Palmer 2001, Carayol 2005, Campbell 2005). Early IDR exposure aids cross-discipline communication (Bridle et al. 2013) and makes students more likely to pursue STEM careers (Daugherty and Carter 2017).

This was an authentic interdisciplinary experience in experiential learning. Students from various backgrounds, specialities and ages were involved in all aspects of this project from inception to completion. While this report focuses on the data analysis, prior collaborations with both college and high school students led to the development of the project. Thus, all involved students participated in a rich, real world learning experience to generate and later analyse a real and meaningful dataset answering questions that were previously unanswered. Students involved in the data analysis found the skills that they acquired through the project to be highly applicable to their post-graduation jobs, with comments such as: "I’ve used a lot of the VBA (Visual Basic for Applications) skills from your classes with the Field Museum". This indicates that this project was good preparation for both research work and industry jobs. As this course was originally developed as part of the PICMath programme, it is a way to both fulfil the goals of the programme and to increase scientific knowledge. Future endeavours could implement student evaluation prior to, during and post course, as student feedback on their learning journey is effective in improving both student satisfaction and learning (Mandal 2018).

Despite the critical role of experiential learning in building student research skills and capacity, few have explored social interaction mechanisms used to facilitate student experiential learning in an interdisciplinary research team (Ryser et al. 2008). This has great potential in future reiterations that could be investigated.

There remain many interesting education and learning questions that could be investigated using the current dataset, as well as future studies embarking on similar large scale projects. For example: How many measures must be taken by each kind of user group? Are there significant differences in measurement facility amongst children, adolescents and adults? Are there significant differences between a facilitated audience and a purely online audience? Limited work has investigated the arc of engagement from secondary to post-secondary education and into adulthood. Examining a cross-sectional population set will allow us to study reasons and motivations of learner engagement moving from a formal to an informal setting. The potential also exists to use this project to explore how authentic research experiences can both develop student interest in STEM and STEM careers (Boyer 2017), as well as promote learning of biodiversity concepts (Gunckel et al. 2012).

Although it is possible to use this dataset to compare kiosk locations, in the future, having a consistent set of self-described demographic categories would allow for a consistent comparison of how different demographics interact in the the different kiosk locations. However, given that the desire to collect demographic information was realised after some data were collected, the addition of a brief demographic survey to the kiosk was a natural course correction. In addition, collection of demographic information before starting the activity interrupts the activity and presents other challenges when switching between participants.

Our experience with this project yielded insight into how to plan for future projects. For other taxonomic projects, the most robust measurements will involve images where the public can easily identify the object to be measured. As noted in the conclusions, the distinction between smaller lobules and the larger underlying leaves led to a number of inaccurate measurements. As this is a large, multi-year project, students who are involved in it will graduate before it is completed. Faculty and museum leaders are the ones maintaining continuity of the project; they need to make sure that the student data work is documented and stored in a way that is both carefully labelled and accessible. This allows multiple years of students to collaborate and make significant continual progress in a robust manner. In terms of project management, the most challenging aspect was maintaining contact and retaining connections between student cohorts. It is critical to plan for this from the very start of development when pursuing such projects.

Though we are able to tabulate a certain interaction level by patrons with the kiosk (Table 7), it is worth noting that we also have sets of observational notes about how individuals and groups interacted with the kiosk display. As we did not have questionnaires and surveys always linked to the activity, engagement levels and observations were noted by onlookers for a portion of the time that the kiosk was in the Specimens exhibit. From observer notes, we were able to compile a word map demonstrating the true inclusive nature of the activity and summarise interactions with the kiosk (Fig. 11). As museums are often visited by families and groups, the kiosk was a place where people gathered to interact and engage not only in science, but with each other. We were able to note many examples of parents, children and peers working together in a truly collaborative manner as is core to community science. Our anecdotal observations of patrons interacting with the kiosk support the supposition that there is a growing body of evidence suggesting that such digital technology can create engaging learning opportunities in museums (Roberts et al. 2018). There is great potential in implementing community science activities in a natural history museum environment using digital technology to help foster curiosity and engagement with scientific collections. Unobtrusive video recording and patron surveys would be invaluable in providing deeper insight.

Figure 11.  

Word cloud of observer notes of kiosk participants who were recorded from an unobtrusive distance.

In today’s society, K-12 students' technological interests with platforms, such as digital making (Lewin and Charania 2018), can enhance students' informal learning experiences (Lai et al. 2013). The MicroPlant kiosk created an informal experience where participants of all ages leveraged their digital curiosity to learn more about biodiversity in under-represented plants. The MicroPlant project's success in the in-person, online and kiosk formats generated an avenue for this theme to be explored on a greater scope. That project focused on participants' ability to produce high quality data, based on measurements of a specific morphological character set. The Unfolding of MicroPlant Mysteries project (Zooniverse 2021a), initiated and designed by two high school interns, focused on participants’ capacity to classify and identify morphological features, such as branching patterns and sexual phenotypes (Fig. 12). Data from the project would then be analysed by undergraduates and Masters' students. This expands data generation beyond measurements; this will greatly aid in accelerating biodiversity discovery and documentation of these organisms. To date, this project has met with enormous success logging over 60,000 classifications from over 800 users with early results showing over a 70% accuracy rate for the reproductive structure task (Gillis et al. 2022). Supervised learning algorithms have the potential to use multiple crowdsourced inputs to generate a response whose reliability is similar to an expert’s (Li et al. 2015). Here, we performed a preliminary and exploratory data analysis using sequential neural networks. Clusters of boxes that users drew around reproductive structures were used along with expert boxes as training data for a neural network; this neural network was then validated using a second set of user drawn clusters and expert data, resulting in over 90% accuracy in classification for the reproductive structure task. The class was also able to make several recommendations to improve the project, such as the addition of a no-branching option, changes to the tutorial and data management suggestions. This content is also currently being mapped on to Next Generation Science Standards (NGSS) (NGSS Lead States 2013) and is being used to explore the utility of a new Zooniverse classroom (https://classroom.zooniverse.org/). The research team is also exploring the potential application of machine learning as recent studies have show that combining human and machine classifications can efficiently produce results superior to those of either one alone(Trouille et al. 2019). With the aim of making science accessible to a large and diverse demographic of learners, we hope to continue to emphasise the important connections and potential collaborations between universities, museums, students, researchers, and the general public.

Figure 12.  

A plant stem of the liverwort genus, Frullania. Participants identified and outlined male parts in blue and female parts, in red and green. The dots in the centre represent the centroids of the boxes; these were used for data clustering purposes.