Knowledge representation Our framework will model and represent the internal semantic structure of claims in terms of abstract domain-specific concepts and their various possible refinements in testable hypotheses, as sketched in Fig. 3 and Fig. 5: e.g, establishment success, which is an element of invasion success, can be measured as breeding success in the ecosystem where an alien species was introduced, which in turn can be measured as offspring mortality. Importantly, this requires the coupling of a domain-specific core ontology with an argumentation-based ontology.

Argumentation and data Our work will integrate multiple ways and dimensions of modeling hypotheses, i.e., in text but also in knowledge representations and through datasets. As illustrated in Fig. 5, INAS will develop a hypothesis refinement tool that aggregates datasets and hypotheses where hypotheses are structured as causal networks that give detailed information on how parts of a general claim have been attested in data.

Dialogue modeling We propose to model hypothesis development in a dialogue system that uses the HNI ontology to compute hierarchical information states (e.g. the general claim, concepts represented in the claim, sub-parts of the given claim) which need to be filled throughout the interaction between user and system. Thus, the system will not need to process or validate an entire argument at once, but rather focus on specifying different parts of the claim in a step-by-step, collaborative fashion, as illustrated in Fig. 3. The components that process user utterances and link them to hypotheses or concepts in the ontology will be implemented as neural language processing components. These can be trained on large biomedical corpora (e.g. to obtain word and sentence embeddings), but also on the paper abstracts currently represented in HNI.

Evaluation To date, there are few systematic insights into how argumentation systems should be set up to really enhance the way users can understand and develop arguments. An important goal of the project is to develop an evaluation scenario and a user study design that fills this gap and, ideally, can be generalized to other domains or other argumentation support scenarios. We plan to collaborate with other RATIO projects on this topic, e.g. with Philipp Cimiano’s and Ulf Leser’s planned project on argumentation support in a clinical domain.

An outline of the work packages with effort in person months is given in Fig. 4.

Milestones The project will be structured by 3 milestones (see Fig. 4).

1. M1: the basic framework for semantic modeling of hypotheses is set up
2. M2: a proof-of-concept system for interactive hypothesis development is set up
3. M3: the framework is integrated, validated and tested in user studies

WP 1: A semantic model for argumentation in invasion biology

A prerequisite to leveraging the power of Semantic Web techniques are shared ontologies to facilitate the seamless exchange of information. In this WP, we will bring together domain experts, philosophers of science, knowledge engineers, and end users to create such ontologies for our domain of interest (WP 1.2) and the argumentation domain and linking the two (WP 1.3). We will support this with text mining to identify key concepts, their definitions and relations (WP 1.4). Creating ontologies is not a one-time task, but rather an iterative community process which requires support for an evolving and deepening understanding of the domain (WP 1.1).

WP 1.1: Process model It is a characteristic of science that the understanding of a field becomes more nuanced over time. For us, this implies, that the domain model will also evolve over time. At the very beginning of the project, taking into account existing work on ontology evolution (Zablith 2007) and interactive ontology development (Jackson et al. 2019), a process model for this project needs to be agreed upon and appropriate tool support needs to be set up.

WP 1.2: Core ontology The core ontology for invasion biology, called HoH ontology, will be used to model the complex structure of knowledge in the hierarchy of hypotheses in the domain of invasion biology. We will adopt the fusion/merge strategy (Pinto and Martins 2004), where the new ontology is developed by assembling and reusing one or more ontologies. We will first identify relevant terms and keywords by eploiting the knowledge sources mentioned above (collection of hypotheses, publications and datasets). We will then identify, subset and recombine suitable ontologies supported by our JOYCE tool (Faessler et al. 2017) and the deep domain knowledge of one of the PIs. Finally, the ontology will be populated semi-automatically using results from WP1.4. As described in WP 1.1, this is not a one-time activity but an iterative process.

WP 1.3: Argumentation ontology Our argumentation ontology will be based on the AIF (Argument Interchange Format, Chesnevar et al. 2006, Rahwan and Reed 2009, Zagorulko et al. 2019), a standard notation for representing the definitions of high-level concepts related to argumentation. These concepts are categorized into three main groups: concepts related to argument entities and relation among them, concepts relate to the interchange of arguments between two or more participants in an environment, and concepts related to environments in which argumentation may take place. In this WP we will extend this ontology, if such need is identified in the other WPs. A special focus of this task, however, is the population of this ontology with instances related to invasion biology and thus the linking of the two parts of our domain model. Again, this is not a one-time activity but part of an iterative process; in particular results of the workshop conducted as part of WP 6 will very likely result in adaptations of the ontology.

WP 1.4 : Term mining The goal of this WP is to semi-automatically obtain lists of names or terms referring to instances of species and locations, and potentially other entity types identified in WP 1.2 from the INAS abstracts. These will contribute to populating the invasion biology core ontology (WP 1.2) and to fine-tune tools for NER and argument linking in WP 3. Based on resources like LINNAEUS (Gerner et al. 2010), Species-800 (Pafilis et al. 2013) and the generic CoNLL-2003 dataset for NER (Sang and De Meulder 2003), we will explore a combination of different off-the-shelf NER tools to obtain a good coverage of entity types, namely

1. BioBERT, a neural transformer-based network that learns word embeddings on large amounts of text from the biomedical domain and fine-tunes them for different tasks, including NER on LINNAEUS and Species-800 and
2. the LSTM-CRF by Lample et al. (2016).

A subset of the automatic annotations obtained from BioBERT and LSTM-CRF will be corrected manually during the ontology development. These can, in turn, be used to fine-tune Bio-BERT to predict species and locations on the INAS abstracts.

WP 2: Hypothesis refinement

While the ontology development in WP1 focuses on the identification and refinement of concepts used in hypotheses in invasion biology, this work package investigates the refinement of the hypotheses themselves. We design a more detailed, nested representation of the hypotheses in the HNI (WP 2.1) and link this to datasets (WP 2.2). Fig. 5 sketches an example representation that ideally results from this framework, i.e., showing how a hypothesis is decomposed into testable parts and which of these parts have already been tested on data for a given species, location, etc.

WP 2.1: Hypotheses as nested causal networks In the invasion biology domain, hypotheses often are formulated as if they would address simple causal relationships (e.g. ”The absence of enemies in the exotic range is a cause of invasion success”). For domain experts, however, these simplifications are hints to basic knowledge about underlying mechanisms, i.e. longer chains or networks of hypothesized causal relationships. In this work package, we will re-formulate the hypotheses contained in the hierarchical hypothesis network as complex, nested causal relationships. For each element in the causal chains, key references from the domain literature will be searched. The nested representations of hypotheses will be used to annotate a subset of 50-100 publication abstracts in our collection. These annotations can be used as a fine-grained test set for the NLP system in WP 3.1 and will be made available as a corpus to the NLP community (see data management plan). To fulfill this task, we will closely cooperate with philosophers of science.

WP 2.2: Data-hypothesis linking In biology, data is an important dimension of argumentation, as it is needed to test hypotheses and to support or refute claims. Detailed information on available datasets is also very important during the hypothesis development process, e.g., for exploring whether and how a certain claim has been tested in prior work (see Fig. 5). In order to provide users with support for leveraging data for argumentation, we will build on ongoing work in the König-Ries group and elsewhere. We will use and adapt two sets of tools currently under development: The first set provides (semi-)automatic semantic annotation of datasets. We will evaluate available solutions to the SemTab challenge (including our own; under development at the time of writing) and pick and adapt the most suitable one for this task. The second set of tools is currently being developed as part of CRC AquaDiva in the König-Ries group and will offer automatic summaries of data. In the unlikely case that the tools are not available in time, we will manually summarize a limited number of datasets related to the annotated publication abstracts from WP 2.1 for use in the framework. This will result in semantic annotations (suitable for finding datasets) and visual summaries of datasets supporting a quick understanding of their key message. These results will be integrated in our work in two places: First, to provide quick access to data used to test argumentation chains (as depicted in Fig. 5), and second, to support exploration of potentially relevant datasets during hypothesis development as part of the dialog shown in Fig. 3.

WP 3: Interactive Support for Hypothesis Development

In this WP, we will build an interactive system that uses the resources for concept and hypothesis refinement in WP 1 and WP 2 to support users in developing a hypothesis in the field of invasion biology. The main novelty and central challenge here is that hypothesis development is a very abstract task where communicative success is difficult to measure. We will build a neural, non-interactive classification model for text-hypothesis understanding (i.e. linking) (WP 3.1) and integrate this with a dialogue system with a predefined action-state space (WP 3.2.), which will be fine-tuned after an initial user study (WP 3.3).

WP 3.1: Text-hypothesis linking An important task of the dialogue system is to determine which general claim or hypothesis the user is talking about. We operationalize this as a classification problem, where the task is to predict whether a sentence entered by a user refers to a hypothesis represented in HNI. We will set up a neural architecture with two encoders, e.g. RNNs that learn hidden representations of the HNI hypothesis and the textual hypothesis. The central research question here is whether we can successfully leverage the symbolic knowledge encoded in the ontology (WP 1) in the neural encoders for the text and hypotheses, e.g. through compositional neural models (Andreas et al. 2016). Thus, in a currently running Master project in Zarrieß group, we carried out a preliminary pilot study on this task and and tested simple bag-of-words classifiers to link abstracts and hypotheses in the HNI. We obtained a rather low accuracy of 40% with this model, which indicates the need to integrate more abstract domain knowledge. The training and testing data for the network is taken from the 1100 paper abstracts in the current HNI resource. We split these abstracts into paragraphs or sentences, and pair them with the hypotheses that they are linked to in the HNI. By splitting the abstract into smaller parts, we expect to simulate the underdeveloped hypotheses that the user will enter when interacting with the dialogue system. This classification architecture will be trained and tested on different levels of granularity of HNI, for linking texts and hypotheses on the level of major hypotheses and sub-hypotheses, and for different parts. The outcome is a text-hypothesis matching system that will be tested automatically on a test set taken from the current papers in HNI and that can be integrated into the dialogue component in WP 3.2. Another use case of this model would be an automatic extension of the HNI resource with new papers, or new hypotheses developed by users.

WP 3.2: Dialogue model We set up a dialogue component for hypothesis development that splits up this process into a sequence of smaller steps, like e.g. discuss

1. the general claim,
2. the species,
3. how to refine concepts in the general claim, etc., extending Zarrieß' previous work on establishing references in installments (Zarrieß and Schlangen 2016).

Once the system and the user have agreed on a general claim, the subsequent states will depend on the hypothesis components represented in the ontology (WP 1), see Fig. 3. We also design templates and actions for the language generation component which includes verbal feedback, but also actions like pointing the user to nested representations of the general claim (see WP 2.1), to datasets (see WP 2.2) or to more specific definitions of concepts and terms in the core ontology (WP 1). Fig. 3 illustrates a simple potential interaction with such a support system. The understanding component (NLU) of the dialogue system will be based on the two components described in WP 3.1. and also term mining system and embedding models in WP 1.4.

WP 3.3: Hypothesis reformulation As a first evaluation of the dialogue system (WP 3.2.), we carry out a pilot human evaluation with students from the biology programme in Berlin or Jena. This study will give us very valuable data on how users reformulate their hypotheses based on feedback of our system and we will use it to conduct a careful analysis of interaction quality in general and the process of hypothesis development in particular. In case we find that the interactions between our system and users are already of good quality and enable users to develop their own hypotheses, we can use the data to fine-tune/learn aspects of the dialogue system’s action space in WP 3.2 (e.g. when to give certain types of verbal or non-verbal feedback). In the other case, the data will be extremely useful to further develop our system and gain a deeper qualitative understanding of how the system can support the very challenging task of hypothesis development.

WP 4: Resources and Integration

The current version of HNI is available on the public website hi-knowledge.org. We extend this interface and integrate it with the models and resources developed in WP 1.3. The extended interface will be used to run user studies and evaluations, as described in WP 5.

WP 4.1: Ontology and datasets We will integrate the ontology from WP 1 such that users can inspect the meanings of terms used in a hypothesis description or a paper abstract. We will set up a database that records the available meta data for the papers represented in HNI, including the paper abstracts which will be indexed to support basic keyword search and links to available data sets as well as their semantic enrichment where applicable.

WP 4.2: Chat interface We will extend HNI with a simple chat interface to integrate the dialogue system from WP 3, using our web-based SLURK tool (Anonymous 2018) designed for easy implementation of web-based, multi-modal chat-bots.

WP 5: Evaluation

One of the central goals of INAS is to build a framework for argument modeling that is closely tied to the needs of human users. We will thoroughly validate and consolidate the ontologies developed in WP1 with experts and conduct user studies, assessing to what extent our systems helps users in hypothesis development.

WP 5.1: Ontology consolidation and validation During a three-day workshop, the core ontology, the argumentation ontology as well as the nested representation of hypotheses will be validated. We will invite domain experts from the invasion biology community and philosophers of science. We will use a combination of pre-workshop tasks, panel presentations, break-out discussions and panel discussions to reach a broad consensus on the main features of the ontologies and the nested hypotheses. The workshop results will be used to consolidate our models.

WP 5.2: User study We will design and conduct a user study to assess the quality of argumentation support system. This includes the definition of a concrete hypothesis development task that users will have to carry out when interacting with our system (e.g. based on a given paper in invasion biology, define a promising hypothesis for follow-up studies), the identification of a target user group and the definition of criteria for assessing hypotheses that users develop with the help of our system. As users might interact very differently with our system depending on their background, we will need to identify two relatively consistent user groups (e.g. undergraduate or graduate students in biology that have taken classes on ecology) to obtain meaningful results. We will conduct a pilot user study with approx. 30 participants towards the end of the second year of the project (Fig. 4), to obtain valuable data for fine-tuning the dialogue system (WP 3.3) and test argumentation support in this novel setting. In the final user study, we identify two versions of our system that will be compared, e.g. a version with and without interactive dialogue support. We use a mixed within-subjects two-by-two design, where subjects from two different groups interact with both systems, approx. 40 participants (20 from each group) which we plan to recruit at FU Berlin.

WP 6: Dissemination

WP 6.1: Conferences and publications The PhD student and PI Tina Heger will present project results at international conferences and workshops. The events will cover the fields of NLP, semantic web, philosophy of science, invasion biology and ecology. The project team will publish at least 4 publications in international journals and high-ranked conferences from the fields of NLP, semantic web, philosophy of science and invasion biology as outlets. We view research data management and in particular the sustainable provision and publication of FAIR data as another important dissemination activity that will be tackled in this WP.

WP 6.2: Workshop ”Modelling the argumentation process across domains” A further element of this work package will be a workshop bringing together research groups working on similar tools in different domains. Aims of the workshop will be:

• to present our results in order to allow for exchange and synergies with related projects, and
• to compare argumentation processes and ways to model them across domains.