Browsing by Author "Hotti, Helmiina"

Diagrams are a mode of communication that offers challenges for its computational processing. The challenges arise from the multimodal nature of diagrams. This means that diagrams combine several types of expressive resources to achieve their communicative purposes, such as textual elements, connective elements such as arrows and lines, and illustrations. Humans interpret diagrams by judging how these different expressive resources work together to reach the communicative goals set for the diagram. In order to do that, humans make inferences of the diagram layout and the implicit relations that exist between different parts of the diagram. In order to build computational methods for diagram understanding, large amounts of data annotated with these implicit relations is required. Traditionally, these types of discourse structure annotations have been annotated by experts, due to the difficulty of the task and the requirement that the annotator is familiar with the theoretical framework used for describing discourse relations. The chosen theory for modeling discourse relations in diagrams is Rhetorical Structure Theory, originally developed for modeling textual coherence but applicable to multimodal data as well. This thesis explores the possibility to gather discourse relation annotations for multimodal diagram data with crowdsourcing; employing naive workers on crowdsourcing platforms to complete annotation tasks for a monetary reward. Adapting the task of discourse relation annotation to be feasible for naive workers has been proven challenging by past research concerned with only textual data, and the multimodality of the data adds to the complexity of the task. This thesis presents a novel method for gathering multimodal discourse relation annotations using crowdsourcing and methods of natural language processing. Two approaches are explored: adopting an insertive annotation task where the workers are asked to describe the relationship between two diagram elements in their own words and adopting a multiple-choice task, converting the formal definitions of Rhetorical Structure Theory to understandable phrases to annotate with. Natural language processing is used in the first approach to validate the language and structure of the crowdsourced descriptions. The results of the first approach highlight the difficulty of the task: the workers show tendencies of relying heavily on example descriptions shown in the task instructions and difficulty of grasping the differences of the more fine-grained relations. The multiple-choice approach seems more promising, with annotation agreement with expert annotators higher than in previous research concerned with discourse relations in textual data. The manual inspection of the annotated diagrams show that the disagreement of the crowdworkers and expert annotators is often justifiable; both annotations represent a valid interpretation of the discourse relation. This highlights one of the main challenges of the task, which is the ambiguity of some of the relations. Future work is encouraged to consider this by adopting an approach that is less concerned with a pre-defined set of relations and more interested in how the different discourse relations are actually perceived.