The very start of the TRM’s workflow, as indicated on the very left in Fig. 1, is the human user thinking aloud. Thinking aloud lowers the number of interruptions that may come up in cases when the user has to enter ideas into the machine via keyboard or touch input device. Even more: users do not have to pay attention to the machine at all; instead, they may concentrate exclusively on their thoughts while the machine is “taking notes” of what is said.

However, as we will mention below, human speech may not be the only input type for the TRM. Imagine already written texts, e.g. an electronic book that a user reads and for which he/she wishes to see a generated overview map of the content. Other inputs may include photographs on which computer vision technologies detect items or OCR (optical character recognition) detects text. In the future, even advanced brain-computer interfaces (BCI) may be used to visualize concepts derived directly from users’ brains.

Our current focus for the first version of the TRM is on texts derived from users’ speech (thinking aloud) or already existing text documents that the user is reading along.

When it comes to automatic speech recognition (ASR), the latter scenario is already covered well by recent software [13]. Spontaneous speech on the other hand is harder to handle, because of the

speakers variability in speaking and thinking aloud [22]. In recent years, many proposed ideas [e.g. 11, 15] and more training data lead to significant improvements on word error rates, yet it will be difficult to apply the same NLP tool chain as for written text or transcribed planned speech.

The main purpose of the TRM is to extract keywords from speech, hence it is not necessary to have a complete transcription, as long as the system is fed with those words. Structured input can be computed by traditional part of speech (POS) tagging retrieving the context of keywords in a sentence. When the application domain is known, named entity recognition (NER) may add more helpful metadata. Spontaneously spoken words, even if they are understood completely, do not form a well structured sentence, especially when a speaker is verbalizing thoughts and ideas in a brainstorming scenario. Modeling these inaccuracies into POS tagging and NER helps to improve error rates [7].

In order to visualize extracted words – more precisely: their relations to each other – some sort of associations have to be determined. We use weights, ranging from 0 to 1, which encode the strength of such a relations. Weights depend on considered features and can either be retrieved from a knowledge base in Hel (passive) or calculated from linguistic, syntactic or temporal features of the speech (active). Actively inferred weights can be refined over time, because entities may appear more than once and in different contexts. Ideally, these weights draw near the original intention of the user and follow the progress of his/her thoughts.

Visualization happens in a 2D space: keywords are laid out such that spatial proximity to each other serves as a visual metaphor for the inferred weights. As described, these weights may get refined, new keywords appear, less important keywords disappear. To handle this dynamic environment we suggest the usage of a spring based metaphor [cf. 19], because approaches based on physics offer a natural behavior of objects when forces are applied, e.g. a new keyword is added. Other visual properties can encode other metadata, e.g. color to represent named entities or size to show the average weight of an entity. As long as the user does not interact with the space (cf. Sect. 3.4), all visual properties are controlled by the system only.

So far the visualization considers two types of nodes: user nodes (i.e. entities derived from the user’s thinking process via NLP) and suggestion nodes (i.e. relevant information coming from existing Hel knowledge bases). The placement of those nodes is completely based on the machine’s computation of associations.

The TRM, however, provides means to the user to create contexts. As a kind of filtering, such contexts take immediate effects on both the visualization and the computed suggestion nodes. The context is created by anchoring nodes on the 2D space. Those anchored nodes keep their positions and visual attributes. They define the context in which any user or suggestion node is placed.

The associations between anchored nodes are computed by the spatial parsers that are part of Asgard’s spatial structure service. Given the resulting graph, Hel suggestions apply to the whole structure rather than individual nodes. The returned suggestion

Human: thinking aloud

Speech to text & NLP

Midgard: displaying entities in relation

Hel: sending relevant suggested nodes from KB

Human: creating context by anchoring nodes

Asgard: computing associations between anchored nodes

Hel: sending relevant suggested nodes from KB

nodes as well as the derived user nodes are then placed semantically meaningful within the given context.

Figure 2 provides a demonstration of the basic user interface. The screenshot was taken from [18] and shows a scenario in the movie domain with movie titles, directors, actors etc. Large nodes are objects entered by the user. Smaller nodes are suggestions added by the machine. Those suggestion nodes depend on the user created context and may appear or disappear depending on how the user modifies the structure of the user nodes. The above mentioned entities extracted from the user’s speech are not considered in the current system and, thus, do not appear in the given screenshot.

Even though Fig. 1 suggests a linear workflow, the various parts of thinking aloud, creating context by anchoring nodes, analysis of the text input, the space, querying of suggestion nodes, and fluid, animated visualization are happening in parallel. This enables the user to switch between thinking aloud, taking notes on the 2D space, creating context or getting inspired by the visualization or the machine’s suggestion nodes whenever needed.

The TRM lets users and machine populate 2D information spaces with knowledge that changes over time by adding further information to the space or by shaping the context in which computed information should be presented. This fosters human creativity.

The result is a spatial hypertext representing both the process and the outcome of human thinking enhanced by the machine’s capability of computing or searching huge datasets. This knowledge space may then, for example, be further developed at a later point of time, communicated to others, or archived.

This scenario is about synchronous cooperation of spatially dis- tributed people. Imagine Otto and his team working on some so- lutions to certain problems at hand. Due to a pandemic the team members have to work at home. They schedule a virtual meeting for discussing their work.

During the meeting the participants’ audio is captured by the TRM, including information about the respective speaker. The TRM processes the audio input and its metainformation and presents the most important terms in a window that is shared among all participants. The presentation implies the computed relationships between terms, considering the given metainformation. Metainformation can be any information that makes computing relationships between topics more accurate, e.g. who of the participants mentioned which idea or in which mood the speaker was (angry, aggressive, happy etc.).

The analysis and presentation happens in “real time” and is frequently updated such that any participant can follow the current state of the discussion in the TRM window. Furthermore, the system presents additional terms from the knowledge base and presents those accordingly.

While the discussion and TRM’s speech/text analysis takes place, Otto takes over the role of a moderator and starts anchoring selected ideas on the space. By doing so he creates a context of terms which are most important to the team. He implicitly associates them by organizing them spatially on the canvas, changing their size or applying color or other visual attributes to them.

TRM’s structure layer Asgard computes the relations of all anchored nodes and queries the knowledge base for relevant information. As a result suggestion nodes are displayed that match the context as a whole rather than nodes individually. By skimming the space during the discussion, Otto and his team detect some new information of which they have not thought yet and anchor those on the space.

At the end of the session the space shows the suggested ideas mentioned by the participants themselves as well as information suggested by the machine that was added by the participants. The created knowledge on the 2D space has evolved during the discussion with little input from the participants. It represents solutions to the problem the team was working on. Otto and his team can now move one step further and transform the knowledge map into a more formal structure for further processing.

This scenario is about live contextualizing texts during the reading process. Imagine Sue, a college student studying English literature. She attends a seminar about Tolkien’s epic fantasy novel The Lord of the Rings. She is asked to read the books in order to get a better understanding of the various plots and characters.

Sue reads a digital edition of the book that is connected to the TRM. While she turns pages, the system considers each new opened page as input to be processed and displayed on the screen. Thus, Sue can follow how the most important terms, characters etc. in

the story evolve while she is reading. In other words, the TRM proposes a view of the story from the beginning to the spot Sue is currently busy reading. Additionally, Sue can speak aloud any idea or association that comes up during her reading. The TRM will capture her spoken words, process them and add the most important ones onto the space. Beside that the system adds further suggested information which may come from other parts of the novel itself as well as from other relevant books, websites, Sue’s personal notes or other sources.

Not only can Sue be reminded of the various characters and issues mentioned in the story so far, but she may also start creating a context in which the story should be presented: She starts organizing a few characters and items mentioned in the story on the 2D space. The system computes the relationships between the anchored nodes and proposes further relevant information from the knowledge bases.

For example, Sue is in particular interested on the impact the ring has (directly or indirectly) on the relationship between the wizards Gandalf and Saruman. She moves items representing both wizards close to each other and applies the same color in order to express a strong connection. Additionally, she places an object representing the ring. TRM’s structure aware components interpret the space and display relevant information in relation to the given context.

During the reading phase, Sue gets notified by the TRM about some relationships implicitly mentioned in the plot which she would not have discovered otherwise. She adapts the context several times based on her respective focus while reading the story. At the very end she has created a knowledge space on which the most important aspects of the story relevant to her seminar are presented. Furthermore, using TRM’s history feature, she may “travel back” in time and review the creation process or suggested nodes [cf. 2], e.g. for preparing upcoming exams.

Summarizing, for the student Sue the TRM combines note taking, entity extraction, recommender system and search facilities. It fosters her creativity and ability to associate, supported by the machine’s capability to process or search masses of data/information. Similar to the first scenario described above she can use the result for further processing (e.g. writing an essay about the novel) or discussing it with other students.