We encounter hypertexts (HT) as a technology everywhere, and if we think only of the World Wide Web, hypertexts are technologies that are used unnoticed and unrecognized just everywhere as the basis for today's everyday facilities. Nevertheless, HT systems are still a marginal phenomenon in the area of three-dimensional systems, virtual reality (VR) or augmented reality (AR) (c.f. [22] for an overview in this regard). Irrespective of this, there has been an increase in this context in recent years and quite a few HT systems have been developed, although not explicitly labeled as such (c.f. [1]). Furthermore, it has been shown that HT systems in VR lack requirements and quantifiable features, which makes it difficult to classify their formal functional scope. To address this challenge, [1] defined twelve criteria, which should be met in order for a system to be sustainably usable as a flexible, platform-independent, parameterizable, reproducible system across different domains. At the same time, Va.Si.Li-Lab  [27] was introduced as a HT framework that addresses the latter flexibility, but is not yet ready to provide it fully, since users still cannot change environmental parameters at runtime so that the perceived virtual scenes reflect the dynamics of everyday outdoor scenes. Beyond that, among other aspects, the feature of modifying a scenario and manipulating the environment and objects within virtual environments (c.f. [9, 31]) is an essential aspect that is currently not adequately implemented by any of the VR systems evaluated [41]. Especially with regard to the development of modern web applications and mobile apps, the dynamic and flexible use of virtual environments should be standardized in VR and AR applications. Since this is a missing feature of Va.Si.Li-Lab, this paper is dedicated to a significant extension of this VR solution by integrating a module for a dynamic event system to create, modify, correct and serialize virtual scenes, environments, objects and interaction functions called Va.Si.Li-EventSystem (Va.Si.Li-ES). All new features can be used within Va.Si.Li-Lab, which is available via GitHub under an AGPL license.

For the next steps towards an increasingly comprehensive HT system, two essential features of  [1] are to be detailed and prototypically implemented in Va.Si.Li-Lab: (B) Scenario editing and (J) Individualization of object and environment parameters. The first specifies that all environments in a HT system must be dynamically configurable, the usage parameters must be external and modifiable during runtime and generally serializable. In this case, it does not matter whether a scenario or an environment is created using a semi-automated pipeline (e.g. [37]), as this ultimately only changes the appearance, but does not create any dynamic events. This feature is primarily about reducing or even omitting the use of expert software editors such as Unity and the development of scripts within the programming language of the HT system, as these topics should be integrated as fully as possible into the program logic to allow even non-experts to develop them. The second feature is related to the previous one, as it describes and defines the general possibility of individualizing, modifying and dynamizing objects as well as the environments in general or partially in virtual environments.

These new features are based on the above criteria, which are not available in most tools (section 3), are defined as follows:

Dynamic event system: As the dynamic quality of an information system increases, its application horizon expands because different conditions can be mapped on a single platform. On the other hand, configurability opens up the system to different user groups and disciplines. This requires a dynamic event system that fulfills the following requirements:

Modularity: To use individual components in different constellations and including different actuator and sensor effects to generate atomic events, modularity is essential. Through a suitable modularization concept, such as the use of interfaces, extensibility is supported.

Extensibility: In addition to modularity, i.e. the clear separation of the individual components from each other in terms of content, but with functional interdependence between them, a basis for the extensibility of the event system is required to ensure dynamic usage.

Concurrency: In a collaborative multi-user context, it is important to support concurrency of events in the sense that events can be active, executable, or neither for a single user, for all users, or a subset of them. At the same time, there are dependencies between events that need to be taken into account by an appropriate algorithmic strategy.

Concatenability: The mapping and reusability of event structures requires that their relationships and dependencies can be defined and new events can be generated dynamically.

Serializability: To ensure that events can be stored, reused, duplicated, and even developed or modified externally, it is critical that they can be serialized and also deserialized for reuse. It is advisable to use a suitable database or serialization technology.

Environment system: An event system with flexible and extensible components should allow for changes in virtual environments at runtime. This may affect perceived brightness (as a result of day/night changes), whether visibility is reduced as a result, or perceptual effects as a result of recorded ambient noise.

User feedback and information: Overcrowded visual environments, unclear structures and poorly selectable interaction objects are just a few of the negative aspects of user interface design [25]. This can be a particular challenging in VR, especially if its interface is not based on conventional UI elements from desktop environments. In multi-user HT systems, it is critical to provide users with appropriate information and notifications. Whenever an environment variable has changed, the user gets a message, an error message, or simply a new feature is made available. Providing appropriate information and the ability to retrieve it without being overwhelmed will lead to a higher level of acceptance and allow for greater immersion. This requires that feedback to users be dependent on the dynamic event system, as information must be created and delivered in a context-sensitive manner.

These properties for environment-dynamic HT systems can mean an advance in dynamization for virtual HT's. section 5 presents corresponding use cases that exemplify this. In the next section, an overview of the available tools including the listed functions are provided.

Since there are various forms of visualization [3] and applications in the field of VR [5, 30], our focus is on applications that include at least one fully immersive component and the ability for collaborative use by multiple users. Since the last requirement rapidly limits the number of tools (e.g. [6, 11, 16, 34, 38] as well as commercial tools for architects [39]) and a previous paper already listed comparable VR systems from the area of STEM (Science, Technology, Engineering, and Mathematics) [29], it becomes obvious how few collaborative and truly multi-user-capable VR applications currently exist.

First, IMUFTS (Immersive multi-user firefighter-training scenarios)[10] provides scenarios for emergency response and firefighting training. As a prototype, IMUFTS describes an approach using multi-user training methods with full-body avatars together with motion-capture-based VR technologies. Admittedly, the existing scenarios have a static design and if events are available, these must also be implemented manually (). It also lacks both of the other features (, ). In the same application context, Megacity [36] is designed to simulate training scenarios in order to prepare users to take the right decisions in an emergency situation. In this environment, a city is simulated in which users and virtual avatars can interact with each other. Also in this application, all scenes and all sequences of events () are created manually and the environment () can only be changed manually, where technically possible.

Table 1: Comparing VR-based systems according to the features of Section 2. Legend: satisfied (), partially satisfied (), not satisfied (), unknown ().

The next tool is the virtual BIM reviewer (VBR) [42], which enables several participants to work together on engineering tasks in an immersive way in order to train collaboration. While dynamic events in a pronounced sense cannot be found, the system does allow some dynamic functions, such as inviting other users to their position, teleporting as well as rotating and scaling of objects (). In addition, however, the virtual environment cannot be modified () and there is no user information beyond a chat ().

Virtual museums can be created within ARCO – Augmented Representation of Cultural Objects [40], whereby the environments can be variably adapted by the respective content creators according to prefabricated templates. Although there is no direct collaborative component, the mixed reality function at least allows for a mutual exchange. Although the creation of the VR environment is dynamic and variable and follows a scheme, no events () can be generated and accordingly no environments () can be modified and there is also no user feedback () in the predefined manner, although the interaction with the VR objects is very strong. Moreover, VR-Liver [12], a tool for collaborative planning of liver surgery, is available in the medical sector, although there is no feature as described either. There is also VR-Nurse [17], a tool available in the same area - just as limited in features as the previous tool, the application allows a training course for nurses. With ElectroVR [20] an immersive tool for the collaborative use of an electrostatic playground is available, which allows different scenarios to be experienced based on a simulation. Although the application contains a large number of different learning tools on various topics (e.g. magnetism, current flow), these cannot be changed dynamically at runtime (), nor is it possible to change the environment ().

Beyond this, there is also VR-XR-Escape-Room [21], in which two participants are in a physical room and a counterpart to this room exists in VR, which entails the possibility of passive haptic experiences, as all objects in VR are also constructed in reality. Nevertheless, VR glasses are used by the users and the sensor technology is also supported by means of Leap Motion, although none of the required features are available due to the setting and the dependency on the physical space.

According to our current state of knowledge, the availability (Table 1) of tools that include a dynamic event system does not exist for virtual three-dimensional realities in the compositions described and required (section 2).

With Va.Si.Li-Lab, a HT application for three-dimensional virtual environments was implemented based on Unity, which enables several users to simultaneously and collaboratively consume multimodal information (texts, images, audio and video) and their relations to each other, as well as to interact and communicate multidirectionally. At the same time, it enables platform-flexible use due to its implementation in Unity and includes a multimodal database in which – in addition to environmental information – activity, interaction and communication information can optionally be stored for further analysis. The multi-user capability and the storage of individual actions and user communication are supported by the integration of Ubiq  [19] and the corresponding backend, which has been adapted for the storage of intra-action information. Collecting various environmental and interaction information (e.g. speech, gestures, object handling, user position) enables a later quantitative analysis with the capability to use Natural Language Processing (NLP) methods using DUUI  [23] for automatic processing and TextAnnotator  [4] for manual annotation. Beyond this, users can modify their own appearance by choosing between different Ubiq and Meta avatars. This enables an individualization of the appearance of users as well as their functional use for the design of various social scenarios, among others. An arbitrary amount of virtual environments (so-called Room) can be created within Va.Si.Li-Lab, in which various scenarios (Scene) with different configurations and parameters can be hosted (see  Figure 5 (appendix)). Room s encapsulate logical units allowing users to use the Scene s contained in the Room s; these are created with the help of the associated database module. In addition, Va.Si.Li-Lab provides explicit annotation of multimodal objects using an existing and reused annotation tool for virtual environments, VAnnotatoR  [13]. With this incorporation, texts, images and their segments as well as video and audio and 3D objects in general can be annotated and related with each other (c.f. [26]). Furthermore, these relations themselves can also be a starting point for further relations, which ultimately allows the creation of hypergraphs [2].

4.1 Event system

As part of the integration of a dynamic and flexible event system (), a series of data structures and features were implemented in Va.Si.Li-ES, which can be stored and reused in serialized form in the associated database. Since the concept of event systems is not new, the implementation Va.Si.Li-ES was based on the work of [8] with some adaptations. In addition, there are already script-based solutions, but rather for behavior generation of avatars [15] or partly also for scenarios [28], but as far as we know, not in the context of Unity and with general events. Therefore, the general idea behind the dynamic event system is that even non-advanced users of Va.Si.Li-Lab can create their own scenarios without having to create them manually, for instance with an editor such as Unity.

Figure 1

Each EventSystem is associated with a specific Scene and contains an optional set of different implemented Trigger s to start it and a set of Event s of various implementations. Using Options, it is possible to repeat it periodically.

Figure 1: Each EventSystem is associated with a specific Scene and contains an optional set of different implemented Trigger s to start it and a set of Event s of various implementations. Using Options, it is possible to repeat it periodically.

Within Va.Si.Li-Lab, the dynamic event system (EventSystem) can be defined for each Scene and can be transferred to other scenes if required, although this is not designed atomically across scenes, in order to avoid dependency problems if, for example, objects exist in one scene while not in others. Furthermore, the EventSystem is intuitively designed for long-term reusability and easy expansion. As a result, the EventSystem consists of just three components (Figure 1) and can be used within Va.Si.Li-Lab by utilizing various UI elements (Figure 6):

Trigger: An EventSystem may have a set of Trigger s that serve as a precondition for the execution of the corresponding EventSystem, whereby the absence of a Trigger implies it will be executed immediately. In addition, all Trigger s assigned to an EventSystem need to be fulfilled in order to execute it. Currently, the following Trigger s are available, although further Trigger s can be added, as these have been implemented based on an interface:

Timer: It creates a delay on the basis of hours, minutes and seconds before the EventSystem is executed (6(d) ).

Area: Creation of an area in Unity as an activation zone for triggering an EventSystem (6(f) ). For this purpose, it is also possible to modify areas by specifying that a specific object must be transported into the area as well as a user must enter it.

Object: In addition to the former Trigger s, the Object-Trigger enables the activation of an EventSystem by interacting with a specific object while holding the target object or contacting it with the user's hand (6(e) ).

Event: Represents the acting component of an EventSystem and, like Trigger s, several Event s can also be executed within it. All Event s are executed simultaneously when the EventSystem is activated, although a delay and other Trigger s can be taken into account by encapsulating them as a follow up EventSystem. Like Trigger s, a set of defined event types that can be utilized in Va.Si.Li-Lab, although an interface-based extension is also supported:

EventSystem: In order to create more complex EventSystem structures and dependencies, each EventSystem can be used to start additional n EventSystem s, whereby the dependencies and relations to each other can be visualized in VR (6(c) ).

Teleport: With the Teleport-Event, the triggering user or all users in the Scene can be moved to any defined location in the same Scene (Figure 2). Like Trigger s or Event s, a UI element marks the target location (flag).

Object: With the Object-Event, new virtual objects can be created, existing ones modified (e.g. resizing, positioning) as well as deactivated or completely removed. For this purpose, objects can be selected from the existing interior of a Scene as well as from the ShapeNet [35] repository.

Environment: The Environment-Event enables modifications to the environment of each scene, which is not primarily represented by concrete and tangible objects. This includes the modification of the ambient brightness and the virtual weather conditions. This means that the environmental brightness (7(a) ) changes between day and night and the weather changes between sun, clouds, snow, rain, fog and thunderstorms. In addition, the weather modifications are provided with an adjustable intensity to simulate different scenarios (section 4.2).

Feedback: Visualization and interaction with complex user information and input through two core functions: The uniform transmission of information and feedback to single or multiple users and the facility to participate in surveys or votes (section 4.3).

Options: There are two optional parameters for an event system in order to implement a lot of dynamics in a minimalist design:

Repeat: The repeat option enables the periodic repetition of an Event, whereby the Trigger s are always checked again, allowing for example new objects to be created regularly with a defined delay or the weather can be changed by integrating the Time-Trigger.

Uncouple: The uncouple option prevents an event from being influenced by the repeated activation of its predecessors through the hierarchy of events, is also closely related to the repeat option.

Through this implementation Va.Si.Li-ES, Va.Si.Li-Lab can simulate a dynamic execution of action sequences based on user behavior and random or intentional interactions. In addition, the EventSystem facilitates the configuration of a scene with 3D objects and content information without manual design by content creators who have to edit the scenarios at script level. For this purpose, EventSystem can be created during runtime of Va.Si.Li-Lab using 3D glasses or the desktop application. In both cases, there is no need to adapt scripts in C# or manually add assets (3D objects) in Unity. To avoid losing track and to reuse any objects created or moved during the execution of an EventSystem in following EventSystem s, a preview function is implemented (6(b) ) which simulates a mock-up run for the corresponding EventSystem as well as its following EventSystem s. In addition, EventSystem s which have been created can be serialized in the Va.Si.Li-Lab database to allow users to interact with the EventSystem when they enter the Scene (or Room containing the scenario) by deserializing and instantiating the EventSystem within the scenario. Serialization is implemented using a JSON conversion of the entire EventSystem with all associated Trigger s and Event s. For this purpose, each EventSystem is identified by a unique name within a scene, which is assigned by the creator of this EventSystem.

Figure 2

Teleportation can address all users in a Scene or only the user currently triggering the EventSystem.

Figure 2: Teleportation can address all users in a Scene or only the user currently triggering the EventSystem.

Figure 3

Virtual objects can be created, modified or deleted as Event s during runtime of Va.Si.Li-Lab. This includes existing objects that already existed in the Scene as well as new objects which can also be duplicated from existing objects or loaded from ShapeNet as new objects. In addition, the scaling for new objects (on the right in the illustration) can be defined to be used for loading them into the Scene.

Figure 3: Virtual objects can be created, modified or deleted as Event s during runtime of Va.Si.Li-Lab. This includes existing objects that already existed in the Scene as well as new objects which can also be duplicated from existing objects or loaded from ShapeNet as new objects. In addition, the scaling for new objects (on the right in the illustration) can be defined to be used for loading them into the Scene.

4.2 Environment system

4.3 Feedback system

Virtual environments tend to become confusing [32], users can quickly feel overwhelmed [7], and the possibility of motion sickness is on the daily agenda for many people [24]. Apart from the last point, structures can also be used to implement suitable feedback and visual aids that enable the user to be actively informed and accelerate an action by means of predefined selection options. As a result, the user can automatically enter a new virtual state upon confirmation (e.g. by invoking a EventSystem) instead of being forced to find a specific virtual room after a defined period of time. Certainly, it would also be conceivable to execute this kind of Event automatically, but the amount of control for the user should not be underestimated, especially with regard to motion sickness where, for example, a teleport to a new position takes place without warning or a sudden automatic movement is caused which is not under the direct control of the user. For this, Va.Si.Li-ES was equipped with a feedback system (), which informs the users in different ways about changes, announces an imminent alteration or progress and also allows users to participate in decisions, which are presented in the manifestation of votes. As shown in Figure 4, all Feedback-Event s have an optional timer that can hide the feedback after it has expired if it has not yet been confirmed or used by the users. In this process, all feedback are shown to the user as push messages, which can be generated as a result of an EventSystem. In order to distinguish between the individual types, each message is visualized differently allowing messages to be differentiated as information, warnings or errors (8(b) and 8(d)) and as votes (8(e) and 8(f)). Through voting, users are generally given the opportunity to participate in the scenario, whereby the individual decision-making options are defined by the administrators, as their consequences can also vary. In addition, there are two types of voting systems in the current implementation: a boolean and a majority voting system. With the boolean vote, a decision can be placed between two choices, each of which can be linked to an EventSystem. This means that the user can decide, for example, whether they want to end a Scene prematurely because all preconditions have been fulfilled or whether they want to continue exploring the Scene. In the first case, the choice leads to a defined end of a Scene, whereas the second simply leads to the closing of the voting. At the contrary, majority voting gives all users a wider range of choices and the EventSystem of the most voted option is executed.

Figure 4

The feedback-Event are basically intended as temporary and persistent information for users in a Scene to inform them about occurrences such as the upcoming end of a round, a user entering or leaving the Scene or an error message. A more complex group are votes, which give users the opportunity to make a decision. In addition, all feedback Event ’s can trigger a new EventSystem.

Figure 4: The feedback-Event are basically intended as temporary and persistent information for users in a Scene to inform them about occurrences such as the upcoming end of a round, a user entering or leaving the Scene or an error message. A more complex group are votes, which give users the opportunity to make a decision. In addition, all feedback Event ’s can trigger a new EventSystem.

The use cases described in [1] can be extended in various ways with the dynamic event system () and the new functionalities presented. Whether in simulation-based learning, crime scene analysis, training and exercise, virtual museums or historical interactive reconstructions, the dynamic event system can lead to greater acceptance and deeper immersion. In addition, the possibility of voting (), especially for virtual museums and historical interactive reconstructions, offers additional user input that goes beyond conventional survey options.

Scenario Design

Social Experiments

This paper presented a dynamic event system with the ability to customize virtual environments by weather conditions and a feedback system with multi-user participation as a module of Va.Si.Li-Lab, so-called Va.Si.Li-ES. Compared to other collaborative VR applications, we argued that the requirements described in this paper are rather unfulfilled, which means that dynamic event systems and their use are generally underrepresented, creating a gap that can now be systematically closed with the Va.Si.Li-ES module. We have described use cases that are now available, all with a focus on reusability by users, not just developers. In order to create even greater dynamics it is planned to load the scripts dynamically, enabling new Trigger s and Event s to be created without changes in Va.Si.Li-Lab. In this context, a web-based REST interface for managing the individual EventSystem s is also to be implemented, which should be a straightforward process overall, as the database for this already exists with the EventSystem. Moreover, the capability of existing Trigger s not only being addressed inwards, i.e. into the Scene, and reacting to them, but also the possibility of external manipulation will be implemented. This includes that external or real-time events can be mapped geo-based to Scene s, so that e.g. the weather at a certain location can dynamically influence the scenario at runtime. In addition, the new features presented in Va.Si.Li-ES have identified enhancements that will be incorporated into future development cycles, with development transparently documented on GitHub. This will allow for future use in various application and research areas and opens a new horizon for the use of virtual HT systems. Unlike many existing applications, it enables collaborative interaction in a flexible environment where events can be customized as well as participation and annotation functions (by reusing VAnnotatoR). In this way, the number of use cases for our approach is growing, and now includes systems for flexible event modeling in simulations of natural environments.

The results of this paper have been prepared taking into account ethical aspects. We investigated Va.Si.Li-Lab in the context of its potential use as a platform for virtualizing a number of different applications. This concerned the use as a framework for HT systems for the design and dynamization of virtual environments, for the representation and interaction of multimodal content as well as for annotation and interaction with this content in a multi-user system. As Va.Si.Li-Lab currently exists as a prototype and its use has so far been carried out exclusively in supervised evaluations, it has been possible to ensure compliance with ethical standards. However, since Va.Si.Li-Lab is to be used as an open framework, a) future users – especially in scientific applications – must establish ethical control instances as well as b) have the possibility to block users in case of violations of the defined guidelines. This, of course, requires a moderation system and active moderation.

The limitations of Va.Si.Li-ES are primarily the missing externalized configuration possibilities through browser-based modification of scenarios, events and triggers.

Giuseppe Abrami, Alexander Mehler, Mevlüt Bagci, Patrick Schrottenbacher, Alexander Henlein, Christian Spiekermann, Juliane Engel, and Jakob Schreiber. 2023. Va.Si.Li-Lab as a Collaborative Multi-User Annotation Tool in Virtual Reality and Its Potential Fields of Application. In Proceedings of the 34th ACM Conference on Hypertext and Social Media (Rome, Italy) (HT ’23). Association for Computing Machinery, New York, NY, USA, Article 22, 9 pages. https://doi.org/10.1145/3603163.3609076

Giuseppe Abrami, Alexander Mehler, and Christian Spiekermann. 2019. Graph-based Format for Modeling Multimodal Annotations in Virtual Reality by Means of VAnnotatoR. In Proceedings of the 21th International Conference on Human-Computer Interaction, HCII 2019 (Orlando, Florida, USA) (HCII 2019), Constantine Stephanidis and Margherita Antona (Eds.). Springer International Publishing, Cham, 351–358.

Giuseppe Abrami, Alexander Mehler, Christian Spiekermann, Attila Kett, Simon Lööck, and Lukas Schwarz. 2020. Educational Technologies in the area of ubiquitous historical computing in virtual reality. Taylor & Francis. https://www.routledge.com/New-Perspectives-on-Virtual-and-Augmented-Reality-Finding-New-Ways-to-Teach/Daniela/p/book/9780367432119

Giuseppe Abrami, Manuel Stoeckel, and Alexander Mehler. 2020. TextAnnotator: A UIMA Based Tool for the Simultaneous and Collaborative Annotation of Texts. In Proceedings of The 12th Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 891–900. https://www.aclweb.org/anthology/2020.lrec-1.112

Ghaliya Al Farsi, Azmi Bin Mohd. Yusof, Awanis M. Romli, Ragad M. Tawafak, Sohail Iqbal Malik, Jasiya Jabbar, and Mohd Ezanee Bin Rsuli. 2021. A Review of Virtual Reality Applications in an Educational Domain. International Journal of Interactive Mobile Technologies (iJIM) 15, 22 (2021), 99–110. https://doi.org/10.3991/ijim.v15i22.25003

Gede Thadeo Angga Kusuma, I Made Agus Wirawan, and I Ketut Resika Arthana. 2018. Virtual Reality for Learning Fish Types in Kindergarten. International Journal of Interactive Mobile Technologies (iJIM) 12, 8 (Dec. 2018), 41––51. https://doi.org/10.3991/ijim.v12i8.9246

Somnath Arjun, GS Rajshekar Reddy, Abhishek Mukhopadhyay, Sanjana Vinod, and Pradipta Biswas. 2021. Evaluating visual variables in a virtual reality environment. In 34th British HCI Conference. BCS Learning & Development, London, UK, 11–22. https://doi.org/10.14236/ewic/HCI2021.1

Jerry Banks, I I John S. Carson, Barry L Nelson, and David M Nicol (Eds.). 2013. Discrete-event system simulation: Pearson new international edition (5 ed.). Pearson Education, London, England. 568 pages.

Kristopher Blom and Steffi Beckhaus. 2008. On the Creation of Dynamic, Interactive Virtual Environments. In IEEE VR 2008 workshop "SEARIS - Software Engineering and Architectures for Interactive Systems". Shaker, Reno, Nevada, USA, 4 pages.

Philipp Braun, Michaela Grafelmann, Felix Gill, Hauke Stolz, Johannes Hinckeldeyn, and Ann-Kathrin Lange. 2022. Virtual Reality for Immersive Multi-User Firefighter Training Scenarios. Virtual Reality & Intelligent Hardware 4, 5 (2022), 406–417. https://doi.org/10.1016/j.vrih.2022.08.006

Chen-Wei Chang, Shih-Ching Yeh, Mengtong Li, and Eason Yao. 2019. The Introduction of a Novel Virtual Reality Training System for Gynecology Learning and Its User Experience Research. IEEE Access 7 (2019), 43637–43653. https://doi.org/10.1109/ACCESS.2019.2905143

Vuthea Chheang, Patrick Saalfeld, Fabian Joeres, Christian Boedecker, Tobias Huber, Florentine Huettl, Hauke Lang, Bernhard Preim, and Christian Hansen. 2021. A collaborative virtual reality environment for liver surgery planning. Computers & Graphics 99 (2021), 234–246. https://doi.org/10.1016/j.cag.2021.07.009

Giuseppe Abrami Christian Spiekermann and Alexander Mehler. 2018. VAnnotator: a Gesture-driven Annotation Framework for Linguistic and Multimodal Annotation. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (Miyazaki, Japan, 7-12), James Pustejovsky and Ielka van der Sluis (Eds.). European Language Resources Association (ELRA), Paris, France, 5 pages.

Christopher T. Conner and Nicholas M. Baxter. 2022. Are You a Werewolf? Teaching Symbolic Interaction Theory through Game Play. Teaching Sociology 50, 1 (2022), 17–27. https://doi.org/10.1177/0092055X211053375

Berardina De Carolis, Catherine Pelachaud, Isabella Poggi, and Mark Steedman. 2004. APML, a Markup Language for Believable Behavior Generation. Springer Berlin Heidelberg, Berlin, Heidelberg, 65–85. https://doi.org/10.1007/978-3-662-08373-4_4

Andreas Dengel, Andrea Auer, Patrick Urlbauer, and Tim Läufer. 2022. Game-Based Teaching of Basic Hardware Components With an Educational Virtual Reality at Different Levels of Immersion. In Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1 (Dublin, Ireland) (ITiCSE ’22). Association for Computing Machinery, New York, NY, USA, 138–144. https://doi.org/10.1145/3502718.3524824

Maria D'Errico. 2021. Immersive Virtual Reality as an International Collaborative Space for Innovative Simulation Design. Clinical Simulation in Nursing 54 (2021), 30–34. https://doi.org/10.1016/j.ecns.2021.01.005

David Ferrucci, Adam Lally, Karin Verspoor, and Eric Nyberg. 2009. Unstructured Information Management Architecture (UIMA) Version 1.0. OASIS Standard. https://docs.oasis-open.org/uima/v1.0/uima-v1.0.html

Sebastian J Friston, Ben J Congdon, David Swapp, Lisa Izzouzi, Klara Brandstätter, Daniel Archer, Otto Olkkonen, Felix Johannes Thiel, and Anthony Steed. 2021. Ubiq: A System to Build Flexible Social Virtual Reality Experiences. In Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology (Osaka, Japan) (VRST ’21), Yuichi Itoh, Kazuki Takashima, Parinya Punpongsanon, Misha Sra, Kazuyuki Fujita, Shigeo Yoshida, and Piumsomboon. Tham (Eds.). Association for Computing Machinery, New York, NY, USA, Article 6, 11 pages. https://doi.org/10.1145/3489849.3489871

Scott W. Greenwald, Wiley Corning, Gavin McDowell, Pattie Maes, and John Winston Belcher. 2019. ElectroVR: An Electrostatic Playground for Collaborative, Simulation-Based Exploratory Learning in Immersive Virtual Reality. In 13th International Conference on Computer Supported Collaborative Learning, CSCL 2019, Lyon, France, June 17-21, 2019, Kristine Lund, Gerald P. Niccolai, Élise Lavoué, Cindy E. Hmelo-Silver, Gahgene Gweon, and Michael Baker (Eds.). International Society of the Learning Sciences, Lyon, France, 997–1000. https://repository.isls.org/handle/1/4597

Austin Hanus, Mindy Hoover, Alex Lim, and Jack Miller. 2019. A Collaborative Virtual Reality Escape Room with Passive Haptics. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, Osaka, Japan, 1413–1414. https://doi.org/10.1109/VR.2019.8798241

Elif Hilal Korkut and Elif Surer. 2023. Visualization in virtual reality: a systematic review. Virtual Reality 27 (2023), 1–34. https://doi.org/10.1007/s10055-023-00753-8

Alexander Leonhardt, Giuseppe Abrami, Daniel Baumartz, and Alexander Mehler. 2023. Unlocking the Heterogeneous Landscape of Big Data NLP with DUUI. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 385–399. https://aclanthology.org/2023.findings-emnlp.29

Chae Heon Lim and Seul Chan Lee. 2023. The Effects of Degrees of Freedom and Field of View on Motion Sickness in a Virtual Reality Context. International Journal of Human–Computer Interaction 0, 0 (2023), 1–13. https://doi.org/10.1080/10447318.2023.2241620

Weizhou Luo, Anke Lehmann, Hjalmar Widengren, and Raimund Dachselt. 2022. Where Should We Put It? Layout and Placement Strategies of Documents in Augmented Reality for Collaborative Sensemaking. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems(CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 627, 16 pages. https://doi.org/10.1145/3491102.3501946

Alexander Mehler, Giuseppe Abrami, Christian Spiekermann, and Matthias Jostock. 2018. VAnnotatoR: A Framework for Generating Multimodal Hypertexts. In Proceedings of the 29th on Hypertext and Social Media (Baltimore, MD, USA) (HT ’18). Association for Computing Machinery, New York, NY, USA, 150–154. https://doi.org/10.1145/3209542.3209572

Alexander Mehler, Mevlüt Bagci, Alexander Henlein, Giuseppe Abrami, Christian Spiekermann, Patrick Schrottenbacher, Maxim Konca, Andy Lücking, Juliane Engel, Marc Quintino, Jakob Schreiber, Kevin Saukel, and Olga Zlatkin-Troitschanskaia. 2023. A Multimodal Data Model for Simulation-Based Learning with Va.Si.Li-Lab. In Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management, Vincent G. Duffy (Ed.). Springer Nature Switzerland, Cham, 539–565. https://doi.org/10.1007/978-3-031-35741-1_39

Naoaki Okazaki, Sohei Aya, Santi Saeyor, and Mitsuru Ishizuka. 2002. A Multimodal Presentation Markup Language MPML-VR for a 3D virtual space. In Workshop on Virtual Conversational Characters: Applications, Methods, and Research Challenges. Melbourne, Australia, 4 pages.

Nikolaos Pellas, Andreas Dengel, and Athanasios Christopoulos. 2020. A Scoping Review of Immersive Virtual Reality in STEM Education. IEEE Transactions on Learning Technologies 13, 4 (2020), 748–761. https://doi.org/10.1109/TLT.2020.3019405

Veljko Potkonjak, Michael Gardner, Victor Callaghan, Pasi Mattila, Christian Guetl, Vladimir M. Petrović, and Kosta Jovanović. 2016. Virtual Laboratories for Education in Science, Technology, and Engineering. Comput. Educ. 95, C (April 2016), 309–327. https://doi.org/10.1016/j.compedu.2016.02.002

P. Rander, P.J. Narayanan, and T. Kanade. 1997. Virtualized reality: constructing time-varying virtual worlds from real world events. In Proceedings. Visualization ’97 (Cat. No. 97CB36155). IEEE, Phoenix, AZ, USA, 277–283. https://doi.org/10.1109/VISUAL.1997.663893

Marius Rubo, Nadine Messerli, and Simone Munsch. 2021. The human source memory system struggles to distinguish virtual reality and reality. Computers in Human Behavior Reports 4 (2021), 6 pages. https://doi.org/10.1016/j.chbr.2021.100111

Esther Sackett and Lisa M. Amoroso. 2024. A Little “Edutainment” Goes a Long Way: Leveraging Among Us®, a Popular Multiplayer Game, to Teach Persuasion Virtually. Management Teaching Review 9, 1 (2024), 7–21. https://doi.org/10.1177/23792981221104197

Shiri Savir, Adnan A. Khan, Rayaan A. Yunus, Taha A. Rehman, Shirin Saeed, Mahnoor Sohail, Aidan Sharkey, John Mitchell, and Robina Matyal. 2023. Virtual Reality: The Future of Invasive Procedure Training?Journal of Cardiothoracic and Vascular Anesthesia 37, 10 (2023), 2090–2097. https://doi.org/10.1053/j.jvca.2023.06.032

Manolis Savva, Angel X. Chang, and Pat Hanrahan. 2015. Semantically-enriched 3D models for common-sense knowledge. In 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, Boston, MA, USA, 24–31. https://doi.org/10.1109/CVPRW.2015.7301289

Sharad Sharma, Phillip Devreaux, David Scribner, Jock Grynovicki, and Peter Grazaitis. 2017. Megacity: A Collaborative Virtual Reality Environment for Emergency Response, Training, and Decision Making. Electronic Imaging 29, 1 (2017), 70–70. https://doi.org/10.2352/ISSN.2470-1173.2017.1.VDA-390

Jan Springer, Carsten Neumann, Dirk Reiners, and Carolina Cruz-Neira. 2011. An Integrated Pipeline to Create and Experience Compelling Scenarios in Virtual Reality. Proceedings of SPIE - The International Society for Optical Engineering 7864 (01 2011). https://doi.org/10.1117/12.879278

Wernhuar Tarng, Chia-Jung Chen, Chi-Young Lee, Chih-Ming Lin, and Yu-Jun Lin. 2019. Application of Virtual Reality for Learning the Material Properties of Shape Memory Alloys. Applied Sciences 9, 3 (2019), 21 pages. https://doi.org/10.3390/app9030580

Dimitrios Ververidis, Spiros Nikolopoulos, and Ioannis Kompatsiaris. 2022. A Review of Collaborative Virtual Reality Systems for the Architecture, Engineering, and Construction Industry. Architecture 2, 3 (2022), 476–496. https://doi.org/10.3390/architecture2030027

Krzysztof Walczak, Rafal Wojciechowski, and Wojciech Cellary. 2006. Dynamic interactive VR network services for education. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (Limassol, Cyprus) (VRST ’06). Association for Computing Machinery, New York, NY, USA, 277––286. https://doi.org/10.1145/1180495.1180552

Wan Mohd Rizhan Wan Idris, Rizhan Idris, Elissa Nadia Madi, Md Yazid, and Mohd Saman. 2011. Developing Dynamic Virtual Environments Using Hierarchical, Tree-Structured Approach. The International Journal of Multimedia & Its Applications 3 (04 2011), 17 pages. https://doi.org/10.5121/ijma.2011.3205

Tzong-Hann Wu, Feng Wu, Ci-Jyun Liang, Yi-Fen Li, Ching-Mei Tseng, and Shih-Chung Kang. 2019. A virtual reality tool for training in global engineering collaboration. Universal Access in the Information Society 18 (2019), 1243–255. https://doi.org/10.1007/s10209-017-0594-0

Description

Figure 5: The general architecture of Va.Si.Li-Lab: Multiple users can interact with and use the application based on different authorization groups. Various Room s can be created within Va.Si.Li-Lab, which are logically encapsulated and can be used by the user; virtual Scene s are loaded within these Room s, in which various environments can be simulated and used collaboratively as well as simultaneously by the users. Both the Scene and its parameters are defined within the document database, whereby only the EventSystem dynamic event management can be implemented at runtime (see Section 4.1). In Va.Si.Li-Lab, explicit annotations can be created by using VAnnotatoR, which is integrated for this purpose; implicit annotations are created based on the movements, gestures and actions as well as audio recordings of the users at runtime with the help of Ubiq and stored in the database. Ubiq is also used for a collaborative as well as simultaneous communication and interaction between users, whereby the Ubiq server has been extended for this purpose. Finally, the multimodal (e.g. speech, movement, object interaction, head position, eye movements) data collected implicitly can then be exported and processed using DUUI  [23]. Since DUUI is currently designed for the pre-processing of texts, an extension is necessary in this respect, although this is already technologically prepared due to the reutilization of UIMA [18].

An overview of the various UI elements in Va.Si.Li-ES for the creation of EventSystem s (a) – (c) and the selection of existing triggers (d) – (f).

Figure 6: An overview of the various UI elements in Va.Si.Li-ES for the creation of EventSystem s (a) – (c) and the selection of existing triggers (d) – (f).

Excerpts of the environment-EVENT: (a)–(b) shows the UI elements for the configuration, while the remaining figures show the results in Va.Si.Li-Lab.

Figure 7: Excerpts of the environment-EVENT: (a)–(b) shows the UI elements for the configuration, while the remaining figures show the results in Va.Si.Li-Lab.

An overview of the various UI elements of user feedback in Va.Si.Li-Lab.

Figure 8: An overview of the various UI elements of user feedback in Va.Si.Li-Lab.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

HT '24, September 10–13, 2024, Poznan, Poland

© 2024 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-0595-3/24/09.

DOI: https://doi.org/10.1145/3648188.3675154