34,361 deaths recorded. Not all the deaths occur at sea, but also in detention blocks, asylum units … Some entries have a name and a story, but the majority are anonymous data points – just over 1,000 are named … Some 400 have taken their own lives; more than 600 have died violently at the hands of others. - The Guardian, June 20, 2018

The conventional myth is that refugee deaths occur only at sea, but the above excerpt debunks the myth. Refugees face adverse environments not only in the Mediterranean Sea but also after crossing the Mediterranean Sea – if they are lucky. The sufferings of refugees continue at detention blocks or asylums. These struggles and subsequent unfortunate events, such as deaths, are pressing social concerns. To explore their traumatic journey, we draw insights from the seminal work 'Les Rites de Passage' (1909; The Rites of Passage) by Arnold van Gennep [33]. Gennep systematically studied the passage of individuals from one social status to another [33]. He has identified three distinct stages of this journey: separation, transition, and reincorporation. To the best of our knowledge, none of the prior AI-based studies have employed this framework to explore the transition of refugees from their home to host nation. A plethora of studies probed refugee-related issues using social media data, such as Twitter [2, 9, 17, 18, 21, 25]. While the relevance of social media data for extracting relevant and actionable information was widely acknowledged, refugee-related studies mainly probed refugee-related societal opinions around a specific crisis or event [25, 26]. These studies do not shed light on their journey from displacement to emplacement. BenEzer and Zetter [7] pointed out that 'the refugee journey is the defining feature of the exilic process: it is a profoundly formative and transformative experience and a 'lens' on the newcomers' social condition'. Thus, we employ Gennep's framework to study their journey.

We have identified four phases of the refugee journey: Arrival of Refugees, Temporal stay at Asylums, Rehabilitation, and Integration of Refugees into the host nation. It is worth noting that we have not attempted to track the journey of an individual refugee for ethical concerns but explored it at the macro-level. We collected multimodal tweets from April 15, 2020, to March 15, 2021. Initially, we have considered unimodal models such as Bidirectional Encoder Representations from Transformers (BERT) [12] + Long-Short Term Memory (LSTM) (for text inputs); and InceptionV3 [30], VGG19 [27], and ResNet [14] (for visual inputs). Next, we have employed an early fusion of these unimodal models, and our BERT+ LSTM + InceptionV4 [29] model has reported an F1-score of 80.93%. Subsequently, we collected multimodal tweets related to the 2022 Ukrainian refugee crisis and tested the generalizability of our proposed framework. Our findings, i.e., an F1-score of 71.88 % for this 2022 dataset, strongly indicate that the phases of the refugee journey from displacement to emplacement were identical.

Refugee issues are associated with a wide range of socio-economic consequences. A challenging environment in the host country can have adverse consequences on the psychological wellbeing of refugees [20]. On the other hand, the significant presence of refugees can impact the economic conditions of the host country [4, 28]. Thus, prior AI-based studies, such as [24, 25], have noted negative sentiments towards refugees. A significant portion of Twitter deliberations revolves around xenophobic and nationalistic themes. Kreis [21] has pointed out that the discourse of racism against refugees is mainly propounded by the nationalist-conservative and xenophobic right-wing political parties in the European context. However, in addition to this widespread antipathy and animosity towards migrants and refugees, Twitter-based studies also noted sympathy and solidarity activities by a certain section of the host nations [19]. In short, two dominant perspectives on the Twitter platform are - an apprehensive far-right perspective where refugees were considered terrorists or rapists, which leads to security and safety concerns in the host countries, and a sympathetic and humanitarian approach to helping refugees [17, 26]. Some Twitter-based studies also probed linguistic contents for analyzing migration movements across nations [32, 35]. Another stream of studies examined the emotional responses to refugee-related image(s). For instance, the death of the three-year-old Alan Kurdi in the Mediterranean Sea was one of the most unfortunate refugee-related events in recent times [2, 9, 15, 23]. This shocking image of the drowned Syrian kid appeared on the screens of 20 million people worldwide in less than 12 hours after the first release, and it became viral with more than 50,000 tweets per hour [34]. Adler-Nissen et al. [2] probed - how the above image influenced emotional responses and, subsequently, the impact of cumulative emotional responses on international relationships and foreign policy deliberations. They observed that the tragic incident had changed the discourse from an open-door approach to stopping refugees from arriving. Similarly, Bozdag [9] also noted that the public interest in refugees gained momentum after this tragic incident. To sum up, this brief review indicates that AI-based studies using social media data mainly probed societal opinions in the context of a specific refugee-related event, such as the terrorist attack in Paris, the sexual assault in Cologne, or the unfortunate death of Alan Kurdi [9, 15, 17, 23, 25, 26] – not the voices of refugees [18] or their journey. Thus, we address this research gap by identifying various phases of their journey from displacement to emplacement.

the passage from one country to another, and, still earlier, even from one manorial domain to another was accompanies by various formalities. These were largely political, legal, and economic, but some were of a magico-religious nature. For instance, Christians, Moslems, and Buddhists were forbidden to enter and stay in portions of the globe which did not adhere to their respective faiths. - Chapter 2: The Territorial Passage

foreigners cannot immediately enter the territory of the tribe or the village; they must prove their intentions from afar and undergo a stage best known in the form of the tedious African palaver. This preliminary stage, whose duration varies, is followed by a transitional period consisting of such events an exchange of gifts, an offer of food by the inhabitants, or the provision of lodging. The ceremony terminates in rites of incorporation – a formal entrance, a meal in common, an exchange of handclasps. - Chapter 3: Individuals and Groups

The above two excerpts from Gennep's antiquarian work indicate a striking resemblance between the century-old 'magico-religious' barriers and today's Islamophobia due to the middle east refugee crisis [33, 36]. Foreigners (i.e., refugees) cannot immediately enter the territory of the host nations. They need to 'prove their intentions from afar' to border forces, and there is a lot of 'palaver' involved in this process. However, during this period, refugees get food and lodging (i.e., asylums) from inhabitants (i.e., host nations). Finally, rites of incorporation can be equated to the social integration of refugees in the host nations. For instance, Castle and Diarra [10] note that the migration process of young Malians 'comprises social and psychological dimensions pertaining to the need to explore new places, experience new settings, and accumulate material possessions in order to conform to peer group aspirations.' They conclude that the migration of these young Malians 'is as much a rite of passage (emphasis added) as a financial necessity' [10].

Gennep's seminal work has identified three distinct phases, namely, pre-liminal rites (i.e., 'rites of separation from a previous world'), liminal (or threshold) rites (i.e., 'those executed during the transitional stage'), and post-liminal rites (i.e., 'the ceremonies of incorporation into the new world'). These three stages are also referred to as separation (getting detached from the previous world and loss of identity), transition, or liminal stage (the individual has got detached from her previous world and lost her old identity but not joined the new world), and incorporation (getting a new identity after incorporation into the new world). Monsutti [22] employed this analogy to explore the journey of young Afghans. In the initial phase, young Afghans get separated from their families and homes. In the next phase, 'they have to prove their capacity to face hardship and to save money … represents a period of liminality', and finally, in the reincorporation phase, they 'return to their village of origin … as adult marriageable men' and mostly they continue to commute between Afghani-stan and Iran for the rest of their life [22].

We also employed the above three stages for probing the refugee journey, which starts after they plunge into the ocean. If they are lucky, they arrive at their desired destination. We have conceptualized pre-liminal rites as the 'Arrival of Refugees'. Refugees are getting separated from their previous world, getting detached from their families, and taking a leap of faith through risky sea routes as anonymous data points after losing their identity. BenEzer and Zetter [7] pointed out that the 'mode of travel may influence the meaning of the journey and its impacts on the individual … For someone who has not … crossed the sea before, the mode of travel will be a highly symbolic part of the experience of the journey’. Thus, for this phase, we have considered tweets deliberating or sharing information about the mode of travel or tweets related to border control forces.

We have considered two interrelated but distinct stages of liminal (or threshold) rites as follows: 'Temporal stay at Asylums' (second phase) and 'Rehabilitation of Refugees' (third phase). Asylums are 'a place of social dissolution and a place of new beginnings where sociality is remoulded in new ways' and it is crucial to 'explore the precarity of life in the camp … in this temporary space (emphasis added)' [31]. Accordingly, our tweets in this category capture the living conditions in refugee asylums and images of asylums and camps. In asylums, refugees need health support because many refugees arrive with health problems. They 'face continued disadvantage, poverty and dependence due to lack of cohesive support in their new country … This is compounded by language barriers, impoverishment, and lack of familiarity with the local environment and healthcare system' [16]. Thus, tweets from this third phase share information and images of various support activities like arrangements of medical aids, donations of food items or garments etc.

We have conceptualized the post-liminal rites as the 'Integration of Refugees' into the society of the host nation. Charitable organizations arrange various support activities, such as helping them to learn a new language. For instance, Syrian refugees in Germany try to learn German for better social inclusion [1]. Hence, this category of tweets shares information about the arrangement of the education system for refugee kids or vocational training programs for adult refugees. These skill up-gradation activities help refugees to settle down in the host nations [6]. Finally, we have identified another distinct type of multimodal tweet where text content might be related to the above phases, but image content shares refugee-related statistics or data points through graphical images or charts. These tweets can be crucial for information dissemination, and we have labeled this category of tweets as 'Infographics'.

We note that prior refugee-related studies mostly considered unimodal inputs, i.e., either text [24, 25] or visual contents [15, 23]. However, multimodal approaches are generally more efficient than unimodal approaches [8, 13], but none of the prior refugee-related studies considered multimodal inputs to the best of our knowledge. Thus, we consider multimodal tweets. We have employed Twitter's search API, and considered a set of keywords like refugee, refugee camps, refugee asylums, migration, migration policy, etc. We have collected 3.98 million refugee-related English tweets from April 15, 2020, to March 15, 2021. Next, we dropped tweets without images, duplicate tweet-ids, and tweets with similar text contents. Our corpus size became 0.23 million multimodal tweets (i.e., 5.7% of 3.98 million).

For resource limitations, we randomly selected around 2500 tweets from our corpus of 0.23 million tweets. For annotating tweets related to 'arrival of refugees', we have considered the following aspects: arrival through sea routes, risk of traveling through sea routes, mode of transport, and activities by border control forces. Similarly, the 'rehabilitation of refugees' phase has considered the activities such as arranging medical aid, charity activities by non-government organizations (NGOs), or facilitating donations of essential livelihoods. We have carefully analyzed each tweet based on its text content and the associated image for assigning the final class. We note that most tweets are about refugees by charitable organizations or volunteers, and only a tiny portion of our sample captures the voices of refugees [18, 19]. Also, we discarded tweets with poor images or cryptic short texts. Two annotators manually annotated 1722 tweets with distinct text-image pairs with an inter-rater reliability of 0.84. These 1722 tweets are distributed as follows: Phase 1: 398 tweets; Phase 2: 387 tweets; Phase 3: 289 tweets; Phase 4: 343 tweets; and Infographics: 305 tweets. We randomly split our 1722 annotated tweets into 80% (as a training dataset) and 20% (as a test dataset) for our analysis. Table 1 reports a few representative tweets from our corpus.

Our sample size and proportion of multimodal tweets are in accordance with prior studies [3]. For instance, Alam et al. [3] also found only 4.8% of multimodal tweets from a total corpus of 3.5 million tweets for Hurricane Irma. One potential limitation of considering multimodal tweets is that we discard more than 90% of tweets from our initial corpus. So, we have carefully explored these discarded tweets and found these tweets are mostly capturing societal opinions - ranging from sympathy to antipathy [19]. Some tweets are implicitly related to various phases of the refugee journey. For instance, we note that hate tweets towards refugees are primarily associated with refugee arrivals. Similarly, some of the sympathetic tweets are related to rehabilitation and integration. Overall, we find that multimodal tweets are more informative than unimodal tweets in our corpus, which justifies our sample selection. Additionally, we note that common objects (except the Infographics class) are human beings for a significant portion of our visual inputs, i.e., images of refugees arriving through sea routes, or images of refugees with border control forces, or images of refugees at asylums, or images of refugees receiving donation or supports. Thus, in comparison to other standard image datasets (which are comprises of distinctly different objects like flowers, animals, or buildings), our image dataset is challenging for computer vision algorithms.

We have initially considered unimodal models for text, i.e., BERT + LSTM, and visual inputs, i.e., InceptionV3 [30], VGG19 [27], and ResNet [14]. Subsequently, we have also employed an early fusion of the above unimodal models. In LSTM, the input sequence feed is only in the forward direction. However, BERT models consider the contextual representation of a word to decipher the difference between two contexts [12]. Consequently, LSTM models with BERT embedding are more efficient in capturing the contextual representation than LSTM models without BERT embedding. We have considered the English language uncased base version of the BERT model with 12 hidden layers, 768 hidden sizes, and 12 self-attention heads. CNN-based algorithms, comprised of convolution layers, max pooling, and fully connected layers, are most efficient for solving complex computer vision tasks. Like language encoders, visual encoders also extract the dominant visual features from an input image and map it into pre-defined categories for the downstream image classification task.

For the VGG-19 model, we have considered the weights based on pre-trained models, but we have trained the output layer using our dataset for the final classification task [27]. Next, we consider ResNet [14] or Residual Networks models. These models skip connection to add the output from a previous layer to a later layer, which tackles the vanishing gradient problem. We have used ResNet with its ImageNet pre-trained weights to initialize the model, and the outputs are followed by dense layers of 256 units and a SoftMax layer [11]. Our third CNN architecture-based model is InceptionV3, which is known for its efficiency in interpreting images and detecting objects [30]. We have also considered the InceptionV4 model for multimodal fusion [29].

Multimodal models leverage the advantages of both modalities by combining two parallel deep neural architectures (i.e., natural language processing for textual inputs and computer vision techniques for visual contents) for the downstream task. We have employed an early fusion-based approach, i.e., integrating features directly after they are extracted, on the outputs from these two models to get the final class from a multimodal tweet. Early fusion allows to perform multimodal representation learning—as it can learn to exploit the correlation and interactions between low-level features of each modality. In other words, this joint representation is concatenating different modality features at a low level [5].

We resized the pictures according to model specifications. For classification, we partially freeze the pre-trained weights and add a dense layer with a dropout layer (p = 0.4), followed by a linear layer to extract the latent features. Next, we have added a dense unit of 128 before concatenating it with the language model stack. We pass both visual stack and language stack through a shared dense layer of size 128 and concatenate the outputs to form a joint vector of length 256. We also apply a dense layer of size 256 before the final SoftMax layer and ReLu activation function. This leads to the final classification layer, i.e., a dense layer with 5 units (for our 5 categories) and SoftMax activation, which will give the predicted class of multimodal inputs. Next, we have used stochastic gradient descent (SGD) for model optimization. For robustness, we have used different learning rates for better learning. We train the model using an SGD optimizer starting with a learning rate equal to 0.001 and then decreasing it using the Reduce Learning Rate on Plateau from keras which automatically changes the learning rate if there is no improvement in training after a certain number of epochs. We have considered the cross-entropy loss function for our modeling because we have multiple classes in our dataset. Table 2 reports the performances of our models using 5-folds cross-validation. For brevity, we have not reported the t-SNE plot, but this plot suggests that the weak performance of BERT + LSTM is primarily due to the Infographics class, where images differ distinctly from other categories but not the text content. As expected, multimodal models, especially BERT+ LSTM + InceptionV4, have outperformed other models.

The previous section leads to the intriguing question: is our proposed framework of four phases generic or context-specific? To probe the practical relevance of our approach, we have considered the 2022 Ukrainian refugee crisis due to the Russian invasion. More than 4 million Ukrainians had to move to neighboring countries, such as Poland, Romania, Hungry, or Moldova, in a span of one and half months. Another 6 million got displaced within the country because of this military invasion [37]. Ukrainians got displaced from their previous world and left behind all their belongings (and sometimes family members) towards an uncertain future.

We have collected 0.6 million tweets, out of which 0.01 million tweets were multimodal, during February 24, 2022, to March 15, 2022. Our data collection and pre-processing strategies were similar to the previous analysis, but for data crawling - we have added a few crisis-specific keywords such as Ukraine, Russia, and so on. Next, we have explored the linguistic and image contents of this corpus. Some of the visible differences are - refugees mostly took the sea routes in the previous corpus, whereas it was mostly rail or road transport during the 2022 crisis. Earlier the travel risk was boat capsizing, and in 2022 it was a missile attack. Similarly, we find images of camps and tents in the previous corpus, whereas it was a temporary makeshift arrangement in hotels and other large public buildings during the 2022 crisis. Broadly, the struggles, risks, and traumas of refugees are identical. A few representative tweets in Table 3 confirm the same. Thus, the intriguing question is: can we apply our proposed framework during this 2022 crisis? If yes, the follow-up question is – whether the previous corpus collected prior to this 2022 crisis (as training data) can help us to extract actionable and relevant information in real-time during the Ukrainian crisis? To test the same, a single annotator (i.e., one of the authors with prior experience of handling migrant-related tweets) has annotated 234 multimodal tweets (evenly distributed across four phases and the infographics class) and employed our best performing model, i.e., BERT+ LSTM + InceptionV4, on this unseen test data. For a complex unseen dataset like ours, we find that the F1-score came down to 71.88%. As expected, F1-scores of BERT+ LSTM (for unimodal text inputs) and VGG19 (for unimodal visual inputs) models are 50.01% and 59.17%, respectively. Unlike the previous corpus, this 2022 corpus also elucidated the evolution of Twitter deliberations. For instance, initial tweets in February were mainly about the arrival and temporal stays, and in the later period, tweets related to rehabilitation and integration gained momentum.

We have employed the Les Rites de Passage framework to elucidate the societal level transitions of refugees from home to host nations. Our study confirms that this framework can be used to extract relevant and actionable phase-wise information from social media platforms. We note, especially in the context of the 2022 Ukrainian crisis, that refugee needs evolve from essential health support in the initial days to social integration in later days. In the domain of applied AI-based studies, our study may aid policymakers in understanding phase-wise concerns and taking appropriate actions. We conclude that it hardly matters whether someone is taking the risky sea route to reach the shore of the host nation or opting for the rail route or migrant caravan to enter the host nation. The traumatic journey from displacement to emplacement is the same– irrespective of who they are or wherever they come from!