Today, Human Smuggling [2, 38, 65, 83] driven illegal immigration via border crossing is the central issue faced by many countries [20, 27, 39, 52, 82]. U.S.-Mexico border crossing [18, 54, 65, 66] is often in the spotlight due to its high number of undocumented migrants [28, 43, 44, 45, 46]. Crossing the U.S.-Mexico border is a complicated and multifaceted issue that has been the focus of in-depth study and investigation [42, 55, 62]. Numerous studies, reports, and publications on various topics related to the U.S.-Mexico border crossing have been done by academics, policymakers, and practitioners [17, 33, 63, 79]. These topics range from migration patterns and trends to the effects of border enforcement policies on border communities’ social and economic dynamics.

The U.S. CBP (Customs and Border Protection) website reports that in November 2022, there were 206,239 interactions with migrants at the border, the second-highest monthly total since March 2020 [13, 30]. This number included Title 8 Apprehensions and Title 8 Inadmissible immigrants held in the United States, and Title 42 expulsions – immigrants sent back to their home country or last country of transit without delay because of public health issues [16]. The CBP website also gives information on the country, age, and family status of individuals apprehended at the border. About 35% of the migrants encountered were people living in a family unit, 25% were unaccompanied children, and 40% of the migrants encountered were single adults. During their journey, migrants often get sick, hurt, infected, or exhausted and may not have access to proper medical care or sanitation [6].

It is important to note that not only Mexicans are crossing the U.S.-Mexico border, but people from other countries [15], including Central American countries (Honduras, El Salvador, Guatemala, Nicaragua, Belize, and Costa Rica), South American countries (Brazil, Colombia, Venezuela, Ecuador, and Peru), and Caribbean countries (Haiti, Cuba, The Dominican Republic, and Jamaica) are also involved in this act.

Smuggling and trafficking of human beings is a primary concern faced by many countries, including the United States [83]. Smuggling is basically a service offered to assist people crossing the border illegally. Coyotes are smugglers who exploit and profit from vulnerable migrants who want to cross the U.S.-Mexico border illegally [11]. Along the route, they are known to lie to, take advantage of, mistreat, or abandon migrants [69]. They are also known for engaging in illicit operations, including drug trafficking, kidnapping, or human trafficking [14]. Contrary to what coyotes may advise migrants, unauthorized entry into the United States is punishable by arrest, detention, deportation, or prosecution [12]. Legal obstacles may also exist for immigrants seeking asylum or other forms of protection in the United States.

Many migrants who depend on coyotes are imprisoned, abducted, subjected to extortion, or abandoned to perish by dishonest criminal organizations [22]. Coyotes frequently employ unsafe techniques to cross the border [23]. They are also known to commit violent crimes like kidnapping and extortion [68]. In addition to endangering migrants, coyotes are also known to fuel the violence and crime linked to the border [14]. Coyotes continue to be a serious issue despite efforts by the U.S. government [12].

To counter coyotes and traffickers, a digital advertising campaign [12] dubbed “Say No to the Coyote” was created on the CBP website to discourage potential migrants in the Northern Triangle nations of Honduras and Guatemala from making the perilous journey to the U.S. border. The campaign sent a loud and unmistakable message: Contrary to what traffickers may have you believe, it is unlawful to enter the United States. The advertisement also referred immigrants to a landing page that described the unfavorable consequences of employing coyotes and offered more details that users could distribute via social media or other channels. While this campaign is helpful in countering coyotes to a certain extent, more direct approaches are required to automatically detect Coyote activities, especially on social media, and stymie their efforts. In this paper, we therefore present an approach to detect deceptive Coyote ads on social media automatically.

Social media platforms host a wide range of content, including text posts, images, videos, links, audio, emojis, and more. This diversity allows users to share personal updates, engage in discussions, post multimedia content, and interact with others. People also share advertisements as content on social media [80]. While many advertisements are legitimate, there are instances where ads can be fake or intentionally misleading [19, 37]. Some posts actively contain or endorse hate speech, discrimination, misinformation, and disinformation [8, 9, 61, 72]. All these can negatively impact users in several ways. Hence, social media platforms have strict content moderation guidelines [56]. Social media content moderation is the process of monitoring and regulating user-generated content posted on social media platforms [51]. The primary goal of content moderation is to ensure that the content posted by users complies with the platform's community guidelines, terms of service, and local laws [67]. Social media platforms actively encourage user-generated content while minimizing transparency in their decision-making processes and rationales for the moderation of objectionable messages [29].

Human smugglers are increasingly relying on technological platforms to reach potential migrants [4]. Coyotes are finding creative ways to use Facebook, WhatsApp, and TikTok to reach potential customers [12]. Coyotes frequently advertise on Facebook Marketplace and local buy-sell groups, where their offers to transport people across the boundary coexist with ads for items such as used cell phones and motorcycles [76]. Although Meta's policy 1 prohibits content that provides or facilitates human smuggling, this has done little to deter human smugglers from posting such ads on social media. Even the content policy of TikTok prohibits 2 posts that promote human smuggling, yet coyotes advertise passage to the United States openly on this platform, frequently offering their services in text over mundane videos of individuals conducting landscaping or construction work with no obvious connection to migration [76]. Moderating such content with human resources can be a challenging task due to the sheer volume of content being uploaded to these platforms.

Other than moderation policy and guidelines, there has been much research work done on identifying the different kinds of objectionable content [50, 64, 71, 73, 81]. For instance, Monit et al. [47] built a Geometric Deep Learning model to detect Fake News on Social Media, and Mozafari et al. [49] built a BERT-Based Transfer Learning Approach to detect Hate Speech on Social Media. Mossie et al. [48] similarly proposed a hate speech detection approach to identify hatred against vulnerable minority groups on social media. To the best of our knowledge, no prior research has been done on identifying coyote ads, as there is no readily available dataset because of their sheer deceptive and discrete nature. Additionally, these ads are mostly in non-English languages such as Spanish and Portuguese, so they are not easy to collect by English-speaking researchers. Inherent risks and ethical concerns associated with handling such sensitive information may also be a reason for the dearth of datasets. In this research, we address this research gap by building a novel dataset for this task and then exploring multi-modal classification methods to automatically detect coyote ads using our dataset.

The data collection spanned from February 2023 to August 2023. We gathered an overall 1500 coyote ads – 1000 coyote advertisement images and 500 coyote ad videos, from three different sources: the Web, NGOs, and regional contacts in South American nations. The main objective of data collection was to produce a varied collection of ads that captured diverse deceptive patterns used by coyotes in real-world scenarios. The ads were collected from popular social media platforms, including Facebook and TikTok, and we even scraped ads in news articles discussing human smuggling.

3.1.1 TikTok and Facebook data collection. We conducted targeted searches on TikTok using specific terms such as #bordercrossing, #coyoteads, and related variations to identify relevant advertisements. Concurrently, on Facebook, our exploration involved scanning pages, groups, and posts using keywords like “illegal border crossing” and “coyote services.” To further refine our search and target areas of heightened concern, we incorporated location-based keywords, including city names and specific border points. To access regional and local content effectively, we utilized multiple VPNs, ensuring that we could gather a comprehensive and geographically diverse dataset. This approach enabled us to capture a broad spectrum of advertisements, reflecting the various linguistic, cultural, and regional nuances associated with coyote and border-crossing ads in the targeted regions. The combination of targeted keyword searches, location-based refinement, and the use of multiple VPNs facilitated a thorough and detailed collection of relevant advertisements from both TikTok and Facebook, enhancing the richness and diversity of the dataset for our study.

3.1.2 News articles data collection. We expanded our dataset by extracting advertisements embedded within numerous news articles 4, providing a more comprehensive and realistic representation of coyote services. Specifically, we sourced ads from articles published by the Customs and Border Patrol (CBP) 5, which actively caution individuals against the deceptive tactics used by coyotes. This addition not only enriched the dataset but also increased its realism, enhancing the robustness of our models in distinguishing between legitimate travel/tourism ads and coyote advertisements. To ensure a balanced and accurate dataset, we also scraped legitimate ads featured in anti-coyote news articles, such as the "Say No to Coyote" initiatives 6 7. These additions not only bolstered the dataset's size but also facilitated the development of robust machine learning models capable of accurately differentiating between legitimate and illicit advertisements.

3.1.3 Collaboration with South American Students. To address the multi-lingual and culturally nuanced nature of ads, particularly in South American languages such as Spanish and Portuguese, South American students from our university were instrumental in the data collection and verification process. Leveraging their linguistic proficiency and cultural insight, these students employed South American VPNs to collect the datasets, ensuring that the advertisements were sourced from within the region. This approach was pivotal in capturing the authentic linguistic and cultural nuances specific to the South American context. The use of South American VPNs was essential in maintaining the integrity and relevance of the collected data. It facilitated the gathering of advertisements that are representative of the language and cultural nuances inherent to the South American audience, thereby enhancing the authenticity and reliability of the dataset. Furthermore, the South American students were actively involved in the meticulous verification of each advertisement in the collected datasets. Their linguistic competence and cultural understanding enabled them to accurately assess the context and linguistic subtleties of the ads.

3.1.4 NGOs and regional contacts. Engagement with non governmental organizations (NGOs) was facilitated by a university colleague who has longstanding collaborations with NGOs focused on border immigration issues. These NGO partners engaged directly with migrants at border facilities to gather the Coyote Ads for our study. Given the sensitive nature of the topic and out of concern for the safety of both the migrants and the NGO personnel, we have chosen not to disclose the specific methods by which data was collected with the assistance of these NGOs. To expand our reach and gather more diverse data, we collaborated with regional contacts. Our South American students, who have connections with researchers in Guatemala, Honduras, and El Salvador, played a pivotal role in this aspect of the data collection process. These regional contacts were instrumental in providing us with a substantial number of both Coyote Ads and non-Coyote Ads, thereby enriching the dataset and ensuring its comprehensiveness. The collaboration with these regional contacts and NGOs enabled us to access a wide range of advertisements from different sources and regions, enhancing the diversity and authenticity of the collected data. This collaborative approach, leveraging the expertise and networks of our university colleagues and South American students, was essential in obtaining a robust dataset that accurately represents the landscape of Coyote and non-Coyote Ads in the targeted regions.

3.1.5 Legitimate travel ads. We compiled a robust dataset of legitimate travel advertisements, comprising images and videos sourced from reputable travel websites, including Airbnb, Hotwire, Kayak, and Priceline. This collection aimed to create a comprehensive dataset of authentic travel promotions. To mirror the linguistic diversity present in the Coyote ads dataset, our legitimate travel ads encompassed non-English content, featuring both images and videos in Spanish and Portuguese. To align with the scale of the Coyote ads dataset, we curated a sample of 1,000 travel-ad images and 500 travel-ad videos, which included the anti-coyote ads from CBP. This brought the total number of legitimate ads in our dataset to 1,500. To ensure the quality and validity of the data, all legitimate ads underwent manual vetting by native speakers proficient in the respective languages. This rigorous validation process was crucial for maintaining the accuracy and authenticity of the dataset, thereby enhancing the reliability and effectiveness of our machine-learning models in distinguishing between legitimate advertisements and coyote ads.

Since none of the members of our research team were proficient in Spanish or Portuguese, we relied on external speakers to verify the data. Four South American students, fluent in their native languages of Spanish and Portuguese, meticulously reviewed and annotated the data. Two Spanish annotators independently labeled the Spanish advertisements to ensure linguistic accuracy. Similarly, two Portuguese students were tasked with annotating the Portuguese ads. The annotators independently verified each advertisement in the collected datasets, classifying them as either coyote or legitimate ads, and provided annotations accordingly.

Separate inter-rater agreement analyses were conducted for the Spanish and Portuguese advertisements to evaluate the reliability and consistency of the annotations. For the Spanish advertisements, the inter-rater agreement analysis resulted in a Cohen's kappa score 8 of 0.72, indicating a good level of consensus among the Spanish annotators. This score suggests a high degree of agreement beyond chance, reflecting the accuracy and consistency of the annotations. Similarly, the analysis for the Portuguese advertisements produced a Cohen's kappa score of 0.79, which signifies a substantial level of agreement among the Portuguese annotators. This score is indicative of good reliability in the annotations made by the Portuguese students. In instances where there was a disagreement between the two annotators, a third annotator proficient in both Spanish and Portuguese was consulted. This third annotator reviewed the disputed advertisements and provided a final decision on their classification as either coyote or legitimate ads. The final determination was made based on the majority voting, thereby establishing the ground truth for each advertisement.

We manually analyzed the collected coyote advertisements, both images and videos. Our analysis revealed many insights – using different languages, varying word counts, and diverse deceptive tactics. We also uncovered the key attributes of coyote videos, emphasizing their brevity, recurring themes, and persuasive strategies to attract those seeking migration and border-crossing assistance.

Figure 2: Language Distribution and Word Count of text in ads

3.3.1 Coyote image ads. We conducted an analysis of the data characteristics in the coyote ad images. Our analysis revealed that the content included diverse languages. These advertisements were featured in English and a combination of South American languages, specifically Spanish and Portuguese (see Figure 2a). On average, Spanish comprised the majority of the textual content, accounting for 83.18%. The percentage of English was 10.57%, while Portuguese constituted the smallest portion of 6.25%. The annotators observed that despite the lesser amount of Portuguese text, it had the ability to convey deceptive and covert meanings, adding a layer of deception to these adverts. Additionally, when we examined the word counts in the advertisements by plotting the box plot (see Figure 2b), we found a significant disparity. The lower 25% of the ads (Q1) had an average word count of 12, while the middle 50% (Q2) had an average of 18 words. Furthermore, the upper 25% (Q3) had a higher average word count of 24.75. Notably, the fourth quartile (Q4) stood out with a notably higher word count, averaging 38 words. This observed variance suggests a potential link between the length of a text and the sophistication level of false information. Our analysis of Flesch Readability score 9 statistics showed a wide range of scores, ranging from 5.5 in the first quartile to 14.7 in the fourth quartile. Some samples even had scores as low as − 2.7. Apart from readability, we also investigated subjectivity scores (computed using the Spacy TextBlob tool10) that quantify the level of subjective language, i.e., the presence of persuasive or biased content. The calculated mean subjectivity score was 0.30, with a standard deviation of 0.27. These findings provide additional evidence supporting the idea that the coyote ads have considerable objectivity in their content, thereby making them increasingly deceptive.

3.3.2 Coyote video ads. The compilation of coyote videos showcased several fundamental attributes. The videos exhibited a concise nature, often spanning a duration of 45 seconds to 2 minutes, effectively capturing the attention of viewers. A prominent recurring theme observed in these videos was the portrayal of individuals crossing the border or presenting their successful arrival in the United States. The videos frequently referenced specific geographical areas to build a sense of authenticity and enhance their trustworthiness. Additionally, a common pattern observed across the videos was the reliance on textual content, emphasizing cost-effectiveness and time efficiency. Figure 3 shows the most common words used in the ads. These videos employed convincing marketing strategies, such as promoting “Early Bird Deals” or “Limited Offers”, to closely resemble legitimate travel advertisements. Some videos included contact details, such as phone numbers, facilitating prospective clients’ ability to establish communication with the service providers. These attributes collectively reveal the coyote videos as straightforward and compelling instruments designed to appeal to individuals seeking aid with border-crossing pursuits.

3.3.3 Legitimate Image Ads. Our analysis of legitimate travel image ads revealed a focus on visual appeal and authenticity. The images predominantly featured iconic landmarks, pristine landscapes, or luxurious accommodations, which were the key selling points. The textual content was concise and informative, predominantly in English, though other major global languages like French, German, and Mandarin were also used, reflecting the international target audience. The text in these ads often included essential travel information such as destinations, pricing, and special offers, with an average word count of 15 words. These ads had a high Flesch Readability score, averaging around 60.0, indicating that the text was straightforward and easily understandable, intended to provide clear and direct information to the consumer. The subjectivity score in these ads averages around 0.10, suggesting a factual and less persuasive style compared to the more subjective deceptive ads.

3.3.4 Legitimate Video Ads. Legitimate travel video ads were longer, ranging from 1 minute to over 3 minutes, designed to fully engage potential travelers by showcasing the travel experience through dynamic and high-quality video content. The videos often began with sweeping aerial shots of destinations, followed by scenes of local culture, cuisine, and available activities, effectively conveying the atmosphere of the destinations. The narration in these videos was professional and focused on delivering a positive and enriching travel experience. Common phrases included “explore”, “discover”, and “experience”, aiming to evoke a sense of adventure and excitement. The videos also included practical information such as tour packages, booking details, and safety measures, ensuring that viewers received comprehensive information. Statistical analysis showed that these videos had a mean duration of approximately 2.5 minutes, optimizing engagement without overwhelming the viewer. The content's readability scores typically ranged from 50 to 70, balancing detail with accessibility.

Figure 3: Word cloud of most common words in ads

Many ad images and videos had textual information in them. We extracted all the textual information from images and videos using Tesseract OCR (Optical Character Recognition) [70]. In the case of videos, we loaded the videos first and then extracted the text frame-by-frame. We created two separate supervised datasets (X,y): one for images and one for videos. In the image dataset, X represents a combination of images and text, while in the video dataset, X represents the fusion of video content and extracted text. Each dataset was structured to associate these inputs X with the corresponding class variable y (1 if coyote, 0 if non-coyote). As mentioned earlier, we balanced our data, resulting in 2000 images – 1000 coyote ad images, 1000 travel ad images, and 1000 videos – 500 coyote ad videos and 500 legitimate ad videos. This strategy ensured an equal and fair representation of both categories in our dataset. We pre-processed the raw data before training the models in the following manner. The textual data was cleaned and pre-processed by removal of non-printable characters, punctuation, and extra spaces. Similarly, the image and video elements were resized to dimensions of 300x300 pixels. To ensure consistency, the pixel values of the images and videos were normalized to a standardized scale ranging from 0 to 1. We normalized the images and videos by dividing all the arrays representing the images and videos by 255. We set aside 80% of the dataset (randomly selected) for training the baseline models, while the remaining 20% was allocated for testing. In the case of LLMs, with the help of native speakers, we carefully selected 10 few-shot examples from both legitimate and coyote ads, and we evaluated the model on the rest of the data.

To assess the quality of our collected dataset comprehensively, we employed several state-of-the-art models. The selection of models was based on recent advancements and their demonstrated performance in related tasks, thereby ensuring a broad and up-to-date evaluation spectrum. This diverse selection covered a broad range of input modalities to provide a comprehensive assessment. For evaluating coyote image ads with text-only input (text extracted from the ads), we fine-tuned a range of models such as GPT-2 [58], Llama-2-7B [78], Gemma 2B, Gemma 7B [75], and Mistral 7B [35]. Additionally, we utilized LLMs such as GPT-3.5 Turbo [10], Gemini-1.0-Pro [74], Gemini-1.5-Pro [60], and GPT-4 Turbo [1], by prompting these LLMs with few shot examples to evaluate their performance. For coyote image ads with image-only input, we employed models such as Vision Transformers [25], Swin Transformers [41], ResNet-50 [31], 3D-CNN, and Gemini (Vision model). For coyote image ads with multi-modal inputs (image + text), we employed the models such as CLIP [57], TAPAS [32], Visual BERT [40], and a combination of Vision Transformers and BERT. Lastly, we assessed the coyote video ads by fine-tuning models such as TimeSformer [7], VideoMAE [77], and ViViT [5]. To achieve a comprehensive multi-modal representation of both the video content and its textual context, we incorporated these models with multilingual BERT (bert-base-multilingual-uncased) [24].

4.3.1 Coyote image ads. For fine-tuning models with text-only input, we initiated the input layer with a shape corresponding to the maximum text length across our dataset. Similarly, for fine-tuning models with image-only input, we initiated the input layer with an input shape corresponding to the image dimensions (300). We then loaded the pre-trained model weights and extracted the embeddings, which were then fed into the subsequent layers of the network. Between the input and output layers, we incorporated 2 dense layers with rectified linear unit (ReLU) activation functions to facilitate feature learning and abstraction for both models. After this, we used a flattened layer to transform the multidimensional data into a one-dimensional array, followed by a final dense layer with a Sigmoid activation function to produce the binary classification output, providing a probability score classifying the ad as either coyote or legitimate.

For fine-tuning models with multi-modal (image + text), we created two input layers: one for the text and another for the image. We employed pre-trained models for both the text and image modalities to extract meaningful embeddings. The extracted embeddings were then concatenated to form a combined feature vector, allowing the model to capture complex relationships between the text and image data. To improve the learning of features and their abstraction, we incorporated 2 dense layers into the model. After this, we used a flattening layer to transform the multi-dimensional feature vector into a single-dimensional array. Finally, the last dense layer utilized a Sigmoid activation function to generate a binary classification result, providing a probability score ranging from 0 to 1 to categorize the input ad as either a coyote ad or legitimate travel ad.

4.3.2 Coyote video ads. We first specified the input dimensions for video frames and textual data and then loaded the pre-trained baseline models into vision and multilingual BERT. Subsequently, we established input layers tailored for video frames and text and processed the inputs through their corresponding models. This yielded video features and text embeddings, which were then concatenated to form a comprehensive joint representation. This integrated representation ensured that the model could synergistically leverage both modalities, enhancing its capability to understand and classify videos accompanied by text. Also, to facilitate binary-class classification, we incorporated a dense output layer equipped with a Sigmoid activation function. The fusion of text and video modalities not only augmented the model's performance but also empowered it to excel in tasks demanding nuanced understanding and classification of videos with associated textual content, leading to significant enhancements in overall performance.

All the baseline models that we fine-tuned had the same training procedure. The model training process utilized a hardware configuration of an NVIDIA V100 GPU, which possessed a memory capacity of 128GB per node. We constructed the model using the Adam Optimizer and binary cross-entropy loss function, with a dynamically varied learning rate based on the model performance on validation data. The model was trained across 25 epochs, with a validation split of 0.2. Each epoch comprised 512 steps. During the model training process, we strategically employed a set of essential callback functions to optimize the training dynamics. To ensure that we retained the best model configurations, we utilized the ReduceLROnPlateau 11 callback, which dynamically adjusted the learning rate based on changes in the validation loss. When the validation loss plateaued for two consecutive epochs, the learning rate was reduced by a factor of 0.1, allowing the model to converge more efficiently toward an optimal solution. The minimum delta parameter to the callback was set to 0.0001, ensuring that only substantial improvements were considered and the learning rate was not reduced too frequently.

Furthermore, we specified a minimum learning rate of 0 using the minimum learning parameter, preventing the learning rate from going below this threshold. Lastly, to further enhance the model's performance and prevent overfitting, we utilized early stopping as another callback function. The EarlyStopping 12 callback was set to monitor the validation loss, and training was halted if the validation loss did not improve for 5 consecutive epochs. This helped in saving computational resources by stopping the training process when the model's performance on the validation data ceased to improve, indicating that the model had converged to its optimal solution. These well-considered callback functions collectively contributed to a finely-tuned and effective model training process, enhancing our ability to find the best model parameters.

The prompt settings remained consistent across the evaluation of LLMs for coyote image ads, encompassing text, image, and multi-modal inputs. The prompt incorporated balanced 10 few-shot examples of both coyote and legitimate ads. These examples were carefully chosen based on the suggestions of South American native speakers, who verified our collected dataset. These examples covered the diversity of deception involved in the ads, which would aid the LLM in classifying the ads precisely. After experimenting with various prompt settings, we configured the models with specific settings to optimize the output. A Temperature parameter of 0.5 was chosen to enhance the determinism and coherence of the responses, reducing randomness and ensuring consistent and relevant content. To maintain conciseness, the Max Tokens were limited to 50, allowing for focused and succinct information about the ad's content and intent. With a Top P (nucleus sampling) value of 0.9, the model selected from a broader range of likely tokens, ensuring contextually relevant outputs. A Frequency Penalty of 0.2 was applied to minimize repetition, encouraging the model to generate diverse and informative responses. This setting was useful for classification as it helps the model to provide varied and detailed classifications of the ads without unnecessary repetition.

Table 1: Evaluation metrics for coyote image ads.

The evaluation of various models provides insightful results into their performance and capabilities. In Table 1, for coyote image ad classification, the text-only models demonstrated a competitive performance, with Gemma-7B and GPT-4-Turbo achieving precision of 0.86 and 0.885, recall of 0.85 and 0.88, and F1 scores of 0.85 and 0.88, respectively. However, the image-only models trailed slightly behind the text-only models, with the Swin Transformers and Gemini achieving a precision of 0.87 and 0.88, recall of 0.84 and 0.85, and F1 scores of 0.89 and 0.87, respectively. Notably, the multi-modal models, both the text-only and image-only models, with the CLIP model achieving the highest precision of 0.94, recall of 0.93, and F1 score of 0.92. This suggests that the combination of text and image data significantly enhances the model's capability to distinguish coyote advertisements.

The superior performance of text-only models, as compared to image-only models, could be attributed to the deceptive meanings and nuances in the textual data that the models effectively captured. Textual data often contains subtle and implicit information, which language models are inherently designed to interpret and understand. On the other hand, image-only models rely solely on visual features and may struggle to decipher the deceptive elements present in the coyote advertisements. This is evident from the results, where the language models consistently outperformed the image-only models across all metrics.

The performance of various deep learning models trained on coyote video ad classification is summarized in Table 2. Notably, the TimeSformer model demonstrates the highest overall performance across multiple metrics. With a precision of 0.86, recall of 0.83, F1 score of 0.86, and accuracy of 0.85, the TimeSformer outperforms other models like ResNet-50, 3D CNN, Video Vision Transformers, and Video MAE. These results suggest that the dataset used for training these models is of high quality, as evidenced by the consistently high performance across multiple evaluation metrics. The robustness and effectiveness of the TimeSformer model, in particular, highlight the potential of the dataset to yield accurate and reliable classifications of coyote videos.

In our comprehensive analysis of the model classification of coyote image advertisements, we delved into an error analysis to uncover the nuances of the model's misclassifications. Figure 4 illustrates a striking observation: all images with accompanying text were correctly classified by the joint TEXT + IMAGE multi-modal model. However, both the TEXT-ONLY and IMAGE-ONLY models faltered in their classifications, pointing to the synergistic importance of both text and image data in discerning coyote advertisements.

Another noteworthy finding was that all advertisements containing Portuguese text were consistently identified correctly across all models. Even though Portuguese words constituted a smaller percentage of the overall text, this could indicate that Portuguese text might harbor a concealed deceptive meaning, which the models might have learned to recognize.

Figure 4: Examples that are correctly classified by the joint TEXT + IMAGE multi-modal model but incorrectly classified by the individual TEXT-ONLY and IMAGE-ONLY models

We noticed that some coyote image advertisements had vague or ambiguous language that is difficult to distinguish from legitimate advertising. For example, phrases like “great opportunities” and “unique offers” were commonly used in both legitimate and coyote ads. The models struggled with these ambiguities, leading to potential misclassification. Additionally, we identified a subtler issue in the classification of some ads as legitimate coyote advertisements. These ads contained buzzwords such as “Early Bird” and “limited spots”, which are commonly associated with legitimate advertising campaigns. This similarity to legitimate advertising language could potentially lead to misclassification, as the models might struggle to differentiate between legitimate and deceptive advertising tactics.

Furthermore, we observed that all the models exhibited a common pitfall of misclassifying general coyote-related news videos as coyote video advertisements. For instance, Figure 5 (in Appendix) depicts a video focusing on general information about the U.S.-Mexico border crossing, yet the models still misclassified it as a coyote ad. This misclassification could stem from the presence of similar words and phrases that are commonly found in both legitimate news content and deceptive coyote advertisements, highlighting the challenge of distinguishing between the two.

One of the obvious limitations of our research was the relatively smaller size of the dataset. As mentioned before, collecting such data is a very challenging task, and moreover, it involves a slow accumulation process. Also, the dataset mainly focused exclusively on coyote ad classification related to illegal border crossing between the U.S. and Mexico and did not include human smuggling advertisements globally. We restricted the dataset because of the constraints on the resources required to build the dataset. Even the data verification required relying on South Americans who speak Spanish and Portuguese, which was tedious; hence, we limited it to coyote-related ads to ensure the quality of the data. Each ad was labeled by only two annotators, with a third annotation used to resolve disagreements. This approach may have introduced potential biases, as varying interpretations among annotators could affect the consistency of the annotations, an issue noted in related research [26]. Our dataset may not represent linguistic and geographical diversity in global human smuggling advertisements. Researchers can use our baseline dataset and expand it based on their needs. We are constantly extending our dataset as new examples pour in from our sources.

Another limitation of our work was that we evaluated our dataset with a relatively small number of baselines. We acknowledge that there exists a diverse array of other multi-modal classifier architectures, LLMs, which we did not explore in this work. The third limitation of our work was that we did not have a working prototype, for example, a browser extension, that leveraged our models to automatically detect “in-the-wild” coyote ads on social media in real-time. We also admit that for simplicity and convenience, we built two separate models for image ads and video ads, but we could have built only one model that could handle both images and videos. Lastly, our work depended on third-party native South American language speakers to verify and label the data in our dataset. As none of the authors spoke either Spanish or Portuguese, we couldn't manually verify the data ourselves or the correctness of the labeling task.

For future research, we plan to investigate how well our approach can be applied in practice to various social media and chat groups via working prototypes. A practical deployment strategy includes collaboration with social media platforms and law enforcement agencies to embed the models seamlessly into their existing frameworks. For social media integration, API integration or plugin development could be explored, ensuring minimal disruption to user experience. Collaborative partnerships with law enforcement would involve the creation of streamlined reporting mechanisms and the integration of the model into their analytical tools. Regular updates and ongoing collaboration would be essential to adapt to evolving trends and maintain the effectiveness of the real-time detection system in addressing human smuggling across online platforms. We intend to learn about our models’ resilience and robustness by analyzing their performance on various platforms and scenarios.

Furthermore, investigating additional modalities like audio could further enhance the multi-modal framework and potentially increase the accuracy of coyote ad recognition. For instance, audio data may include auditory signals that are salient in coyote ads. Critical aspects that need further attention are interpretability and explainability. In sensitive areas where explanations are necessary, developing techniques to explain the choices made by our model will promote confidence and transparency. The models’ decision-making process can be clarified, and the discriminative elements influencing coyote ad detection can be found using approaches like attention visualization and feature attribution methods. Additionally, different methods [36, 59] can be employed to improve the performance of the models. Lastly, our novel dataset and proposed model to detect coyote ads is just an initial step towards addressing human smuggling on social media and the web. Developing a browser extension to detect and flag such ads on the web is the next step, which is in the scope of our future research.

Human smuggling is a significant global issue with serious consequences, including the exploitation of human lives, undermining of national security, and perpetuation of organized crime. Social media platforms have increasingly become tools for “coyotes” to advertise, lure victims, and coordinate illicit activities. This study contributes towards addressing this issue by developing a novel dataset and employing advanced multi-modal classification models to detect human smuggling ads on social media with high accuracy. The findings offer a valuable tool for law enforcement agencies and NGOs, enabling them to formulate and implement more effective strategies to combat human smuggling. By providing a data-driven approach to identifying and disrupting human smuggling networks on social media, the research helps these organizations allocate resources more efficiently and focus efforts on high-risk areas. Moreover, by mitigating illicit behaviors on social media platforms, the study contributes to creating a safer online environment.

The use of coyote ad detection algorithms for images and videos on social media platforms and the web fosters responsible web practices. The use of this measure guarantees that the adverts exhibited on these platforms are devoid of unsuitable, deceptive, or detrimental content, hence bolstering user safety and fostering trust. This is in accordance with the principles of responsible web practices, which promote the development of a digital environment that is both ethical and focused on the needs of users. The implementation of ad-content screening actively contributes to encouraging a responsible and user-centric online environment that places emphasis on factors such as quality, transparency, and the promotion of a good user experience.

The deployment of machine learning models for detecting human smuggling advertisements may raise several ethical considerations. Primarily, the potential for false positives could unjustly implicate innocent travel agencies or individuals, causing reputational damage and legal complications. Moreover, relying on AI for this sensitive task could lead to overdependence on automated processes, which may overlook the nuanced and evolving tactics of smugglers. Additionally, the collection and use of data from various sources, including social media and NGOs, necessitate strict adherence to privacy laws and ethical standards to prevent misuse of personal information and ensure respect for human rights.