The problems of a single ranked list tuned to the “best overall” interests have been long recognized in research on recommender systems. Historically, most attempts to address this problem focused on producing a “better” one-shot list or recommendations rather than treating the recommendation process as a human-AI collaboration. One of the oldest streams of related work is the research on increasing the diversity of recommendations[43]. This research was motivated by similar observations - stuffing the top of the recommended list with very similar items is not productive even though all these items have top relevance score. This research stream introduces diversity as one of the measures of recommendation quality and explored a range of approaches to diversify the ranked list[25, 38]. A more recent approach known as context-aware recommendation acknowledged that user's current preferences are defined not just by the overall interest modeled by traditional recommender systems, but also on immediate needs and interests defined by the context. However, instead of collaborating with the user in understanding this context, context-aware recommenders relied on AI alone to model both the overall interests and the context to produce a one-shot “context-adapted” ranked list[1, 2, 37].

In contrast, closely related streams of research on “conversational”, “dialogue”, “critique-based”, and “interactive” recommendation specifically focused on human-AI collaboration. The oldest stream of this work suggested structuring the recommendation process as a human-machine natural language dialogue where a human provides feedback (known as critique) to each (and originally single) suggestion of a recommender system[8]. This feedback allows the system to better understand users’ current preferences and gradually improve recommendations. More recent research generalized this critique-based approach and attempted to make it more efficient using graphical user interfaces (GUI) in place of the natural language dialogue[10]. The use of GUI made the interaction process more efficient by recommending a list of items at each step and enabling the users to contribute more information through a more complex compound critiques[34]. Modern interactive recommender systems[23] enhanced the use of GUI making it a centerpiece of human-AI interaction. Instead of a turn-taking dialog with a recommender agent, these systems offered users an opportunity to tune the results of recommendation continuously using sliders and other forms of direct manipulation[4, 6, 30, 31, 40].

Carousel-based recommender interfaces offer an interesting compromise between these two streams of research. Carousel-based interfaces inherit serious attitude to ranking from the AI-focused stream of research on recommender systems. At the same time it follow the stream of work on human-AI interaction where each partner does what they could do best. It is left for the human user to choose the current topic(s) of interest (represented by one or more offered carousels), an easy task for the user but a really hard one for AI that even good context-aware recommenders can't do reliably. It is left for the AI component to rank the topics of interests and the items within each selected topic, the task where AI ability to carefully analyse user interaction with the system supersedes user abilities.

With the growth of size and complexity of hypertext networks, the research on navigability emerged as an important topic in the hypertext and Web research. The navigability research usually focuses on the ability (or probability) to reach specific designation(s) when navigating various kinds of hyperspaces with or without help of navigation artifacts. The first generation of research on navigability focused mostly on understanding the navigation properties of the Web as a novel information artifact by itself[7, 27]. The results of this research were critical to better understand the nature of the Web and to develop more efficient search and navigation approaches. With the gradual development of various navigation artifacts (structures) such as concept indexes for concept-based navigation[11], semantic categories for faceted browsing[42], and tags for tag-based navigation[17] the navigability research refocused on studying the navigation properties or these artifacts and comparing different approaches to create these artifacts from navigability prospect. For example, Venetis et al.[39] compared navigability of different approaches to generate navigational tag clouds, Trattner et al.[36] compared navigability of regular and faceted tag clouds, and Helic et al.[24] examined the effect of automatic linking on navigability. This second generation of navigability research plays an important role in developing more efficient types of navigation artifacts and finding best approaches to generate them.

To explore navigability, researchers use both theoretical and empirical approaches. For example, Chi and Mytkowicz[12] examined the navigability of tag clouds from information-theoretical prospects while Venetis et al.[39] performed this analysis empirically. In turn, empirical evaluation could be performed using data-driven simulation[33, 39] or through a user study[36, 42]. While a user study is considered the most realistic approach to evaluating navigability, the complexity of organizing large-scale user studies leads to the domination of data-driven empirical approaches[7, 17, 33, 39]. The key idea of data-driven evaluation is to simulate user navigation in the information space with or without navigation artifacts using realistic models of user behavior. Navigability of information spaces could be considered as the simplest case of data-driven evaluation since it typically uses simple behavior models such as random walks[7] and requires no artifact generation. Personalization and navigation artifacts introduce additional complexity to data-driven evaluation. Personalization requires engaging realistic user data, for example, user bookmarks were necessary to evaluate personalized PageRank[22]. Navigability evaluation of navigation artifacts requires building more complex behavior models to simulate user interaction with these artifacts and using real data to build realistic artifacts, for example, realistic tag clouds[33, 39].

In this work, we explore navigability of a carousel-based recommendation interface, a new example of navigation artifact in the information space, by comparing it with traditional ranked list. To perform a comprehensive evaluation, we use both an analytical algorithmic-theoretic approach and an empirical data-driven approach. Since we study navigability of personalized navigation artifacts, we have to build a realistic model of user carousel interaction and bring in real user data to generate realistic interfaces and simulate user behavior in a personalized way.

Interaction of users with ranked lists of recommended items has been long studied under the name of click models [13]. Many click models exist [3, 9, 15, 19, 20, 32]. Essentially all of them try to explain the user behavior by a generative model, which can be learned from data. As an example, the cascade model [15, 32] assumes that the user examines the list of recommended items from top to bottom until they find an attractive item. After that, they click on that item and leave satisfied. This seemingly simple model explains the position bias in recommender systems, that lower-ranked items are less likely to be clicked than higher-ranked items. This information can be used to debias logged data [29], or to learn better ranking policies either offline [13] or online [14, 28]. In this work, we study a generative model of user behavior that explains why interactions with carousels can be more efficient than with a single ranked list.

To quantify the benefit of carousels, we formalize the problem of carousel recommendation using a mathematical model, which we call a carousel interaction model. We have a matrix of m × n recommended items, where m is the number of rows (carousels) and n is the number of columns (items per carousel). Each carousel is associated with some topic, such as a movie genre. To simplify exposition, we assume that each item belongs to a single topic. We refer to the item at row i ∈ [m] and column j ∈ [n] as (i, j).

The user preferences are defined by two sets of probabilities. The first are topic preferences. Specifically, p i ≥ 0 is the probability that the user is interested in topic i, for any i ∈ [m]. The second set are topic-conditioned item preferences. Specifically, p j∣i ≥ 0 is the conditional probability that the user is interested in item j given that they desire topic i, for any i ∈ [m] and j ∈ [n]. We assume that $sum{i = 1}^{m} p{i} = 1$, and that $sum{j = 1}^{n} p{j mid i} = 1$ for any topic i ∈ [n].

The user interacts in the carousel model as follows. First, the desired topic and item in that topic are realized in the mind of the user, and then the user seeks them. In particular, the desired topic is sampled as $I sim C a t left(right. left(right. p{i} left.right){i = 1}^{m} left.right)$ and the desired item is sampled as $J sim C a t left(right. left(right. p{j mid I} left.right){j = 1}^{n} left.right)$, where Cat(θ) is a categorical distribution with outcome probabilities θ. In plain English, exactly one topic is chosen with probability p i, and exactly one item is chosen with probability p j∣I conditioned on that topic. An equivalent way of thinking of this process is that exactly one (i, j) is chosen with probability p i, j = p j∣i p i. The user seeks item (I, J) as follows. They start by examining the first carousel. If its topic does not match that of I, they proceed to the next carousel. The user examines all carousels, from top to bottom, until they stop at carousel I. After that, the user examines the items in carousel I, from left to right, until they find the desired item, in column J. A flowchart of how the user interacts with the carousels is presented in fig. 2. The two End of Session nodes on the bottom left and right represent different scenarios under which the user might leave the system. The session may end after the user successfully find the desirable item (left) or because none of the items are desirable for the user and there is no more item and topic to examine (right).

Figure 2:Interaction between the user, topics, and items in the carousel interaction model.

The complexity of interacting with the carousels can be measured in many ways. We define it as the number of examined carousels and items until the desired item is found. Specifically, if the desired item is (I, J), the number of interactions is I + J. Therefore, the expected number of interactions of the user with the carousels is $sum{i = 1}^{m} sum{j = 1}^{n} p{i , j} left(right. i + j left.right)$. To illustrate this formula, suppose that each item is desired with probability p i, j = 1/(mn), which can be attained by setting p i = 1/m and p j∣i = 1/n. Then the expected number of interactions is

This result is notable for the following reason. Although we recommend mn items, and they are equally popular, the expected number of interactions to find the desired item is only O( m + n). This is because the items are organized and examined in a structured manner.

We would like to comment on two aspects of the model. First, the expected number of interactions is minimized when the carousels are sorted in descending order of p i, and the items in each carousel are sorted in descending order of p j∣i. Second, p i and p j∣i are typically unknown and have to be estimated from data. We do that in our experiments in section 5.

To show improvements over a single ranked list, it is useful to think of the matrix of m × n recommended items in section 4.1 as a single ranked list, which is examined row by row. The user starts at position (1, 1). If that item is not desired, the user proceeds to the next item (1, 2). The user examines row 1, from left to right, until the desired item is found or the end of the row is reached. If the end of the row is reached, the user moves to item (2, 1), the first item in the next row. Then the user examines this row, from left to right, and this process continues until the desired item is found. A flowchart of how the user interacts with a ranked list is presented in fig. 3.

Figure 3:Interaction between the user and items in a single ranked list.

Similarly to the carousel model, we measure the complexity of interaction by the number of examined items until the desired item is found. This means that the number of interactions to find item (i, j) is n(i − 1) + j. This immediately leads to the following observation. For any item (i, j) where i ≥ 2, the number of interactions in the carousel model, i + j, is lower than in the single ranked list, n(i − 1) + j, when the number of columns is n ≥ 2. This is because

when i ≥ 2 and n ≥ 2. The number of interactions for items in row 1 is higher in the carousel model because the user has to examine the carousel topic before it examines the items.

The above result suggests that the carousel model can lead to major improvements when most desired items are not concentrated in the first carousel. To show this, we consider the example from section 4.1 where each item is desired with an equal probability 1/(mn). In this example, the expected number of interactions in the ranked list is

In comparison, we get O( m + n) interactions in the carousel model. Under the assumption that m ≈ n, the savings in the number of interactions are the root of the number recommended items, $O left(right. sqrt{m n} left.right)$.

We use the MovieLens 100K Dataset [21] which consists of 100,836 ratings applied to 9,724 movies in 19 genres by 610 users. In our experiments, we only utilize the information about the user ratings and movie genres. We apply a pre-processing step to remove movies with no genres. A total number of 34 movies was removed from the dataset through this process.

We assume that the user adopts two distinct browsing behavior when seeking movie (I, J) providing that the results are presented as a single ranked list or a set of carousels. These two browsing behaviors are explained in sections 4.1 and 4.3, respectively.

To generate the recommendations, we consider two sets of probabilities. The topic preferences and the topic-conditioned item preferences. The preferences are computed as follows. The dataset of ratings is a set of tuples $mathcal{D} = left(left{right. left(right. k{t} , j{t} , r{t} left.right) left.right}right){t = 1}^{n}$, where k t is the index of the user in data point t, j t is the index of the rated movie in data point t, and r t is the corresponding rating. The topic-conditioned item preference reflects how representative the movie is of a genre. We compute it as the sum of all ratings of the movie over the sum of all ratings in its genre. Formally, let $mathcal{G}{i}$ be the set of all movies in genre i. Then for any movie $j in mathcal{G}{i}$, the topic-conditioned item preference of movie j in genre i is

We set p j∣i = 0 for any $j notin mathcal{G}{i}$. For any user k, the topic preference reflects how much the user prefers a genre. We compute it as the sum of all ratings of the user in a given genre over all ratings by that user. Formally, the topic preference of user k for genre i is

Having the user profile assigned to each user, we generate two sets of recommendations as follows: For the first set of recommendation for carousels, we use the topic preferences to sort them and then populate each one with movies using the topic-conditioned item preferences. This approach generates a set of carousels each representing a genre (19 carousels for 19 genres in the dataset). Each carousel contains all the movies within the representative genre. With an average of more than 475 movies in each genre, we assume that is a realistic enough scenario for the user to be able to scroll down or right and examine all items and find the desirable movie. The movie are sorted by their scores, where the score of movie j is $sum{i = 1}^{m} p{i , j}$. Due to the sheer number of movies in the dataset, we assume that users will be able to scroll down in the list to find what they are looking for. In this evaluation, the user profile and the recommendations were not affected by further user interactions and remained unchanged throughout all sessions.

We define a session as a single instance of evaluation in which the user seeks a movie (I, J) from the set of recommended results, which can be displayed as a single ranked list or carousel interaction model. The process of simulation is as follows: For each setting, we first generate two sets of recommendations (one using single ranked list and another using carousel interaction model) for every user in the dataset. Next, we ran 100 independent sessions for every user that includes selecting a genre, selecting a movie within that genre, and calculating the number of interactions required to reach that movie in both models. We consider the average value of these 100 sessions as the outcome of the experiment for a given user in a given setting.

To simulate user navigation in each session, we assume that the desired genre and a movie in that genre are realized in the mind of the user. The desired genre is sampled as $I sim C a t left(right. left(right. p{i} left.right){i = 1}^{m} left.right)$ and the desired movie is sampled as $J sim C a t left(right. left(right. p{j mid I} left.right){j = 1}^{n} left.right)$. This process is described in detail in section 4.1. In each session, the user is only interested in a single genre and a single movie within that genre.

There are many ways of measuring the complexity of interacting with the recommended items in single ranked list and carousel interaction model. We employ two metrics to evaluate our proposed approach. First, we define navigation effort as the number of examinations by users until the desired item is found. These examinations include browsing genres as carousel topics and movies as items. A lower navigation effort means less effort to find a desirable item. Second, we define the exiting Probability which determines on average what proportion of users left the session after a certain number of interactions. For example, in 5a on average, just under 50% of users in carousel interaction model exited the session with fewer than 30 interactions. The total number of interactions includes all examinations done by the user to find the desirable movie. It is important to state that the exiting Probability only can be considered as a positive metric under the ideal setting where the user continues the examination until finding the desirable items. Unlike the ideal setting, in distracted and impatient settings, the exiting Probability could be an indication of either satisfactory, due to finding the desirable items or unsatisfactory, due to impatience or distraction and without necessarily finding the desirable items. In our experiments, we only compare the exiting Probability under the comparable settings.

Figure 4:The correlation between navigation effort for a selected movie in carousels and ranked list models.

5.2.1 Ideal Setting. In the first setting, we assumed that the user continues to examine topics and items until finds the desirable item. The behavior of such a user is described in section 4.1. We are aware that this browsing behavior is unlikely to occur in a realistic situation due to the position bias effect [16]. However, we include this setting in our evaluation to highlight the difference between this and other more realistic behavioral patterns.

5.2.2 Impatient User. To better model a browsing behavior of an actual user, we assume that the user has limited patience for finding the desirable item. We implemented this behavior as follows. The user starts by examining the first topic or item at position (1, 1). The user exits with a probability of p q = 0.02 after examining either a carousel or item. Generally, users are likely to abandon the session after 50 interactions on average, when no items or topics are desirable. This is the same as the ideal setting except for exiting with probability p q = 0.02 upon each examination, of either a carousel or an item.

5.2.3 Distracted User. We initially assumed that the user always knew which carousel (with a genre as a topic) includes the desirable movie. However, in reality, the user might get distracted and as a result, begin browsing the wrong carousel or pass the correct carousel and miss out on finding the desired item. We consider this assumption to be an extension of the previous assumption described in section 5.2.2.

Figure 5:Comparing the cumulative exiting probability in single ranked list and carousel interaction model and in different experimental settings reveals the advantages of using a carousel based representation compared to a ranked list based representation. In all experimental settings users leave after 50 interactions regardless of success in finding the desirable item.

In both ideal and distracted user settings, when the user examines an undesirable carousel, they will move to the next carousel with a probability 1. We define p d = 0.05 as the distraction probability. Here user moves to the next carousel with probability 1 − p d and starts examining items in the undesirable carousel with probability p d. Similarly, when the user examines a desirable carousel, they move to the next carousel with probability p d and start examining items in the desirable carousel with probability 1 − p d. Considering a user as distracted only applies in carousel interaction model. Including this assumption in the carousel interaction model allows us to capture the complexity that comes with providing additional information to the user in the form of carousel topics.

Because of lacking a large enough data set that can accurately estimate the parameters of our proposed settings, we set the values of p q and p d intuitively based on how we presume the user would behave under those settings.

In this section, we present the results of our experiments. In section 5.3.1 we compare the distribution of navigation effort for carousel interaction model and single ranked list models with samples from different number of top movies in the dataset. In section 5.3.2 we demonstrate how different browsing behavior affects the exiting pattern among users.

5.3.1 Navigation Effort. 4a shows the distribution of navigation effort values for all the movies (9708) and users (610) in our dataset. The experiment was conducted under the ideal setting described in section 5.2.1. Similarly, 4b 4c 4d display the distribution of navigation effort values for the top 1000, 500 and 100 movies respectively. These results indicate that the carousel interaction model significantly reduced the number of required interactions to find a desirable movie. It is evident that even though the slope of the lines remains relatively steady, the higher number of movies resulted in a slightly more prominent improvement from single ranked list to a carousel interaction model.

5.3.2 Exiting Probability. To compare the behavior of our model under more realistic settings, we visualize the average exiting probability of users after certain number of interactions with the recommendations in 5a and 5b . in 5a we observe a significant difference between the carousel interaction model and the single ranked list under the ideal settings. In ideal setting, user continues the examination until reaches the desirable item. We limit the number of interactions to 50 meaning the user would exit unsatisfied if they could not find the desirable item in first 50 interactions. The higher exiting probability in carousel interaction model (blue line) shows that more users exit the system satisfied by finding their desirable item. A larger spike in exiting probability on single ranked list at the end indicates a larger number of users that left without finding their desirable item. It worth noting that based on the result of this experiment, a significantly larger portion of users (just under 75%) exit the system after finding their desirable. This number drops to close to 30% when recommendations are presented in the form of a ranked list. The exiting behaviour of the simulated impatient and distracted users is displayed in 5b .

Although the gap between the probability of exiting the session in carousel interaction model and single ranked list models is less significant, the former still performs better. Comparing the Impatient and distracted exiting behavior indicates a non-significant difference between the two settings but shows a slight decrease in performance in carousel interaction model. Unlike 5a where the exiting probability promote a positive event (satisfaction of finding the desirable item), in 5b there can be also adverse reasons for exiting a session, such as ”impatience” and ”distraction”. Therefore, the improvement of this metric compared to the ideal setting is not necessarily a positive sign. Despite this, since we compare carousel interaction model and single ranked list in 5b under the same setting where the probability of ”impatience” is the same, an improvement in the metric likely signal a positive event.