Navigational Correlates of Comprehension in Hypertext

Authors: John E. McEneaney

Publication: Proceedings of the Eleventh ACM Conference on Hypertext and Hypermedia (Hypertext 2000), ACM, 2000.

DOI: 10.1145/336296.336504

Source: Supplied ACM HyperText 2000 proceedings PDF (SHA-256: 503e989120ddfddf6be56d6453df7ac374bcf4d0bf2ac7ea18b234dcdf6aef4e).

Full text

Source page 1

Navigational Correlates of Comprehension in Hypertext

John E. McEneaney

Department of Reading and Language Arts

Oakland University

Rochester, MI 48309-4494, USA

E-mail: mceneane@oakland.edu

ABSTRACT

Despite a substantial literature on problems related to user

navigation, we know remarkably little about the relationship

between navigational strategies and successful use of

hypertext. Revealing the complex interactions of navi-

gational decision making and comprehension, however, will

require objective, reliable, and empirically significant

navigational metrics that go beyond the largely informal and

indirect measures that have traditionally been used. The

purpose of this paper is to describe a study that replicates

and extends earlier work that defines and validates two

objective navigational metrics [5]. Results of this study

confirm the empirical significance of these metrics, support

the reliability of their relationship to hypertext com-

prehension, and provide indirect support for a model of

reading comprehension that postulates greater demands on

higher-level processing in hypertext compared to traditional

print.

KEYWORDS: comprehension; empirical validation;

navigational metrics; navigational patterns; user paths.

INTRODUCTION

The present study employs two navigational measures, path

compactness and path stratum, that are based on structural

metrics originally developed to describe hypertext networks

[1]. The purpose of these metrics is to yield global,

network-based assessments of structure grounded in node-

based centrality measures, with each metric ranging from 0

to 1. Network compactness refers to the complexity or

degree of linking within a network. As defined, more

sparsely linked networks result in values for compactness

closer to 0, while densely connected networks yield

compactness closer to 1. Stratum, on the other hand, refers

to the degree of linearity of a network, as indicated by the

extent to which a network is organized so that certain nodes

must be read before others.

The path metrics employed in this study also range from 0

to 1 and similarly reflect degree of complexity and

Permission to make digital or hard copies of all or part of this work for

personal or classroom use is granted without fee provided that copies are

not made or distributed for profit or commercial advantage and that

copies bear this notice and the full citation on the first page. To copy

otherwise, or republish, to post on servers or to redistribute to lists,

requires prior specific permission and/or a fee.

Hypertext 2000, San Antonio, TX.

linearity. Path metrics, however, are based on path matrices

that reflect use of links during a reading session rather than

network adjacency matrices.

Path compactness (PC) is defined in a manner analogous to

the compactness structural metric with the one noted

difference that path matrices are employed instead of

adjacency matrices. The metric is formally defined as

(

)

Max

(

)

Max

Min

Readable transcription: PC = (PMax − Σ_i Σ_j Pij) / (PMax − PMin).

where PC refers to the converted distance matrix of the path

and PMax and PMin refer, respectively, to the maximum and

minimum converted distance values that the path matrix

can assume for a completely-connected (PMin) and

completely-disconnected (PMax) network consisting of as

many nodes as are in the path. PMax and PMin are given by PMax

= K(n

2-n), and PMin = (n

2-n), where n is the order of the path

matrix and K is a matrix conversion constant.

Path stratum (PS) is also defined in a manner analogous to

its structural equivalent with

path

absolute

prestige

LAP

Readable transcription: PS = (path absolute prestige) / LAP.

where the path absolute prestige is defined as in a network

with the exception that the distance matrix is derived from a

path matrix rather than an adjacency matrix. The linear

absolute prestige (LAP) that serves as the normalizing

measure for this metric is also defined analogously to the

structural metric, based on the number of distinct nodes in

the path.

METHOD

Participants in the study were 133 students at a medium-

sized midwestern public university in the USA, with data

collection extending across both terms of the 1998-1999

academic year. The experiment required subjects to answer

a set of academic advising questions using a hypertext

student handbook that was presented through a web-browser

interface. Subjects answered as many questions as possible

in a 15-minute period. The handbook consisted of

approximately 31,000 words in 78 text nodes structured in a

hierarchical-linear fashion with major handbook divisions

organized hierarchically and nodes within those divisions

organized in a largely linear fashion. Subject movement

within the handbook was recorded to the browser cookie file

by JavaScript code embedded in the handbook pages.

254

Source page 2

Following experimental sessions, subjects' path data were

retrieved from browser cookie files. Data was formatted for

visual display and path compactness and stratum metrics

were calculated for each subject. Responses to academic

advising questions were scored on a three-point scale. Zero

points for incorrect and omitted responses, 1/2 point for

partially correct and correct-but-incomplete responses, and 1

point for complete and correct responses [6].

RESULTS

As in the earlier study [5], there were pronounced visual

patterns evident in the path graphics [4]. Higher scoring

subjects consistently followed more complex "criss-

crossing" paths while lower scoring subjects tended to take

more linear paths through the document. Moreover, the

graphical analyses relating readers' paths to their scores on

the hypertext task were confirmed by follow-up statistical

analyses based on the path compactness and stratum metrics.

Results of the statistical analyses (Table 1), replicate earlier

findings indicating that performance on the hypertext task

had a moderate positive correlation for the Pc metric and a

moderate inverse correlation for the Ps metric. Moreover, as

in the previous study, both of these metrics correlate more

highly with the number of questions answered correctly than

a variety of other variables, including total pages viewed,

different

pages

viewed,

and

even

print-based

comprehension measure administered prior to the hypertext

reading task.

Variable

Vocabulary Pretest

.000

.529

Comprehension Pretest

.023

.289

Total Pages Read

.073

.202

133

Distinct Pages Read

.026

.383

133

Path Compactness (Pc)

.000

133

.381

Path Stratum (Ps)

.000

133

-.353

Table 1. Correlations of experimental

variables with hypertext reading scores.

Furthermore, although in the earlier study print reading

measures failed to correlate significantly with the hypertext

task, in this investigation there were significant correlations.

Related to this, it is interesting to note that the vocabulary

pretest accounted for more than 3 times the variance of the

hypertext score than did the comprehension pretest, a result

that seems counterintuitive given the fact that the hypertext

task assesses comprehension rather than vocabulary.

CONCLUSIONS

The results of this study are important for a number of

reasons. One reason is that they confirm earlier findings

that navigational metrics account for a significant degree of

variance in measures of hypertext comprehension. Given

the robust character of this association, and the ready

availability of real-time path-based data in electronic

reading environments, it seems likely that the metrics used

in this study can make significant contributions to user

models that will certainly be important elements in adaptive

hypermedia systems [2,3]. A second reason these results are

important is that they lend still greater credibility to the

broader theoretical framework used to derive these metrics

and, as pointed out in a previous paper [5], this framework

provides a basis for a more systematic and objective

analysis of reader navigation than has been possible before.

Finally, a third reason for interest in these results is that they

are consistent with independent empirical evidence [7]

suggesting that hypertext makes different demands on

higher-level comprehension processing (as compared to

traditional print), while lower-level processing remains

largely unchanged. Specifically, the smaller correlation of

the comprehension pre-assessment may simply reflect the

fact that comprehension processes are altered in a hypertext

environment while lower-level processes (like vocabulary)

are not. If this is the case, the differences in higher-level

processing would tend to erode the correlation between the

hypertext task and a print-oriented comprehension measure,

while there would be little reason to expect a reduction in

the association with lower-level vocabulary processes.

REFERENCES

Botafogo, R. A., Rivlin, E., & Shneiderman, B. (1992).

Structural analysis of hypertexts: Identifying hierarchies

and useful metrics. ACM Transactions on Information

Systems, 10(2), 142-180.

Dattolo A. & Loia, V. (1997). Active distributed

framework for adaptive hypermedia. International Journal

of Human-Computer Studies, 46, 605-626.

De Bra, P., Eklund, J., Kobsa, A., Brusilovsky, P., & Hall,

W. (1999). Adaptive hypermedia: Purpose, methods, and

techniques. In Klaus Tochtermann, Jörg Westbomke, Uffe

K. Wiil, & John J. Leggett (Eds.), Proceedings of the

Tenth ACM Conference on Hypertext - Hypertext '99,

199-200. New York: ACM.

Ellson, J., Koutsofios, E., & North, S. (1998). GraphViz

1.3 [Computer software]. Murray Hill, NJ: Lucent

Technologies.

McEneaney, J.E. (1999). Visualizing and assessing

navigation in hypertext. . In Klaus Tochtermann, Jörg

Westbomke, Uffe K. Wiil, & John J. Leggett (Eds.),

Proceedings of the Tenth ACM Conference on Hypertext

Hypertext '99, 61-70. New York: ACM.

McKnight, C., Dillon, A., & Richardson, J. (1990). A

comparison of linear and hypertext formats in information

retrieval. In R. McAleese & C. Green (Eds.), Hypertext:

State of the art (pp. 10-19). Oxford: Intellect.

Wenger, M.J. & Payne, D.G. (1996). Comprehension and

retention of nonlinear text: Considerations of working

memory and material-appropriate processing. American

Journal of Psychology, 109(1), 93-130.

255