|
Visual Thinking
and
Visual-Verbal Communication
This page has not been revised
since May 2001, but
the version on another website has been revised since then,
and it's part of a new website that has many more pages,
so I strongly recommend that you read
THE
REVISED VERSION.
The following paper, written mostly in 1995,
was originally intended to be one section in my PhD dissertation:
This section will discuss the educational value of communication that combines visual and verbal representations, with a focus on the relatively underutilized visual aspects of this partnership. It is commonly accepted that visual representations can serve a number of valuable functions, both affective and cognitive. Although the affective functions of illustrations (such as motivating students) are often important, the current discussion will concentrate on cognitive functions, beginning with some ways in which meaning can be expressed visually.
When discussing the communication of visual meaning it is useful to distinguish between literal representations that are intended to resemble the object they portray, and symbolic representations. { These categories can also be labeled using other terms, such as realistic and abstract, respectively. } Of course, there is a continuous range between literal and symbolic, and many pedagogically useful pictures are a complex mixture with characteristics that fit into both categories.
To the extent that a representation is literal, the intended
meaning is obvious. But a learner must still form mental images of
the object, and -- especially with an unfamiliar object such as a biological
cell -- this may require some concept-formation or concept-restructuring.
Always, drawings are intended to be accurate in some respects but not others.
For example, in a textbook a drawing of a cell will be simplified, with
the amounts and types of simplification depending on the instructional objectives;
different types of cell drawings will be used for elementary school students,
high school biology students, graduate students, or experienced scientists.
Similarly, if a drawing of a subway system is intended for use by passengers,
it may show the proper sequence of stops but not accurate distances; but
a drawing that is made for use by subway designers or construction workers
will be more accurate, especially in those characteristics for which accuracy
is important.
Sometimes the meaning of literal must
be carefully defined. For example, in order to construct a literal
interpretation for one type of depiction for a 3p atomic orbital, a viewer
could imagine (contrary to actual possibility) that a "magic camera"
can take multiple-exposure photographs of an electron that continuously
remains in a 3p orbital; the result, if this could occur, would be the multiple-dot
photo that is being viewed. One potential difficulty with this depiction
is that if a viewer misinterprets the ways in which the picture is and is
not literal, the result will be a misconception about quantum mechanics.
On the other hand, if a viewer understands the picture -- including a claim
that "although quantum mechanics refuses to predict, based on Photos
#1 and #2, where the electron will appear in Photo #3, it can make a statistically
correct prediction for a multiple-exposure photograph" -- the result
will be a stronger, more accurate understanding of quantum theory.
Winn (1987) discusses three types of
symbolic visual representations -- charts,
graphs, and diagrams
-- and describes their position at the middle of a
continuum between realistic pictures (which resemble what
they represent) and words (whose symbolism
is based on arbitrary convention): "From words they inherit the attribute
of abstraction; but like pictures they exploit spatial layout in a meaningful
way. Their abstract nature makes them well suited to explaining how
processes work where realistic pictures would fail. ... Presenting
information graphically allows students to scan it rapidly and quickly to
discover patterns of elements within the diagram that are meaningful and
that lead to the completion of a variety of cognitive tasks. ... [This
is especially useful] in mathematics and science where patterns and structures
are themselves important properties of the content area. (pp. 152, 191)"
In symbolic visual representations, meaning is partially communicated by
the spatial organization of information -- by supplementing the symbolic
meaning of individual elements with meanings implied by the spatial positions
and spatial relationships of these elements. In a chart, meaning can
be conveyed in a number of ways, including the classification of items in
categories such as those indicated by the rows and columns of a table.
In a typical x-y graph, some quantitative characteristics of an item are
indicated by its location in the x-y space of the graph; in other types
of graphs, magnitudes (or other characteristics and relationships) are conveyed
in other ways. In a diagram there is greater freedom of expression,
and a wide variety of symbolic conventions can be used: conceptual
closeness can be symbolized by spatial closeness; inclusion in a category,
or exclusion from it, can be shown with an enclosing box or a symbolic color
scheme; visual sequences can be explicitly stated with arrows, or
implied by following the standard convention for verbal sequences (in European
languages) of left-to-right and top-to-bottom ordering; similarly,
hierarchies can be implied by linking-lines and relative placement of elements.
And because verbal information is often incorporated into visual representations,
in any diagram (symbolic or literal) meaning can be conveyed by verbal or
typographical cues, such as captions, element-labels, and type size.
In an effort to explain the relatively efficient recall of pictures,
Paivio (1978, 1986) proposed a theory of dual coding.
According to this theory there are two types of memory coding -- in a verbal system and an image
system. Verbally presented material is encoded only in the
verbal system, while visually presented material is encoded in both the
verbal and image systems. In contrast with the memory's "single
coding" for text, pictures have "dual coding" in two types
of memory codes; if these two codes provide more cues for recall, then it
generally should be easier to remember pictures. But, as pointed out
by Schnotz (1993), "Graphics offer various advantages to the process
of knowledge acquisition which go far beyond a mere memory effect."
Therefore, scientists have attempted to build theories that can explain
the functions of verbal and visual information in helping learners construct
their own mental models of the subject matter that is being portrayed in
the verbal and visual material.
Mayer (1993) outlines a framework, derived
from Paivio's dual coding theory, for interpreting the cognitive processing
of information that is presented both visually and verbally. As shown
in Figure 2.2, this framework postulates the formation of three types
of mental "connections": 1) visual material is
used to mentally form a visual representation, thus forming a connection
between the external visual material and the internal visual representation;
2) verbal material is used to form a verbal representation, thus forming
a verbal representational connection; 3) the learner builds referential
connections between the visual representation and verbal representation.
A dual coding theory of learning from visual and verbal materials. (Mayer, 1993)
This theory can be elaborated (Schnotz, Picard, & Hron, 1993; Schnotz, 1993) by interpreting the qualitative differences between verbal and visual representations in terms of their differing functions as symbolic and analog representations, respectively. Verbal information, based on the meaning of individual words and the relationships implied by grammatical structures, is used to mentally construct a propositional symbolic representation, which can then be used to construct a mental model. Visual graphics, which convey information by implied analogy between certain spatial characteristics of the graphic and characteristics of the content that is being described, can allow a more direct construction of a mental model. Thus, "Texts and graphics are complementary sources of information insofar as they contribute in different ways to the construction of a mental model." (Schnotz et al, 1993, p. 183) Along these same lines, Kirby (1993) describes a mental-models approach (such as that of Johnson-Laird, 1983) that "emphasizes the importance of connections between mental codes and ... a real-world model. These real-world models take the form of mental images ... and the spatial mode of processing, from presented images, is thought to be the most efficient means of developing the appropriate representation. The verbal mode is but one means of accessing that central representation, and perhaps an awkward one at that. (p. 203)"
Based on a review of experimental and theoretical work in this area, Winn (1987) describes the cognitive value of visual representations and visually-oriented processing:
These visually-oriented cognitive processes include, but are not limited to, the construction of mental models.
In a dual-coding model such as Mayer's framework in Figure
2.2, there are three types of connections: visual, verbal, and referential.
Kirby (1993) discusses the contexts in which interactions between visual
and verbal information are collaborative (to support learning) or competitive
(to inhibit learning). First, with difficult tasks where visual and
verbal memory-encoding is not sufficiently automated, there can be "interference"
due to competition for the limited executive resources needed to control
the two types of processing. Second, some information is easier to
process in a particular mode, either visual or verbal; if a teacher attempts
to force some of this information into the less effective mode, it can distract
a learner from in-depth processing in the more efficient mode. Similarly,
some types of tasks may be facilitated by thinking in a visual mode, while
for other tasks a verbal mode will be more effective. Finally, individual
learners differ in their affinities -- with respect to abilities, prior
knowledge, strategies, and interests -- for visual and verbal learning styles.
Kirby recommends using verbal memory-coding
for some information in some situations, visual coding in other contexts,
and forming referential connections between these codes -- which is possible
"when there is some degree of overlap or redundancy in the two sets
of information" -- so that each type of coding can access the other.
He emphasizes the value of verbal-visual instruction to "teach students
how to perform conjoint processing optimally," and concludes his paper
with a description of "integratability" and "conjoint education":
With a written text, accurate communication of ideas between an author
and reader depends on the existence of a set of shared assumptions about
the meaning of the verbal symbols that comprise
the text. Similarly, the quality of visual communication depends on
the visual symbols shared by an artist and
viewer. The set of shared assumptions concerning the meaning of symbols
-- including a vocabulary for individual symbols and "grammatical rules"
for combining symbols with each other in various ways -- can be considered
a culturally constructed language, whether
this language is based on symbols that are verbal or visual. Whether
communication is verbal or visual, achieving a "language match"
between the information sender (author or author/artist) and receiver (reader
or viewer) is essential.
I find it useful to imagine the visually
mediated communication of ideas as a two-step process of encoding-and-decoding:
mental-to-visual, followed by visual-to-mental. First, THE AUTHOR/ARTIST'S
MENTAL MODEL OF A SYSTEM is encoded, by using analogy to move from conceptual
characteristics (mental) to spatial characteristics (visual), to a DIAGRAM
that is a symbolic visual representation of the system. Second, this
DIAGRAM is decoded by a viewer, using analogy to move from spatial characteristics
(visual) to conceptual characteristics (mental), to form THE VIEWER'S MENTAL
MODEL OF THIS SYSTEM.
A similar process of encoding-and-decoding
occurs in verbal communication, where a verbal text (analogous to a visual
diagram) is used as an intermediary. With either visual or verbal
communication, there is not a direct correspondence between the original
system and the mental model formed by a learner. Instead, an understanding
of the system, including its conceptual characteristics and their integration
into larger structures of domain knowledge, is filtered through several
layers of interpretation: there must be a perception of the system and formation
of a mental model by the artist or author, an encoding of this mental model
to make a visual or verbal representation based on a culturally constructed
language, and a decoding by the learner to form a personally customized
mental model. An improvement in any of these steps can facilitate
improved learning.
As expected, the interpretation of visual
language is a skill that depends, to some extent, on experience in a particular
domain of knowledge. For example, in a study to compare the ways in
which professional meteorologists and non-meteorologists construct mental
representations from a weather map diagram, Lowe (1993) found distinct differences
between the performance of professional meteorologists and non-meteorologists.
While the nonmeteorologists focused on superficial, domain-general, visuo-spatial
features, the meteorologists were more skillful at selecting those visual
features that are essential for developing an understanding the state of
the weather system being depicted. The nonmeteorologists could recognize
spatial patterns in the diagram, but they were not proficient at translating
this spatial knowledge into weather knowledge. The meteorologists,
due to their deeper understanding of the concepts and visual symbolism associated
with weather maps, were better able to decode the semantic analogies --
between the visuo-spatial characteristics of the diagram and the physical
characteristics of the weather system -- that were encoded into the maps
by the map-makers.
Many educators believe that skills of
visual interpretation, such as those used by the experienced meteorologists,
can be taught in schools. Moore (1993) describes a program that adapts
reciprocal teaching (Palincsar & Brown, 1984) for instruction in visual
skills. Reciprocal teaching -- a metacognitive training approach typically
aimed at enhancing verbal skills through the use of reciprocal interactions
between experts and novices in explicit demonstrations of strategy use --
is designed to help students learn how to plan, monitor, and evaluate their
own learning strategies and outcomes. Adapted for instruction in visual
skills (Moore, 1993), students are urged to metacognitively employ a repertoire
of "SLIC" strategies for Summarizing, Linking diagrams with text,
Imaging, and Checking for understanding. To help students develop
a deeper, broader range of skills in visual-verbal learning, these strategies
could be explicitly developed and practiced in a variety of domains, using
a variety of diagrams. Peeck (1993) also discusses the modification,
to include visual skills, of programs originally intended to help students
focus their attention on skills for verbal processing. Specifically,
Peeck recommends "adding to the learning material specific instructions
and tasks that require desirable learning activities, such as intensive
processing of the pictures. (p. 233)" These instructions and
tasks can be: general directions to "pay attention to the illustrations";
specific directions about what to look for, in general or in a particular
picture; or an assignment that requires students to actively construct a
response or product based on their interpretation of an illustration.
Research on visual representations has produced mixed results.
Commenting on this, Peeck (1993) says, "There is therefore a good deal
of ambivalence and paradox in the position of text illustrations in the
educational process. On the one hand, there is a general acknowledgment
of their potential value, as their continuing and probably increasing presence
in educational material testifies; on the other hand, there is plenty of
reason to regard their effects with realistic pessimism. (p. 228)"
Part of the mixed results, especially
in early studies, can be explained by a lack of attention to detail in designing
experiments, or by inadequate interpretations of observations. For
example, Levin & Mayer (1992) describe the confusion caused by not distinguishing
between the stages of "learning to read" (at this time, illustrations
are often detrimental because they can act as a crutch for students who
would rather not depend on obtaining meaning from the text) and "reading
to learn" (at this time, when students can read skillfully, illustrations
that supplement text can improve comprehension and retention). Similarly,
Winn (1987) cites research by Holliday (1976) in which a diagram accompanied
by text was less effective than the diagram by itself, evidently because
students tend to ignore a diagram -- instead of studying it intensely because
it's all they have -- if they believe they can get all the information they
need by only reading the text. Kirby (1991), as discussed above, might
describe this as a "competitive" effect caused by distracting
the focus of attention away from the processing mode that would be most
effective.
Winn (1987) and Peeck (1993) suggest
that research should be interpreted by carefully considering the effect
of three types of factors: treatment, learners, and task. As with
any instructional technique, the effectiveness of a diagram will depend
on the entire context of treatment, including the diagram characteristics
(e.g., is it realistic or symbolic) and quality (has the artist expressed
the content clearly and appropriately), the support system (such as the
"reciprocal teaching of SLIC" or "specific instructions and
tasks" discussed above), the classroom environment, whether the treatment
is well designed to achieve the educational objectives, and other relevant
considerations. It is also essential to consider characteristics of
the learners, such as affinities (abilities, strategies, experience, field
dependence, locus of control, interests, preferences) for learning and thinking
in visual and verbal modes, prior knowledge of the domain being studied,
attitudes toward schoolwork and the subject domain, and so on. Also,
is the evaluative task appropriate for the "treatment and learners"
situation, and does it really indicate the extent to which the educational
objectives (conceptual understanding, acquisition or improvement of skills,
retention, transfer,...) have been achieved?
The current consensus of scholars is
that, when interpreting past research and planning future research, the
objective should be to determine, with greater precision, how the effectiveness
of various types of visual-verbal instruction depends on the context in
which they are used, and how we can develop teaching methods that are more
effective. In making these evaluations, a wide range of relevant criteria
-- including the characteristics of the visual-verbal instruction, the nature
of the learners and instructional environment, and the educational objectives
-- should be carefully considered.
{ references: sources for the citations in this paper will be provided here later, probably by July 15 }
{ some related ideas are in "Coping with Complexity" on my EDUCATION-page }
http://www.sit.wisc.edu/~crusbult/methods/visual.htm
home science design goals education x-rated ISM bonus