<nettime> Jump over Proust: Toward Multimedia Writing

[manovich@ucsd.edu (Lev Manovich)]

Lev Manovich
http://jupiter.ucsd.edu/~manovich

Jump over Proust:
Toward Multimedia Writing


        How does fictional, critical, theoretical, and historical writing
can take advantage of all the multimedia capabilities offered by the Net?
How to write about ideas using images, animation, sound, video and 3D worlds?
How to communicate complex concepts while also being able to employ
traditional rhetoric functions (seduce, convince, scare, inspire) using
multimedia? How to allow the user not just to be simply a "co-author"
(which is what the ideologists of interactivity naively aim at) but rather
to take him/her "inside" the mental space of a text, inside the thinking
process of another subject? How to think through multimedia? What are the
historical precedents for "multimedia writing" in cinema, book design,
theater, concrete poetry? [1]
        In my attempts to deal with these questions I was inspired by certain
filmmakers who appear to be obsessed not simply with using cinema as a
medium to convey ideas and arguments (which is what conventional
documentaries are supposed to be doing) but rather as a medium capable of
presenting the very process of thinking. Among these filmmakers I would
single out Eisenstein, Marker and Godard.
        The first tried to formulate the notion of intellectual montage and
planned to film Marx's Capital. Already at the end of the 1920s he was
predicting that in the future philosophy and history will be presented as
films:

"The proclamation that I'm going to make a movie of Marx's Das Kapital is
not a publicity stunt. I believe that the films of the future will be found
going in this direction (or else they'll be filming things like The Idea
of Christianity from the bourgeois point of view!) In any case, they will
have to do with philosophy...the field is absolutely untouched.
Tabula rasa." [2]

Marker showed that cinema can be used to construct intellectual essays, the
essays which associatively move from idea to image, and an image to another
idea (for instance, in his recent "Level 5"). And Godard has already explored
computer multimedia's new language in his films from the 1960s onward by
systematically juxtaposing moving images, non-realistic sound and
graphically presented texts; more recently he created a true multimedia
masterpiece, an essay which takes us along his thinking process while using
every multimedia code available: pages of text, still images, moving images,
voice and sound ("JLG by JLG. Self-Portrait in November").
        Not only to convey complex ideas through multimedia, but to take the
reader along the process of thinking -- this is the challenge of multimedia
writing. The use of a computer as writer's tool can only be justified if we
can
evolve more subtle, more complex and more precise ways of conveying what
it means to think, of how it feels to move from one association to the next,
from one memory to another, from one insight to the next. Only when we
will give justice to the common view of a computer, which accompanied it
from its very beginnings half a century ago, as a model of a human brain. A
machine which has memory, which can store words and images, which can
search and match, which, most importantly, can link, i.e. associate -- even
if it is not a human mind, it has most of the functions we, humans, perform
when we think. Therefore, we should be able to use a computer to portray
human thinking in a more precise and engaging way than literature and
cinema have already done. To do this is the challenge of multimedia writing.
        Earlier in this century, Proust, Nabokov, Joyce and other modernist
giants came up with new techniques to represent our mental life: thinking
and remembering, forgetting and repressing, formulating concepts, moving
between the sensorial world outside and the mental world inside. Literature
became the best mirror for the modern psyche, achieving highest fidelity in
relation to our inner world. But other arts did not match its achievement.
Cinema, multimedia's main precursor, is particularly disappointing in this
respect. By and large, its language followed 19th century novel, rather than
trying to match -- and go beyond -- the microscopic view of human inner
experience recorded by Joyce, Nabokov and other moderns.
        This is particularly surprising, given that the first theoretical
text on cinema -- Hugo Munsterberg's The Film: A Psychological Study (1916)
-- proclaimed that the essence of the new medium lies in its ability to
reproduce, or "objectify" various mental functions on the screen. According
to Munsterberg, "The photoplay obeys the laws of the mind rather than those
of the outer world." [3]  In a provocative analysis, Munsterberg correlated
the
main cinematic techniques to different mental functions such as attention
and memory, one-to-one. For example, in the close-up, "everything which
our mind wants to disregard has been suddenly banished from our sight and
has disappeared," analogous to how our attention selects a particular object
from the environment. Similarly, the "cut-back" technique objectifies the
mental function of memory.
        Yet, despite this promising analysis made by Munsterberg early on,
cinema hardly took up the challenge of being a mirror of mind's operations.
In my view, the only real systematic attempt in cinema to do this has been
Godard's recent work, such as already mentioned "JLG by JLG" and also the
majestic and monumental "Histories of Cinema" (which currently can be
seen in Documenta X in Kassel). In the latter, Godard uses new cinematic
techniques of his own invention in order to portray thinking process more
accurately. For instance, he often superimposes 2, 3 or more images which
gradually fade in and out, but never dissappear completely, staying on the
screen for a few minutes at a time. It is as though these are ideas or mental
images floating around in our minds, coming in and out of mental focus.
Another technique involves replacing one image by another not through a cut
or a dissolve but through a repeated oscillation, with two images flickering
back and forth over and over, until the second image finally replaces the
first. This technique can be interpreted as an attempt to represent mind's
movement  from one concept, mental image or memory to another -- the attempt,
in other words, to represent what, according to Locke and other associationist
philosophers, is the basis of our mental life -- forming associations.
        Yet, along with these innovative techniques which would certainly
please Munsterberg (who, accidentally, was a professor of Psychology at
Harvard University) by being visual equivalents of mental operations (or
shall we say video-temporal equivalents, since time obviously plays a crucial
role in Godard's techniques) Godard also often "capitulates" to cinema's more
conventional way of narration: showing reality (here, a person thinking)
from a third person point of view, i.e. from the outside. In "Histories of
Cinema," we repeatedly see close-ups of a book page, or Godard himself,
standing next to a bookshelf, getting a book, reading a sentence or two; or,
finally, Godard sitting at a table and typing or writing. Perhaps these can be
thought of as being equivalents of "establishing shots" in traditional
cinematic narration: Godard's shows us his location (i.e., his
mind) from the outside, so to speak, before taking us inside. Perhaps these
images can be also interpreted as challenges to the conventional cinema and
its extension, computer multimedia -- lets focus on intellectual life, on
human mind rather than external actions and stories.
        So far a computer, despite his persistent association with a human
mind, has served as even worse artistic mirror for our mind than cinema.
This is strange given the fact that while only Munsterberg and few others
made a connection between human mind and cinema language, in the case of
a computer making similar connections became the research focus of a
number of new fields, enormously successful, fields which keep expanding
more and more: artificial intelligence, cognitive psychology, neuroscience --
in short, a whole set of disciplines grouped together under the name
cognitive science, whose ultimate purpose is to map the language of the mind
and the language of a computer one into another. While the attempts of
artificial intelligence to simulate human mind have met with some limited
success in such areas as parsing human speech, understanding stories,
planning actions and interpreting images, the reverse problem -- the cultural
problem -- using a computer to represent human mind in all
its complexity and specificity (i.e., modeling not just the rational-
computational part, as in artificial intelligence, but the phenomenological
whole), pushing beyond what arts has accomplished so far  -- was hardly even
raised. Obviously, current language of multimedia -- presenting a user with a
page containing a small number of links leading to other pages -- is hardly an
adequate mirror of our mental life, or how we think, remember, plan, make
connections.
        At present, software tools themselves are more revolutionary than
multimedia applications they are used to design. They are better artistic
visions of our inner life. Relational databases; pointers; control structures
("if... than," "case," etc.); object-oriented programming -- these and other
programming concepts point towards potentially complex, dynamic and rich
cultural representations of human mind. Even such seemingly trivial
concept as a hierarchical file system is already more suggestive than the
typical pages with hyperlinks which are being served to us
in the 1990s under the slogan of "new media." Whatever it may involve,
human thinking is certainly more like a computer program under execution
(which involves translating between a hierarchy of computer languages,
writing and reading data, keeping track of a current place in a program,
clearing space in memory for new data and so on) than a set of pages linked by
hyperlinks.
        To bring this new level of complexity, already achieved in software
design, into the realm of cultural representation -- this is the challenge of
multimedia writing. To do this, we need to be looking both at best cultural
achievements in "mind modeling" -- Proust and Nabokov, Joyce and Godard
-- and at the concepts of computer science, at the structure of computer
hardware and software. Only when our multimedia texts will do justice both
to the complexity of the machines used to compose and distribute these texts
-- computers -- as well as to the complexity of what it feels to be a human
being today: to think, to reflect, to carry the burden of human cultural
history and of never before available amount of information and news from
around the world, to interact with artificial minds of computers and with
minds of other humans -- and also, as always, still to respond to the physical
environment outside, the presence of others, to light, touch, and smell. In
short, to be human, to reflect and to exist, to be inside and to outside at
the
same time. To represent this uniquely human, embodied thinking --
this is the challenge of multimedia writing.



NOTES
        1. In chapter 2 of my Ph.D. Dissertation, I explore many modern
attempts to come with the techniques to represent ideas and logical
arguments and to think through images. Lev Manovich, "The Engineering of
Vision from Constructivism to Virtual Reality" (University of Rochester,
1993, unpublished). The historical material used in this text is drawn from
this chapter.
        2. Quoted in Annette Michelson, "Reading Eisenstein Reading
'Capital'," October 2 (1976): 28.
        Einstein's theory was not an isolated development. Many in the
artistic
left of the 1920s shared a similar belief in the cognitive power of new visual
forms such as montage. In the late 1920s Alexander Rodchenko promoted the
use of montage sequences in graphic design and, like Eisenstein, he saw
montage as being equivalent to "dialectical" reasoning. In this formulation,
an individual image corresponded to a single concept, and thinking was
thought to be provoked when a number of images were juxtaposed in a
series. Walter Benjamin's notion of  "dialectical seeing," central in his
unfinished Passagen-Werk project, also depends on montage, but within a
single frame, so to speak. "Dialectical seeing" was conceived by Benjamin as a
way to grasp the forms of the present by looking to the past and to the future
in the same instant, juxtaposing them in the same mental
image.
        3. Hugo Munsterberg, The Photoplay: A Psychological Study (New
York: D. Aplleton & Co., 1916), 41.






---
#  distributed via nettime-l : no commercial use without permission
#  <nettime> is a closed moderated mailinglist for net criticism,
#  collaborative text filtering and cultural politics of the nets
#  more info: majordomo@icf.de and "info nettime" in the msg body
#  URL: http://www.desk.nl/~nettime/  contact: nettime-owner@icf.de