The human brain processes visual information differently than text. We’re able to see patterns and relationships in a unique way when we solve problems visually.
Dr Anjan Chatterjee ought to know: he’s a world-renowned neuroscientist at the University of Pennsylvania Center for Cognitive Neuroscience. He’s done extensive research on how our brains process visual information.
I was fortunate to have met Anjan in person at the Interact London conference last year. See his presentation from that event at the end of this post.
He graciously agreed to conduct an interview with me via email. Below are his thoughts on visual problem solving and how the brain really works.
JIM: The human visual system is very complex, but crucial in comprehending the world around us. In a nutshell, how can we best approach grasping the relationship between what we see, what we say and what we understand?
ANJAN: This is a deep question, one to which we have only provisional answers. The question can be rephrased as what is the relationship between perception and conception?
Perceptual systems have an analog structure, which makes sense because they make contact with and extract useful information about the external world.
So, the visual system from the retina to early visual cortex in the occipital lobe maintains an analog organization - for example, the upper parts of what we see is represented in the lower parts of primary visual cortex and so on.
Our conceptual system, which is best indexed by language, is inherently symbolic.
The neural organization of words (lexical-semantics) and combinatorial rules (syntax) is not related transparently to the organization of the external world. But language is enormously flexible and generative.
JIM: But the brain must tie vision and language together, right?
ANJAN: Since we can talk about what we see, these systems must connect, giving rise to several questions. What are the mapping principles between these different kinds of information? What are these kinds of information good for?
We are working on the idea of an intermediate level of representation called visual schemas that have both analog and symbolic properties.
Visual schemas are pared down representations of the kind you might see in icons or rough sketches. Essential relationships are retained but bleached of unnecessary detail.
Schemas are depicted by lines, simple shapes, vectors, etc. We think that such representations are not simply heuristic devices; they have neural reality.
Our brains are wired to process schemas that maintain a relationship with the external world by virtue of their analog structure and are flexible by virtue of their symbolic properties.
Externalizing schemas, such as in a sketch, allows them to compensate for our brain’s processing limitations. We have a limited working memory; we can only hold a few bits of information in mind at any one time.
JIM: Tell us a little more about the upshot of schemas. Are you saying that if I visualize a concept (e.g., sketch it) I'll understand it better because it's now hard wired in my brain?
ANJAN: The upshot of schemas is that they are accessible and give leverage. A brain that is tuned to schematic forms means that we perceive and understand them easily.
Unlike written words, which are also made up of simple lines and shapes, schemas are less subject to cultural variation and are easily accessible elements of thought.
Leverage comes in because schemas highlight relations rather than features (a circle above a line indicates “above” regardless of what figure object is above what ground). Relational thinking is critical to generating ideas.
Externalizing schemas, such as in a sketch, allows them to compensate for our brain’s processing limitations. We have a limited working memory; we can only hold a few bits of information in mind at any one time.
Externalizing them offloads working memory demands, and allows us to see relationships and combinations of relationships. An external spatial display compensates for an internal capacity limitation.
JIM: Fascinating. You’re suggesting that by externalizing schemas we can find new patterns that we wouldn’t have otherwise seen. Can you talk more about seeing new relationships? How does the brain do that?
ANJAN: Thinking about relations allows us to trade the precision of perception for flexibility and generativity.
What do I mean by that?
If I ask you to describe different kinds of lions you might have a pretty short list unless you are a big game hunter. If I ask you to describe different kinds of predators, you are likely to generate a longer and more interesting list.
Lion is an entity noun meaning that it refers to something with a set of features, whereas predator is a relational noun meaning that it refers to a set of relations that could apply to many entities with very different features (e.g. lion, eagle, shark, even lawyer).
JIM: I suppose you have researched this topic quite a bit, then.
ANJAN: We have preliminary data that suggest that the right supramarginal gyrus (SMG) is tuned to process simple visual schemas (http://www.ncbi.nlm.nih.gov/pubmed/22070948).
This observation is interesting because the left SMG seems tuned to phonemes- sound elements of language. The right and left SMG might specialize for visual and sound elements of visual and verbal thought respectively. Simple schemas can be concatenated.
The combinatorial power of putting them together offers a chance to “see” new things. We often see new relationships by using analogy and metaphor. We also have data from my lab, that the right rostral prefrontal cortex- one of the most recently evolved part of the brain, is involved in understanding visual analogies (http://www.ncbi.nlm.nih.gov/pubmed/21982934).
JIM: Let’s break down visuospatial analogical reasoning a little. What exactly is a “visual analogy” and can you give a simple example? In a nutshell, how does visual reasoning then help us solve problems exactly?
ANJAN: An analogy is a comparison in which objects that appear dissimilar share an underlying similarity. For example, this analogy is attributed to Einstein: “You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York, and his head is meowing in Los Angeles.”
Superficially, a telegraph and a cat are pretty different, and yet the analogy conveys an important message about the relation of a local action to a long-distance effect. Einstein uses language to evoke visual images to make his point. We also see relations in non-verbal information.
For example, this arrangement:
is analogous to:
but not to:
-- even though the third set contains the same objects as the first. The analogical similarity has nothing to do with surface properties.
Visual analogies help us solve problems by revealing hidden structural regularities despite surface differences. They allow us to apply relational knowledge from familiar situations to novel ones- such as thinking of a cat as being like a telegraph.
Sometimes the most creative solutions arise from the ability to disengage from surface properties of a problem to discover hidden relational similarities so that old solutions can be applied to new problems and new solutions can generalize to future problems.
JIM: What’s your opinion of a tool like Mural, and how it might help people better solve problems?
ANJAN: Not having used Mural as a tool, I will limit my comment to some features that strike me as useful.
The ability to use different formats- pictures, schemas/icons, and text allows one to capitalize on the advantages of each kind of display- the precision and accessibility of pictures, the accessibility and flexibility of icons/schemas and the flexibility and generativity of words.
Visual displays offer a powerful shorthand for communication.
- Grouping elements chunks information and creates an efficient nested organization.
- Size of elements can quickly indicate relative weight.
- Horizontal and vertical axes have informational resonance.
We think of time as moving left to right, meaning the horizontal axis can be used to indicate time. Non-temporal information can be pegged vertically. Higher is typically more important or desirable. The general point is that location within an overall display conveys its own analogical information without having to be spelled out.
The consequence of a visual display where the nature and organization of elements conveys information efficiently relieves working memory.
Participants can then devote more cognitive resources to manipulating elements and discover relations and patterns that would presumably help to creatively solve problems.
JIM: Thanks, Anjan! The human visual system is complex and fascinating for sure.
Learn more
Check out Anjan’s talk at Interaction London:
For more on visual thinking in Mural, see: "Do You Suffer From Workshop Amnesia?"
