Research notes

Visual cues help us process what we hear

By mapping eye movements in fractions of a second, a Brown researcher has found that humans attempt to make sense of what they are hearing through visual cues long before they have heard an entire idea. The finding offers insight into how the mind uses vision to rapidly process information.

Julie Sedivy, assistant professor of cognitive and linguistic sciences, is interested in the process by which humans assign meaning to words and phrases. Psycholinguists know that as humans process language they make many split-second decisions about the words they are hearing. But questions remain about how humans cope with uncertainty at every stage of that moment-by-moment decision process.

In a series of studies involving approximately 150 people, participants sat either in front of a computer screen that displayed an image of objects or in front of a work surface set with objects and received verbal instructions concerning the objects. Researchers used a headband-mounted camera to map the participants' eye movements every thirtieth of a second.

Given a scene of a table set with a drinking glass and pitcher, the participants heard instructions such as "pick up the tall glass." Researchers found that participants frequently looked first at a pitcher in the display, indicating attempts to interpret "tall" early, and before hearing the entire noun "glass."

"On the basis of one or two sounds, we saw the participants' eye movements begin to shift," said Sedivy. "As soon as they identified a word, they began to map it."

However, when a short glass was added to the scene so that there were three objects - a pitcher, tall glass and short glass - participants were more likely to look at the taller of the drinking glasses when they heard "tall" because size was the distinguishing factor between the two glasses.

The finding suggests that humans consult a whole domain of information, including visual cues and expectations about rational communicative behavior, in resolving the uncertainty involved in processing a sentence, according to Sedivy.

There appears to be a set of mutual expectations between conversational partners, for example, that redundant information is typically avoided. In the example with the pitcher and two drinking glasses, "tall" would be redundant in referring to the pitcher, because there is only one pitcher, while there are two glasses, Sedivy said.

If that type of complex and subtle information were not available, the immediate moment-by-moment mapping of sounds to meaning would only serve to introduce a great deal of uncertainty to language processing, according to Sedivy. For example, if mapping to an object begins upon hearing "tall" rather than waiting until the following word "glass," given a scene in which there are two tall objects, the chance of an initial mapping guess being correct is only fifty percent.

Not subject to the conscious control by humans, the automatic eye movements are so subtle they are unnoticed by study participants, who may feel simply that their eyes are taking in the whole scene all at once when, in fact, the eyes are darting rapidly from one very specific location to another.

"This is a surprising relationship between highly intelligent processes of language understanding and low-level automatic processes such as eye movements," Sedivy said. "As humans, we have to deal with multiple levels of information simultaneously, and those different levels of information must be incorporated into the study of linguistics."

Sedivy conducted the research with Daniel Grodner, a postdoctoral fellow in cognitive and linguistic sciences; Anjula Joshi, research technician; and current and former undergraduate assistants including Charles Joseph, Estelle Reyes, Gitana Chunyo and Rachel Sussman. The work was funded by grants from the National Science Foundation and the National Institutes of Health. – Kristen Cole

A look at one state’s programs for gay, lesbian students

A two-year evaluation of the nation’s only statewide program for gay and lesbian high school students found more positive climates for sexual diversity in schools that had implemented the program than in those without the program.

Laura Szalacha (pronounced sha LAH ha), visiting associated professor of education at Brown, led the study of the Massachusetts Department of Education Safe Schools Program for Gay and Lesbian Students for her doctoral thesis in human development and psychology at Harvard University’s Graduate School of Education.

She interviewed 1,646 high school students and 683 faculty and professional staff between 1998 and 2000. There were Gay-Straight Alliances in 62 percent of the state’s secondary schools at the time of the study, a figure that has since increased to 71 percent.

Contrary to arguments that such programs promote homosexual behavior, the percentage of students who self-identified as gay or lesbian was the same – 7 percent – in schools with and without Gay-Straight Alliances, said Szalacha.

“There is no reason to believe this program does harm and every reason to believe it does good,” she said.

The study looked at the effectiveness of the Commonwealth’s four mandates within the Safe Schools Program for Gay and Lesbian Students: develop policies protecting gay and lesbian students from harassment, violence and discrimination; offer training to school personnel in violence and suicide prevention; offer school-based support groups for gay, lesbian and heterosexual students; provide school-based counseling for family members.

Students, faculty and professional staff were questioned on their knowledge of the official school policy, their assessment of the school’s sexual diversity climate in various dimensions, and their measure of the individual student’s behavioral comfort level with gay and lesbian issues.

Two factors appeared to contribute to the success of such programs in producing a safer climate in schools, according to Szalacha: faculty training in the program and a desire on the part of students for safer sexual diversity climates.

Thirty-five percent of the students in schools with Gay-Straight Alliances reported that gay, lesbian and bisexual students could safely choose to be open about their sexual identity, as opposed to only 12 percent of the students in schools without the alliances.

Szalacha recently presented her findings at Brown Center for the Study of Human Development. Massachusetts implemented the program with recommendations from a Governor’s Commission; Rhode Island has conducted its own Governor’s Commission but has not yet implemented any measures, according to Szalacha. – Kristen Cole