PROVIDENCE, R.I. [Brown University] — Brown undergraduates Adrienne Tran and Max Song didn’t just take Thomas Serre’s machine vision course in Fall 2015 — they ultimately took an innovation from his lab to consider its potential for commercialization.

The technology, which Serre’s lab has developed with Brown colleague Kevin Bath over the last several years, automates the observation of research animals. Normally students and other research assistants must painstakingly review hours of video footage to determine when and for how long a mouse sleeps, wakes, feeds, drinks, grooms another mouse, nurses its pups and so on. Those factors are key to understanding the potential effects of an experimental intervention on the mouse’s behavior.

But now that can be done more accurately and much faster by artificial intelligence software developed by Serre (who studies vision in machines and in the human brain) and many lab members including computer science alumnus Zack Nado, postdoc Sven Eberhardt and staff engineer Youssef Barhomi. Bath, who uses mice in his research on stress and mood, has helped to refine it by advising the team on mouse behaviors and which ones should be monitored.

“We had been talking about doing a company for a while in order to optimize the technology in the commercial setting, but I’m an academic and I’m an assistant professor and 99.9 percent of my time is teaching and research,” said Serre, who is based in the Department of Cognitive, Linguistic and Psychological Sciences and (along with Bath) is a faculty affiliate in the Brown Institute for Brain Science.

“This was all about finding a good opportunity, and a good opportunity means finding talented young people who can push that toward a business, which is something that I have no expertise in.”

Last September, Tran and Song were friends and roommates who had just returned to Brown after taking more than a year off to explore the Silicon Valley startup scene. Back for their last semester before graduation, they debated whether to take Serre’s class.

“I wasn’t going to take the course, but Max came home every day very enthusiastic about the lectures,” Tran said. “When we became involved in the research, we realized all the exciting projects in the lab that, if shared with more people, could provide high impact to society, starting with the life sciences field.”

So they secured Serre and Bath’s blessing and assistance from Katherine Gordon, who directs the University’s Technology Ventures Office, to spin the technology out of Brown. They invited friend and Brown alumnus Kurt Spindler, a former software engineer at Uber, to work with them and Barhomi.

Less than a year later, they’re actively evaluating whether there is a business model in providing the technology’s automation of lab animal observations to drug companies or other commercial research concerns, which might license aspects of it for their operations or pay for automated monitoring as a service.

The concept, which they dubbed “Neurocurious,” is showing early promise. In May, the team won the MedTech Award at the Rhode Island Business Plan Competition, which provided a cash prize and in-kind professional and consulting services. Later in the month Tran and Song skipped Commencement (they did attend the Midyear Completion Ceremony, where Tran was an orator) because they were invited to present it at the Pioneers Festival in Vienna, Austria. They also won a National Science Foundation “I-Corps” grant to further explore the idea.

A better mouse monitor

Serre first set off down this research road while he was a postdoc studying computer vision at the Massachusetts Institute of Technology. There scientists Christof Koch and Andrew Steele challenged him to come up with a better way to monitor mice. Serre and colleagues came up with a first version of the technology and published it in Nature Communications in 2010.  Even then, the proof-of-concept system — which his lab at Brown has since completely remade — annotated basic mouse behaviors as well as people did.

By that time he had joined Brown’s faculty and had met Bath, who is now an assistant professor of cognitive, linguistic and psychological sciences. Bath quickly saw the utility of it in his research. Ever since the two have collaborated to make the technology better and better.

“The system has improved throughout the years,” Serre said. “What we’ve been able to do in the past couple of years has been to increase the reliability and the repertoire of behaviors that we’re able to recognize.”

The underlying technology is an application of “deep learning,” the same type of artificial intelligence technology architecture used by Google and Facebook to analyze images to determine their content.

Serre’s team programmed the software to learn from expert human annotations of animal behavior video and then do the annotation itself. In a small room, a bank of cameras controlled via a web interface watches each cage 24 hours a day, seven days a week. The video data is stored — about a million gigabytes so far — and the analysis is performed in Brown’s large Center for Computation and Visualization clusters. The system, Bath said, can do in minutes what would take human analysts months to do. And it does so with no danger of human biases.

As the team advances the system, it’s the collaboration with Bath that continually makes it more applicable for scientists. It is Bath, after all, who knows what conditions they must be kept in, how they behave, and what aspects of that behavior must be reliably observed and recorded.

“Before this we used tracking tools that were relatively crude in that they tell you where the mouse is in two-dimensional space, but it didn’t tell you much more than that,” Bath said. “The ability to use computer vision to go above and beyond that to actually identifying behaviors is a huge leap.”

The system now recognizes about 25 behaviors, and there are several new goals. Among them, Serre said, is getting the software to notice when behaviors have become more or less frequent or occur in a sequence, recognizing social interactions among mice, and analyzing behavior of other animals, including people.

All the while Bath has turned the system into a significant research tool on the Brown campus and beyond.In 2012, with funding from the Brown Institute for Brain Science through the Robert J. and Nancy D. Carney Scientific Innovation Fund, Serre and Bath opened their lab up as a “core” called the Rodent Neurodevelopmental Behavioral Testing Facility. Bath’s Early Career Award from the Brain and Behavior Research Foundation supported some of the earliest experiments to use the facility. The core facility is available to the entire Brown research community as well as scientists outside of the University.

Many scientists at Brown and Providence College have used the core for studies on ALS, the biology of aging, circadian rhythms, anxiety and mood disorders, autism and basic neuroscience. They have added to the expertise Bath has provided in refining the system’s abilities and applicability.

Enter the entrepreneurs

As the technology was developing so, too, were the entrepreneurship skills of Tran and Song.

Tran came to Brown from San Francisco, where she was already steeped the area’s legendary startup culture, having worked at one. But she didn’t get involved just for the sake of launching a startup.

“As a designer and developer, I both had a desire and felt an obligation to invent technology that makes our lives easier, rather than accept all the time humans lose to repetitive and tedious work,” she said.

Song, who came to Brown to study bioengineering, caught the West Coast startup bug after he joined Brown’s International Genetic Engineered Machine team in 2011. The team was a collaboration with Stanford University and while out in Silicon Valley (at one point he attended a lecture sitting next to Apple co-founder Steve Wozniak) he said he realized entrepreneurship was an alternative track to research for making a positive contribution to the world.

With mutual interests, Tran and and Song met in 2012, their sophomore year, and co-founded Brown Venture Labs (now it’s the Brown Venture Fellowship) to share entrepreneurship with their fellow students.

To gain more experience, both Tran and Song took time off from Brown to work in Silicon Valley. Tran worked at the Human Advancement Research Community, a lab founded by computer science legend Alan Kay.

“HARC’s mission is to invent technology that allows all humans to see further and understand more deeply,” Tran said. “My experience at the lab was formative in thinking about how I ought to spend my time.”

For Song, starting from a summer as a teaching fellow at Singularity University, an institute of futurists devoted to artificial intelligence, he became enamored with the potential of machine learning to accelerate biological research. During his time off from Brown, he worked for Ayasdi, a machine learning company spun out from Stanford.

That’s why by his last semester at Brown, Song was so interested in Serre’s class.

“When I took Professor Serre’s vision class, I was immediately struck by his pragmatic thinking — all of the homework sets were applied and involved programming, giving the students a chance to not only know a concept abstractly, but really implement it in code,” Song said. “I requested the chance to work in his lab to learn more from him, and in the process discovered that there was this amazing wealth of research in the mouse behavioral modeling.”

It’s still too soon to say what will happen from here, but Song and Tran’s curiosity, building on the research of Serre and Bath and Barhomi’s focused engineering, could take the technology far beyond Brown and into the marketplace.