Video on the brain

  • Images are transmitted to the brain via each eye’s optic nerve. (A) These signals are first processed in the primary visual cortex. (B) Scientists examine the primary visual cortex to produce images from brain activity. Illustration by Richard Perez
  • The eye sees: a scene from a Steve Martin movie.
  • Reconstructed image: researchers record brain activity and match it with a video database, producing a human form with something around his neck.
  • The eye sees: an aircraft in flight, and foliage in the foreground.
  • Reconstructed image: a match of brain activity and database videos shows a horizontal shape, a blue backdrop, and darkening on the right.
Date:27 February 2012 Tags:, , , ,

Taking strides toward understanding how the brain processes stimuli to recognise images, researchers figure out how to project neural activity on to a TV screen. – By Steve Rousseau

How do they do it?
UC Berkeley professor Jack Gallant and his team use MRI to track blood-fl ow changes in a subject’s primary visual cortex – the brain’s largest visual processing centre – as he or she watches a movie. The researchers then create a model of the visual cortex that matches the blood-flow pattern with the images the subject is viewing.

Algorithms are applied to compare the brain signals with a catalogue of about 5 000 hours of YouTube video. The images that most accurately correspond to the brain activity are compiled into a composite video that resembles the YouTube footage.

“I think it’s very impressive that they can get these admittedly crude re-creations of our internal representations of video,” says Marcel Just, director of the Carnegie Mellon University Centre for Cognitive Brain Imaging, who was not involved in the study.

What’s it good for?
Such brain-visual linkages could one day aid communication with stroke or coma patients. Gallant says that once his team’s technique has been refined, it could also record and play back dreams. The obstacle to this is understanding how the brain’s visual processing changes when a person is sleeping or awake. Gallant is confident this will happen, commenting: “It’s only a matter of time.”

Video: Click Projecting neural activity on to a TV screen to watch a video of Gallant explaining how they succeeded in decoding and reconstructing people’s dynamic visual experiences.

Latest Issue :

May-June 2022