About Genna Erlikhman

I'm a second year doctoral candidate in Cognitive Psychology at UCLA, specializing in Computational Cognition. I am broadly interested in perception, with particular interest in the perception of objects, wholes and parts, and affordances.

BLOG HAS MOVED

Hello Readers!

Thank you for an awesome year. We’ve had a great response and are hoping to expand to bigger and better things! Our new address is www.psychologyinaction.org. We will no longer be updating this site. Check out some posts on the new blog!

Please update your RSS feeds, daily routines etc.

 

Hope to see you there!

Advertisements

Mind-reading, lie-detection and telekinesis with fMRI and EEG – Science fact and fiction

In the last few years, fMRI and EEG have made it into the popular press as tools for reading minds (here, here, here, here and here for a sample), lie-detection (here, here and here), and telekinesis – controlling/ moving objects with our thoughts – (here, here, here, and here). I think there was even a recent episode of House where the doctors were able to display the dreams of someone in a brain scanner. While these and similar technologies have had many successful applications like allowing paraplegics to control wheelchairs and computers or allowing people with locked-in syndrome to communicate (at least give rudimentary yes-no answers to questions), it’s important to know the limitations of these tools and not get carried away into science fiction.

For people concerned about invasions of privacy or some sort of remote scanning of people’s thoughts or personalities, knowing how fMRI, EEG, MEG etc. operate is especially important. First, these tools do not read minds. You can’t plop someone into a brain scanner and be able to determine (right away) whether they are lying or not. And while one of the 60 Minutes clips linked above does show the reporter using EEG without any training to type letters with his thoughts, that device, like almost all others, is simply using a statistical trick in conjunction with some general properties about brain activity to produce its results.

The goal of this post isn’t to explain how fMRI and EEG work in minute detail, but to illustrate how researchers have exploited their properties to achieve what has been interpreted as mind-reading and telekinesis.

So this post ended up being a lot longer than I expected, so here’s a quick summary of what follows for those who just want a short version:

Some places have reported that people can control wheelchairs, robotic arms, computer cursors, etc. with their minds. Simply by thinking “go left” or “stop” the wheelchair will follow the commands it receives given only in thought. Almost all of these demonstrations are not quite true and instead rely on a statistical trick to get the devices to do what people are thinking. The computer picks up on a pattern of brain activity measured with either fMRI, EEG or intracranial electrodes. This pattern can be reliably reproduced by the person. For example, every time you think of stopping, the same part of your brain might become active. The computer then learns to pair this particular pattern with the “stop” command. This is not mind reading! It could just as easily have paired your thoughts of hamsters with the stopping command if those thought patterns could be reliably reproduced. Almost all of the tools mentioned in the articles linked above require this kind of learning on the part of the computer. It is a going to be a long while yet before we can stick someone in a machine and know exactly what they are thinking about. (Although we can sometimes guess about the category of thing that they are thinking about, like house vs. face or animate vs. inanimate object, but also with the proper learning on the part of the machine.)

The long version:

fMRI measures changes in blood flow to various areas of the brain over time. It is believed that blood flows to brain tissue that is active. So, the reasoning goes, if we put someone in the scanner, and ask them to think about, say, playing tennis, then blood should flow to the part of the brain that is involved with thinking about tennis playing. Furthermore, it is believed that when we think about the same thing again on a different occasion, the same brain area (more else) will become active. Indeed, this is most certainly the case (at least when we think about certain things) – for example, a region of the brain called the parahippocampal place area (PPA) shows reliable activity (blood flow to that region) every time a person thinks about a place or is shown a picture of a place (like a house). This finding is robust across many individuals (e.g., Epstein and Kanwisher 1998, Kanwisher 2010). (This region also responds to things other than places and lots of other parts of the brain are active when people see or think about places, but let’s ignore that for now.) There are several other regions of the brain that reliably activate to certain kinds of thoughts of percepts, like faces, certain actions, animate vs. inanimate objects etc.

This means that we can put a person in a scanner, show them a bunch of pictures, and pinpoint a part of their brain that responds more reliably to houses, say, than any of the other things. That is about as close as we can get to mind-reading at the moment: if we scan someone a bunch and find their “house area,” we can then tell, on a later scan, whether they are thinking about houses or not. A few remarks: (1) This isn’t perfect. As mentioned earlier, the PPA and similar areas activate to stimuli other than houses. (2) remember that fMRI measures blood flow to relatively large parts of the brain – while it’s potentially possible to find a “place area” and a “face area,” it is unlikely that with current technology we’ll be able to find a “hammer area” or the like. That is, if hammer representations are stored in a small, localized batch of neurons (a highly contentious claim!), fMRI is unlikely to be able to detect it. This means that we can’t stick a person in a machine, show them a bunch of pictures, find out what parts of the brain light up and then, whenever we stick them into an fMRI again, be able to tell what they’re thinking about.

However, we can exploit some of these more reliable activations. For example, we can correlate reliable signals with certain computer commands. We can train a computer to recognize the place signal and the face signal. When a computer recognizes the face signal, we can have a wheelchair turn to the left, when it recognizes a place signal we can have the wheelchair turn to the right. In this way, we can control the wheelchair with our thoughts. Notice that this is just a fancy statistical trick! If we could get the computer to reliably learn what our brains look like when we think about going to the left vs. going to the right, we can use our thoughts of going left or right to control the wheelchair’s direction. However, there’s nothing special about those particular thoughts! We could have just as easily correlated our thoughts about apples (assuming they could be reliable detected) with the motion of a wheelchair or a cursor on the computer screen.

Lie-detection can be done in a similar way – someone lies in a scanner repeatedly, the computer learns what that person’s brain looks like when they are lying (it finds a commonly activated area across all lies) and then can use that to determine whether future utterances are lies or not by comparing them to previous scans of false statements and true statements made by that person. Notice that this means that we first need to get the person in the scanner telling lies before we can actually use it as a lie detector. Likewise, for controlling a wheelchair or a computer cursor with our thoughts, we first need to train the computer on some sort of brain pattern that we can reliably reproduce.

The story is the same with EEG. While it measures electrical activity on the surface of the scalp instead of blood flow, some patterns can be reliably reproduced by thinking about the same thing or same category of things. A computer can learn these patterns and then, as with fMRI, correlate a pattern with a particular command or function. There also happen to be some well-known patterns of activity that we can use in a special way. For example, there is a signal called the P300 which is often associated with decision-making and is independent of the nature of the stimulus. We can exploit this to create a brain-controlled device that doesn’t require any training (learning what a reliable brain pattern looks like). Suppose we ask someone wearing an EEG cap a question and then flash the words “yes” and “no” alternatively on a screen. The P300 signal can sometimes be thought of as being a mental “That’s it!” If the person is thinking of “yes” as the answer to the question, then we should observe the P300 whenever the word “yes” is flashed on the screen. We can use this to ask simple yes-no questions of people who cannot communicate in a normal way (e.g., if they have locked-in syndrome) and we can do this without training the computer! Again, as with the fMRI work, this is just a statistical trick – we could just as easily have used a different, reliable signal to get to the yes/no answer; we’re not actually reading people’s thoughts.

All that being said, these tools are extremely useful. We can use them to help people communicate who otherwise could not. We can use them to allow people who cannot move to move wheelchairs, prosthetic limbs or computers. At the same time, it’s important to keep in mind the limitations of these technologies and not get carried away too far into science fiction.

Epstein R. and Kanwisher N. (1998) A cortical representation of the local visual environment. Nature vol. 392, 598-601

Kanwisher, N. (2010) Functional specificity in the human brain: A window into the functional architecture of the mind. PNAS, vol. 107, no. 25, 11163-11170

How do 3D movies work?

You might have heard that Avatar is responsible for a whole new kind of movie technology and that thousands of theaters had to be built or upgraded just to show these movies. What exactly are these technological miracles and if they’re so great, why do we still have to wear those silly glasses?

First of all, I want to point out that seeing in 3D is very hard. Sure, we do it all the time, but it’s actually a quite tricky process. Think of your eye like a regular camera. Light comes in through the pupil or aperture, is focused by the lens and then hits the back of our eye, which is called the retina. The retina, like the back of a camera, is a flat, rounded surface. That means that 3D objects in the world are represented in 2D on our retina. Our visual system, then, has to take this 2D signal and build back out of the 3D object that caused it. This is a difficult problem because lots of different 3D objects can produce the exact same 3D projection:

The Problem of Depth Perception

Irrespective of the inherent difficulty, our visual system does a pretty good job. Even when we go to the movies and watch something on a 2D screen, we have a fairly good understanding of where things are in depth. Think about the last movie you saw. Were you ever confused about which object was supposed to be closer to you/the camera? Or which objects were closer to each other? Below is a still of Billy Crystal and Danny Devito from Throw Momma from the Train. Even without a $20 movie ticket and fancy glasses, you should be able to tell that Billy’s head is a little in front of Danny’s and that the two of them are in front of a set of railroad tracks that recede into the distance.

Danny and Billy

There is a distinct sense of depth in the image. Evolutionarily, this ability is extremely important – knowing how close something is to you or how apart two things are in depth is critical for, say, swinging from branch to branch, reaching for fruit, or seeing whether you can successfully jump over the ditch or not.

One of the things that helps our visual system understand where things are is by how much the lens in our eye focuses. Just like in a camera, the lens focuses light. It changes shape to focus light differently depending on where the light is coming from. This is called accommodation, or the current focal position of the lens. Close one of your eyes and hold your finger out in front of you. If you focus on the finger, the things behind it will become fuzzy. Without putting your finger down, focus on the background. Now your finger should double and appear out of focus. This happens because muscles in your eye stretch or relax changing the shape of the lens. Knowing how much these muscles are stretched tells your visual system about how your lens is focused and whether the light that is being focused is coming from far away or from nearby. Unfortunately, even in 3D movies, you still look at and focus on a flat screen so that information won’t help you see depth.

However, there is a whole slew of information in the world that can help. These clues in the environment, or depth cues, tell us how far things are away from us (absolute depth) and how far apart they are from each other (relative depth). Here are some examples:

Occlusion: When a part of an object is blocked by another object, this tells us that the blocked object is behind the one that is visible. Devito’s right arm is in front of Billy’s sweater.

Shadow: Billy casts a shadow on Devito, not the other way around, suggesting that he is in front of him (based on where we infer the light source to be). In general, we expect light to come from above; this helps us understand whether objects are concave or convex. For example, makeup works the same way: if you want to emphasize your cheekbone, you would put a light color above it and a dark color below it to simulate light hitting it; this makes the bone look more prominent.

Perspective: The railroad tracks appear to get closer to each other higher up in the picture; however, we know that they are really parallel. The fact that they approach each other is a sign that they are receding into the distance. This is also called a pictorial cue.

Atmospheric Perspective: The top part of the picture is a little out of focus and foggy. That could be because of picture quality, but, in general, things that look hazy or out of focus are generally farther away.

Relative Height: Things higher up in pictures tend to be farther away. This is another pictorial cue.

Relative Size: There is no example of this in this picture. If there are two equally sized objects, say two quarters, then the one that is closer to us would appear larger than the one that is farther away.

There are also a few cues that we get from motion and moving objects (like parallax and dynamic occlusion), but I won’t go into them here.

All of these cues are available in regular photos and the motion ones can be seen on TV or in movies, so what’s special about 3D film? You might have noticed that all of these things can be seen in the environment with just one eye – they are monocular cues. If you close one of your eyes and look at the picture, it should look exactly the same. Partly, this is because no matter where you look in the image, your lens’ focus doesn’t change (accommodation). However, the same is true in the movie theater. The added sense of depth, both in 3D movies and the real world comes from the special property of seeing the same part of the world with two eyes.

Take your finger and hold it up in front of you again. Alternate between closing your left eye and your right. You should see slightly different parts of your finger depending on which eye you close. The difference between what you see in your two eyes is called binocular disparity and the information about depth provided to your visual system by that difference is called stereopsis. Knowing only the distance between your eyes, the difference between what each eye sees, and the distance to where you are fixating (accommodation), the visual system can recover the depth of a scene. How much the two eyes angle toward each other is called convergence, and this is also a hint about where objects are located in the world.

Accommodation and convergence

Remember the “old” 3D movies with the red and green glasses? They would show the same green and red picture, slightly shifted, on the screen. Each lens of the glasses would filter out one of the images, so each eye would see only one color picture. This would trick your brain into thinking that each eye is seeing the same thing from slightly different positions and would result in a perception of depth. If you’ve heard of a stereoscope, this is exactly the same idea, except in a stereoscope you would typically see stationary photos and here you could show a movie.

But these new 3D movies look even more 3D than those red and green ones! That’s because the red and green movies would use one camera to film the movie, then they would double each frame and project them over each other onto the same screen. That means that each eye would be getting exactly the same information! In reality, each eye sees slightly different things. The cool thing about the new movies (or at least Avatar) is that they are filmed with two cameras, about eye-distance apart so that the same scene actually looks a little different depending on which camera you look through.

In addition, they automated the cameras to angle in our out depending on how far away the focal point is. For example, if they are filming a closeup, the cameras would angle inward just like your eyes. Hold your finger out in front of you and slowly move it closer to your nose, focusing on it all the while. You should feel your eyes turning toward each other as it gets closer. The same thing happened with the cameras. The result is two shots of every scene, shifted a little bit depending on whether it is shot with the left or the right camera and adjusted depending on whether the objects are near or far. This gives an added sense of perspective to each shot.

There are different ways of getting the two images to the two eyes. The red-green glasses allowed only one of the color images to get to one of the eyes, while the other saw the other color image. Alternatively, there are shutter-glasses – the glasses are synchronized with the projector so that when the image for the left eye (shot with the left camera) is projected, the right eye lens is “shuttered” or blocked off and vice versa for the other eye. Since the frames are shown so quickly, your brain integrates the information and thinks that what your left and right eyes are seeing is there at the same time. The visual system then uses the differences between the two eye images, the perspective cues created by turning the cameras inward, and the rest of the monocular cues to give the sensation of depth. What seems to be used now are polarized glasses. There is a decent article on wikipedia that talks about 3D polarized glasses (see, for example, http://en.wikipedia.org/wiki/Polarization_%28waves%29#3D_movies). A special lens needs to be installed in front of the projector to display the two eye images on the screen. Just like with shutter glasses, often two polarizations are interleaved. A special screen is also needed.

This new technology is not entirely perfect. For one thing, the focal point does not change – the two images are still projected onto the same screen at the same distance from your eyes. This can cause headaches for some people. And this is why it’s still not quite as 3D as the real world. Also, stereo vision is only good for a few feet, so elements of the scene that are more than about 20 feet in depth (most of the background) will appear as if on one depth plane, that is, very flat. There can, therefore, be a rather jarring jump in depth between the foreground and background. While I think this is a cool way of filming and showing movies, there is no way that this technology will ever come into our homes glasses-free.

Incidentally, not everyone can see these movies in 3D. Some people are “stereo-blind.” This means that their brains use only the information from one eye. This might occur because their eyes are incapable of converging or focusing on the same point in space. There is an interesting article by Oliver Sacks in the New Yorker called Stereo Sue about just such a person who was stereoblind, but, with training, was able to learn to see in stereo.