Providing Feedback
The ability to provide feedback to a user is of utmost importance in human-computer interfacing studies because it makes communication between human and machine a closed loop process. Without feedback, learning to use a human-computer interface would be very difficult in some cases and impossible in others. Currently there are a few interesting technologies that bridge the gap from machine to human, but compared to the vast amount of research that has been conducted on reading biological signals into a computer the technology is still rather new and primitive.
Visual/Auditory Feedback
Visual and auditory feedback are the simplest and most primitive feedback mechanisms employed in human-computer interface systems because they use the natural senses and do not require any extra equipment. Basically in systems that use visual or auditory feedback, once a computer has decoded intention from a subject and used that intention to cause a hardware effect, feedback is simply observed by the subject. While these forms of feedback are easy to implement and are used almost exclusively to provide feedback in brain-computer interface systems that use a cursor system, they do have specific disadvantages for human-computer interface systems that are used to control robotic prostheses (or a subject’s natural muscle using FES). For prosthetic systems, visual and auditory feedback do not provide information about pressure, temperature, vibration, or grip strength. And even in brain-controlled computer interfaces, visual and auditory feedback may not be sufficient if a subject is lacking in these senses.[15]
Vibrotactile Feedback
Vibrotactile feedback makes use of tactors (small vibrating hardware) placed on the body to provide somatosensory feedback to a subject. Tactors may be placed in specific locations, which can be determined from a dermatome map, to be perceived by certain nerves. Modern tactors are often electromagnetic and function like small speakers, except instead of vibrating the air to produce sound waves, they vibrate the skin to produce pressure waves that may be sensed by Pacinian, Meissner’s, Merkel, and Ruffini corpuscles in the skin. Frequency, intensity, timbre (waveform shape), and duration may be modulated to produce different sensations from the same tactor[11]. Tactors are used currently in some commercially available prosthetics to provide pressure/grip-strength information. Currently researchers are also investigating how tactors may be used to provide feedback in brain-controlled cursor systems[11].
Figure 1. Tactor [11]
Direct Stimulation of the Sensory Cortex
Just as signals of movement may be read directly from the sensory-motor cortex, feedback signals may be fed directly back to the primary sensory cortex to generate tactile sensations. This process, often called S1 microstimulation, has so far only been tested in animal studies with mixed results. Scientists have concluded that quantitative information can be delivered to the primary sensory cortex in monkeys, and that the monkeys are able to make rough decisions based on the information (Miller et al., 2005, Oliviera et. Al, 2005), but currently the technology is imprecise does not deliver the constant stream of somatosensory information that is naturally provided by the peripheral nervous system. Also, because this feedback mechanism requires that electrodes be implanted into the brain, it carries biocompatibility and open-brain surgery risks. However, if the technology progresses, S1 microstimulation could provide an elegant way to replace lost sensation in disabled individuals.[15]
Peripheral Nerve Electrode Implantation
Peripheral nerve electrode implantation is a feedback mechanism in which small electrodes are implanted into peripheral nerve fascicles and used to deliver electrical pulses to the brain as if they had originated from natural sensory cells. The technology is similar in principle to vibrotactile feedback, but is more precise and only requires peripheral nerve residuals. Researchers Dhillon and Horch demonstrated in 2005 that electrodes implanted in the residual nerves of amputees could be used to provide graded movement and tactile signals to allow a subject to set prosthetic joint position and judge grip strength.[15]
Targeted Muscle Reinnervation
Targeted muscle reinnervation, discussed in more detail in Reading Signals, is a process in which residual nerves from an amputated limb are surgically rerouted to a non-functional muscle so that action potentials from those nerves may be read by electromyography. This technology was originally developed to read biological signals of intent, but recently scientists have found that the same rerouted nerves may be stimulated to generate sensations of movement or touch in the amputated limb. Researcher Todd Kuiken at the Rehabilitation Institute of Chicago and his team have mapped the sensations perceived by a TMR subject as shown in the diagram below:
Figure 2. Mapped Sensations for a TMR Patient[2]
Targeted muscle reinnervation is an exciting technology because it gives an amputee the ability to both control and sense a prosthetic as if it were the amputated limb. [2]
Synthetic Skin for Prosthetics
Synthetic skin is different from the feedback technologies discussed so far as it is used to read a signal from the environment rather than deliver that signal to the brain. Current prosthetics that provide sensory information (like Dean Kamen’s Luke Arm in Causing an Effect) usually do so through silicon based pressure, temperature, and position sensors. A synthetic skin would include embedded sensors such as these in a skin-like polymer. Researchers involved in the governmental DARPA prosthetics program are working to create a synthetic skin material made of polyimide and embedded with carbon nanotubules that is strong, flexible, temperature resistant, and piezoelectric. The skin is designed to sense pressure with its piezoelectric properties and to sense temperature through sensors placed under the polyimide layer. The nanotubules in the skin are very conductive to heat and when vertically aligned can transfer heat from the surface of the skin to the temperature sensors below the polymer very quickly. Other approaches to synthetic skin are also being investigated—in 2005 Japanese researcher Takao Someya and his team developed a different type of synthetic skin that made use of pressure sensitive rubber, organic temperature sensing diodes, and organic transistors used to relay pressure information. [36]
Figure 3. ORNL Synthetic Skin Cartoon