Interfaces and Keyboards For Human-Dolphin Communication: What Have We Learned?

:

Interfaces for studying the cognitive and communicative abilities of various primate and bird species have been used for decades.Over the decades a variety of unique technologies have been designed and employed to access the sensory systems and cognitive tendencies of different species.For example, primate work has focused on manual dexterity tasks and computerized "touch" keyboards, and bird work has tapped into species' vocal mimicry abilities.Adapting such work for dolphins requires special technological considerations due to their aquatic environment and their acoustic sense, which is often primary to vision.
Much of what we understand about dolphin cognitive abilities comes from playback experiments and comprehension tests.Playback of acoustic signals, and the monitoring of behavioral reactions, is one of the more common applications for research with dolphins (e.g., Bruck, 2013, Sayigh et al., 1999, for signature whistle playback to related dolphins; Deecke, Slater, & Ford, 2002, for orca (Orcinus orca) playback to harbor seals (Phoca vitulina)).Early studies of dolphin cognition focused on comprehension of symbol use.Using a computer-generated sound-based language system with one dolphin (Tursiops truncatus) and a human-produced gestural language system with a second dolphin, Herman, Richards, and Wolz (1984) demonstrated the dolphin's understanding of both the semantic and syntactic components of imperative sentences.
We also know that dolphins in experimental situations can learn to label objects with symbols, both visual and acoustic.For example, we know that at least some toothed whales (e.g., dolphins, belugas (Delphinapterus leucas)) are good acoustic mimics and sometimes spontaneously mimic human speech (Ridgway, Carder, Jeffries, & Todd, 2012).Richards, Wolz, and Herman (1984) taught dolphins to vocally label individually presented objects by mimicking computer generated whistles and burst-pulse sounds.Although successful in this task, the dolphin did not subsequently utilize these same vocalizations to label any objects spontaneously.In other related work, dolphins trained to label objects with sounds also produced these sounds as a request for the named objects (e.g., Herman & Forestell, 1985;Herman, Pack, & Morrel-Samuels, 1993;Murayama et al., 2012).Given the propensity of dolphins and other species to either use referential signals in the wild (Seyfarth & Cheney, 1990;Slobodchikoff, Kiriazis, Fischer, & Creef, 1991) or artificial systems in the lab (Herman et al., 1993), pursuing language interfaces seems reasonable.
However, mimicking a signal may not be the same as understanding its function.As noted in bird cognitive work (e.g., Pepperberg, 1981), vocal object labels can be imitated prior to an understanding of the referring function of these symbols.That the referential function was, ultimately, understood was demonstrated through the use of these labels in different contexts, including in the absence of the objects they represent.As a result, designing and implementing an interface that both exploits the dolphins' natural mimicry abilities, while also allowing cognitive exploration of the functional use of such mimics, is desirable.
There is a history of keyboard type interfaces for dolphin studies, used to provide a means for dolphins to make requests of a human companion.In such studies, each key was associated with an object, event or location, and also with a unique sound.By employing technology that can discriminate and produce such sounds, a two-way interface can be developed.Such two-way systems offer multiple advantages, some that are unique to these systems.These include: (1) building a bridge across sensory systems with technology, which may be necessary for communication between species with different sensory abilities, (2) using computerized information that enhances data acquisition and processing of information, and (3) empowering a nonhuman operator of the system to elucidate previously unexplored cognitive processes or strategies less apparent from other approaches.
This paper reviews some of the relevant interfaces used in dolphin research.We will review the technology, the methodologies, and look at the efficacy of these tools over the decades.Then we will look at current systems that are being employed to bridge the gap for dolphins in a variety of cognitive or communication studies.

History with Primate and Bird Interfaces
The majority of interfaces have been created for work with social mammals, specifically primates, and social species of birds.Computerized symbol interfaces, based on "finger" operated touch screens were eventually designed for a variety of taxa.Here we review some of the early attempts to communicate or develop protocols and interfaces, along with their contributions or mistakes.
Early Work (1950's and 1960's+).Human attempts to encourage primates to produce English words were some of the earliest experiments in interspecies communication (Hayes & Hayes, 1952).The unrealistic expectation of asking a nonhuman with the wrong anatomy to produce human vowels and consonants was a major deterrent to success in this work.It serves as an early example of an inappropriate request to make of another species.
Later researchers discovered the more natural communication modality of chimpanzees (Pan troglodytes) in gestural articulations and utilized this discovery to engage in sign language studies with chimpanzee subjects (Fouts, 1973;Gardner & Gardner, 1969, 1975a, b;Terrace, Pettito, Sanders, & Bever, 1979).This too served to illuminate the importance of finding a species-appropriate modality (in this case gestural signs) that could enhance interspecies communication.This method had the added advantage of enabling researchers to watch the transmission of shared gestural information between the chimpanzees themselves.In fact, researchers did observe chimpanzees passing down what they learned to their offspring.
Shortly after, other scientists (Rumbaugh & Gill, 1977) were designing technological interfaces to enhance the study of communication.This included a touch keyboard that utilized visual symbols called lexigrams and provided direct treats to Lana the chimpanzee.A key motivation for using lexigrams was that, as signals, they were less ambiguous than many of the apes' gestures.This study served as an attempt to take humans out of the equation of communication, as Lana interacted only with the keyboard and only as a successful means of receiving food.
Later Work (1970Work ( 's, 1980Work ( 's 1990's)'s).In time, other researchers began working with a variety of primates, especially the great apes (Gorillas (Gorilla gorilla) -Patterson, 1978, Orangutans -Miles, 1983).In many of these sign language studies, researchers described the spontaneous use of, and combinatorial aspects of the gestures during the process of communication.Meanwhile, Savage- Rumbaugh and colleagues (1993) were working with common chimpanzees and pygmy chimpanzees (bonobos, Pan paniscus).In contrast to the research done with Lana, tests utilizing cooperative tasksin which the animals used the keyboard to make requests for one another's helpwere particularly successful with the chimpanzees, Sherman and Austin.In this case, the subjects did show immediate transfer of the meanings of their lexigrams to novel contexts, suggesting that the two-way, communicative nature of the task facilitated such understanding.
Research with bonobos also utilized a touch keyboard with visual symbols, ended up producing major breakthroughs (Savage- Rumbaugh & Lewin, 1984).The infant Kanzi, while observing his mother explore the keyboard, was able to use the keyboard himself without individual training.After that Kanzi became the main subject of the keyboard work.Eventually a portable keyboard was developed so the researchers and Kanzi could interact in a rich social environment in a wide variety of settings, much more akin to that in which human children are exposed to language.Perhaps this was the best example of a tool for bridging the interspecies gap as humans were now seen as important as a social participant in the process of exploring interspecies communication.
At this time other researchers began exploring the cognitive abilities of an avian species.Irene Pepperberg (1999) and her team were now working with African Grey parrots (Psittacus erithacus), a species that could accurately imitate human speech, and so a keyboard interface was not needed.Working with Alex the African grey parrot, vocal imitation was specifically requested and shaped during training (Pepperberg, 1981).The "Model/Rival" training technique that they used was particularly effective.In this technique, a trainer first requests a response -such as asking for the name of an object -of another human, while the parrot watches.If the human says the name correctly, the person is given the object.The parrot, watching this interaction, presumably takes this second human as a rival for the attention of the trainer, which encourages the parrot to respond when the trainer then turns to the bird with a similar request.Plus, if Alex provided an acceptably accurate vocal label, he too was provided with the named object as a reward.Under this regime, Alex learned a large repertoire of English wordsincluding names and descriptorsand could use them appropriately to identify and classify objects and their relationships.This work served to revitalize some of the most critical concepts in interspecies communication; a being needs to be motivated to communicate and it helps to observe the communication system before interacting.By applying the "model-rival" concept, which more closely resembles the process of natural communication in a social species, Pepperberg was able to expedite the process of functional understanding of words by a nonhuman species.

History with Dolphins Early Work (1960's+).
Acoustic systems.In the 1960's the U.S. Navy, along with researchers in Hawaii (Batteau & Markey, 1967) devised an acoustic communication interface (Figure 1).Advanced for its time, the system arbitrarily matched a dolphin whistle contour with a human vowel or phoneme.A computer translated combinations of human vowels to generate sinusoidal dolphin whistles.The dolphins learned to respond to 35 command strings generated by humans using five-word sentence structures.In studies with the dolphins Puka and Maui these sounds were projected into a lagoon where they were heard by the dolphins who successfully executed the commands.The dolphins also learned to mimic and repeat whistles on command.Essentially the system was an acoustic command system, and although the dolphins could mimic the whistles, and learn to understand them as streams of commands, no functional or combinatorial understanding was evident.This early study served as a species-appropriate signal design, albeit one-way and command oriented.Also in the 1960's John Lilly, a neurophysiologist, began working with dolphins (Lilly, 1965(Lilly, , 1978;;Lilly & Miller, 1961a, b).A house was built in the Virgin Islands where trainer Margaret Howe lived with and interacted with Peter, a male bottlenose dolphin (Figure 1).In a method reminiscent of Hayes' early work attempting to shape the mouth of chimpanzees to speak English words, Howe attempted to get the dolphin to mimic English words.Regardless of the species-inappropriate assumptions, some indications that prosodic features (rhythm, intensity) were mimicked are evident on the videos (e.g., Lilly, 1965).However, no functional understanding of words was observed in the study.
John Lilly also started Project JANUS (Joint Analogy Numerical Understanding System) at Marineworld Africa USA in northern California (Figure 1).Although not well documented, this interface used human words that were turned into series of high-pitched clicks (32 khz).Dolphins learned sequences of "clicks" as commands and executed these commands.Again, this 40 whistle-word vocabulary and software emulated dolphin vocalizations.As in Batteau and Markey's (1967) work, the animals showed mimicry and an understanding of trained commands, but did not demonstrate further functional understanding.The work was incomplete, and did not scientifically assess other communication aspects of the system.
Although such acoustic interfaces had the "potential" for the dolphins to mimic these signals back to humans or to conspecifics, these systems were not designed to be two-way in nature.Neither project involved rigorous testing, and few results were published in scientific sources.Only reports and internal documents exist to assess the results.In addition, computer technology was just beginning to emerge and at the time did not have advanced features such as fast processing and pattern recognition programs that might have enhanced these systems.

Keyboard interfacestank-side visual and acoustic.
Informed by the limited success of the early acoustic interfaces, and inspired by the relative success of keyboard interfaces in ape language studies, subsequent work with dolphins on two-way communication focused on keyboards.Reiss and McCowan (1993) implemented a tank-side keyboard at Marineworld Africa USA in northern California (Figure 2).Their keyboard incorporated visual and acoustic symbols and were activated when the dolphins, or humans, pushed the physical key.After the dolphins began to experience the contingencies of receiving one of several toy objects or desirable actions (e.g., receiving a rub) after the activation of a key, the dolphins spontaneously produced mimics (defined as a facsimile immediately following computer production) of the symbol-associated sounds.They also produced facsimiles of these sounds, and combinatorial whistle facsimiles, in behaviorally appropriate play with the objects, or even preceding a key-hit on the keyboard during sessions.The researchers also reported spontaneous production of the whistles with conspecifics during non-testing play times.Around this time, Delfour and Marten (2006) implemented an underwater computer touch screen with visual and acoustic symbols at Sea Life Park on Oahu, Hawaii (Figure 3).Researchers observed through an underwater viewing lab while the dolphins operated or "touched" the screen by breaking an infrared beam (inside the tank).The disruption of this underwater infrared beam essentially allowed the dolphins to point to the computer screen (behind the glass).Dolphins could create action on the screen (movements, sounds) by pointing to objects and images using the infrared trigger.Humans sometimes Herzing 248 modeled the system at the underwater window, demonstrating the contingencies of the system, while the dolphins watched from in the tank.Multiple dolphins explored the interface during the study.In both Reiss's and Marten's systems the researchers interacted with the dolphins either at the surface of a tank or through an underwater window.Both systems allowed the dolphins to explore the system in addition to experiencing specifically designed tasks.Both systems successfully incorporated human interaction to encourage attention to the keyboard and to demonstrate associations or contingencies of the communication system.In 1992Xitco, Gory, and Kuczaj (2001, 2004) designed and initiated use of an underwater keyboard at the Epcot Center in Orlando Florida (Figure 4).This large underwater keyboard held objects in recessed tubes.Dolphins or divers operated the keyboard by breaking an infrared beam to "point" in the tube that held an object.English words were then generated for the diver's knowledge.Similar to Pepperberg and Savage-Rumbaugh's approaches these researchers used a human-based modeling procedure in which a dolphin watched as a model and receiver respectively activated and responded appropriately to key presses.Eventually, each of two dolphin observers learned to respond to human-activated keys and to activate keys themselves, often swimming ahead of the human to the object or location associated with that key.Also in the late 1990's Herzing, Delfour, and Pack (2012) implemented an underwater keyboard in the Bahamas with a semi-habituated pod of free-ranging dolphins (Herzing, 1997(Herzing, , 2011)).The visual and acoustic system (Figure 5) was activated when the human activated a sound button or when a dolphin pointed or oriented to a physical key.The system then generated a whistle that was associated with the visual symbol.Using the "model/rival" concept, humans in the water modeled and used the system between each other, exploring the contingencies of the system in the presence of the dolphins.Dolphins also had the opportunity to observe humans using computer-generated sounds in conjunction with humaninitiated gazing and pointing behaviors to request various play toys with the overall goal of examining whether and in what contexts the dolphins would begin to produce the same sounds.A small number of female juvenile dolphins were exposed to this keyboard over a four-year period.

Keyboard interfaces -underwater visual and acoustic.
Both of these systems had human divers interact directly with the dolphins and the system in real time.Both systems also incorporated modeling between humans to demonstrate contingencies of the system to the dolphins.Both were also designed for exploration by dolphins and had visual and acoustic cues to allow human clarification of dolphin behavior.One constraint in these studies was the speed at which humans could "engage" with the dolphins (or swim to respond), often hampered by the difference in swimming abilities between the two species.Keyboard interfaces-back to an acoustic system.In the late 2000's Amundin and Starkhammar created ELVIS (Echolocation Visualization Interface System) at Kohlmarden Park in Sweden (Amundin et al., 2008;Starkhammar et al., 2009).A forty-seven-element hydrophone array tracked the sonar beam (Figure 6).Primarily a tool for visualizing sound pressure level dynamics in a sonar beam ELVIS could, in effect, also act as an interactive acoustic "touch screen."The system also could project visual symbols on the tracking screen to associate sounds with a symbol.This system has great potential for both cognitive and communication studies.In 2010, Herzing and Starner joined forces to design a fully acoustic underwater interface, CHAT (Cetacean Hearing Augmented Telemetry), with real-time underwater sound recognition.CHAT is a wearable underwater computer (Figure 7) that sends and receives sounds and has real-time sound matching for pre-programmed artificial sounds in the system (Kohlsdorf, Gilliland, Presti, Starner, & Herzing, 2013).The system is activated when a diver pushes a key on a keypad or when the dolphins mimic a whistle accurately; the whistle is then matched in real-time by the computer.CHAT is also an interactive acoustic keyboard that records every keystroke as well as high frequency sounds up to 96 kHz.
Human researchers used the "model/rival" technique to demonstrate, using multiple CHAT units in the water, to the free-ranging dolphins the contingencies of this system.Four objects were labeled with synthesized whistles that the dolphins can mimic.The system recognizes whistle matches, producing an English word equivalent to an earphone worn by the diver.As a result, the diver can provide the requested toy, or demonstrate which whistles labels a specific object.In addition, the system can play specific signature whistles created for each of the researchers, along with actual signature whistles of prime dolphin candidates who work with the system on a regular basis.This allows the system to function like dolphins do (e.g., to greet each other with their signature whistlessee King & Janik, 2013), at least to some degree.The system has all the advantages of a computer; it is fast, data is fully recorded, and programming changes can be made in the field if needed.However, challenges in the field are different from tank-side interfaces.Weather, getting enough exposure with the same dolphins, and underwater computers in salt water (vs.tank side) all compound technical and user issues.Both Herzing's and Starkhammer's system utilize computing power in the acoustic realm that has not been used before in two-way interfaces.In CHAT the power of the system is alerting the human in real time that a match or mimic of a whistle has been produced, allowing the researcher to respond quickly.In ELVIS the dolphin's own sound is used to activate the system and allow exploration of the dolphin's natural calls.While CHAT is designed to be fully interactive between dolphins and humans, ELVIS is currently used more to investigate echolocation use, although interactive scenarios could potentially be developed with this tool as well.Both systems utilize acoustic triggering, a new way for a dolphin to "activate" a keyboard, rather than physically touching a key, which may not be as natural for a dolphin as producing sound.

Discussion
What have we learned over the decades about the efficacy and usefulness from these different interfaces and methodologies?Interfaces have used visual forms, 3-D objects, and acoustic signals.Some systems have been tank-side, others underwater.Some interfaces have used physical "touch" triggers (like a key or infrared beam), while others have used acoustic triggers.Some interfaces involved direct and detailed human interaction, whereas others kept the interface separate from humans.In many cases, early systems were completely one-way, and represented more of an operant-based training scenario where humans simply gave dolphins commands to follow, either visually or acoustically.We will outline what has worked and why and also what has failed and created stumbling blocks in the work.
1. Why did we go from sound symbols to visual interfaces?Researchers initially started with acoustic interfaces (knowing that dolphins were primarily acoustic), but at the time computers and other available technology were simply not adequate to do the job.We still know very little about meaningful features, or flexibility of sound aspects, of dolphin vocalizations even to this day.2. Why did we follow a primate analogy for our keyboards, vs. develop species-specific interfaces for dolphins?The primate field had successfully utilized visual and physical keyboards with chimpanzees.Dolphin researchers followed these techniques partly because they were accepted and partly because being terrestrial creatures ourselves it is simply easier for us to operate something tank-side then it is to operate apparatus underwater.Adding a true underwater and fully immersed scenario is complicated by our human abilities (and limits) and our technology.3. Why so many types of triggers?Going from a "physical touch" activation system to infrared beams allowed the dolphin to "touch" something using its long rostrum by breaking a beam of light.Of course, going to an acoustic trigger is the ultimate as a dolphin "touch" option, since it does not require immediate proximity with the device, allowing the animals to interact while swimming freely.But again, technology has only allowed us to explore this option recently.4. Should we involve humans or not?Many social species interact with each other and so interspecies communication is a legitimate focus of interest.Using human researchers, and their facility for relating to other species, can promote the learning and understanding process, as long as data are rigorously recorded and objectivity is not compromised.5. Why haven't we used better technology in the past?As is often the case, science often depends on new tools to ask new questions.Technology can lag even when ideas are plentiful.We simply had no good acoustic pattern recognition tool in the past, and small acoustic devices of any nature were not available until the last 10 years or so.Plus, many of these approaches are expensive and time consuming, since most programs and equipment are custom made and remade.These factors add to the challenge of updating equipment and sustaining a project of this nature.
Other studies have shown that nonhuman animals, like Kanzi and Alex, not only make requests but can also make statements and answer questions.Dolphin studies have attempted to provide a means for those animals to make requests of a human companion by using a keyboard in which each key was not only associated with an object, event or location, but also with a unique sound.Clearly assessing interspecies communication has evolved over the decades by designing and enhancing speciesappropriate interfaces along with a better understanding of the communication process both within and between species.Many elements of dolphin communication are relatively unstudied but may be of interest when designing an interface.These include acoustic prosodic features including rhythm and synchronization (Herzing, 2015) or perhaps emotional cues found in dolphin vocalizations.Creative innovation can be made in the future with emerging improvements in technology and new ideas and approaches to engaging dolphins and other underwater species.

Figure 1 .
Figure 1.Early interface systems for dolphin communication.Upper left panel: Wayne Batteau and the Man/Dolphin Communicator.Upper right panel: John Lilly's Project JANUS.Lower panel: Margaret Howe in the Virgins Island Facility with Peter, a bottlenose dolphin.

Figure 2 .
Figure 2. Tank-side keyboard with physical visual symbols and associated sounds used at MarineWorld Africa USA.

Figure 3 .
Figure 3. Underwater touchscreen using infrared touch used at Sealife Park, Oahu, Hawaii.Reprinted with permission from Elsevier: Behavioural Processes.

Figure 4 .
Figure 4. Diver working with dolphin at EPCOT Center's underwater keyboard in Orlando Florida.Picture used with permission -Walt Disney World Corporation.

Figure 5 .
Figure 5. Underwater keyboard with visual and acoustic symbols used in the Bahamas with free-ranging Atlantic spotted dolphins.Picture used with permission -Wild Dolphin Project.

Figure 6 .
Figure 6.ELVIS -Underwater acoustic touchscreen for tracking sonar beam used at Kohlmarden Park in Sweden.Reprinted with permission from AIP Publishing LLC: JASA through CCC.

Figure 7 .
Figure 7. CHAT -Underwater wearable computer with real-time sound recognition used in the Bahamas with free-ranging Atlantic spotted dolphins.Picture used with permission -Wild Dolphin Project.