Augmented Cognition (augcog)

Augmented Cognition (AUGCOG) is a term that originated with a DARPA program that sought to enhance human information processing capabilities through the design of adaptive interfaces that heavily leverage neuroscience technologies. The DARPA AUGCOG program ended in FY2006 under the direction of LCDR Dylan Schmorrow, but similar DARPA research has continued under Dr. Amy Kruse under the heading of Improving Warfighter Information Intake Under Stress.  However, formal AUGCOG research currently continues under the U.S. Army, headed by Henry Girolamo, the advanced technology manager, integration and transition, at the U.S. Army Natick Soldier Research, Development and Engineering Center. This section will first discuss the original DARPA AUGCOG program, and then will discuss the more recent work under the U.S. Army AUGCOG program.

Gaining information superiority though greatly enhancing an operator’s cognitive abilities was proposed to occur through the use of neurological and physiologic measures (e.g., respiration, heart rate, EEG, functional optical imaging), in conjunction with traditional electromechanical computer input devices (e.g., mouse, joystick). In a futuristic operational setting, an operator would wear a headset which would combine electroencephalograph (EEG), functional optical imaging, eye-tracking, etc. Through the measures generated by this headset, as well as those via galvanic skin response sensors, force feedback in input devices, pressure sensors in a seat, etc., the system would enhance an operator’s cognitive ability by dynamically controlling the rate as well as source of information.

The former program manager of the DARPA AUGCOG project, Dylan Schmorrow, envisions that within ten years, technologies developed as a result of the AUGCOG program will be operational, and in twenty years, he envisions that the resulting technology will be pervasively woven into the fabric of our daily interactive computing lives (Schmorrow 2006).

Proof-of-concept for the DARPA AUGCOG program occurred in two phases. In the first phase, researchers attempted to detect changes in cognitive activity in near real-time in an operationally-relevant setting. In phase 2 of this validation process, an operator’s cognitive state was manipulated as a result of the measurement technologies developed from the first phase.

For Phase 1, one relatively large experiment/demonstration, called the Technical Integration Experiment, was conducted for the first phase with mixed results (St. John, Kobus et al. 2003). The objective of the experiment was to determine which psychophysiologic measures could consistently detect changes in cognitive activity in a supervisory control task.  Using 20 cognitive state gauges (CSGs) such as EEG, fNIR, and body posture measures, as well as input device measures (mouse pressure and mouse clicks), eight subjects completed a series of four simplified aircraft monitoring and threat response tasks in one hour. While eleven of the CSGs were reported as significant and reliable (including FNIR and EEG measures), no CSG was significant across all three of the independent variables (St. John, Kobus et al. 2004). Moreover, the only CSGs that demonstrated statistically significant results across two of the independent variables were mouse clicks and mouse pressure, which are arguably not the best indicators of neural or physiologic activity. Moreover, they were likely highly correlated.

There were many significant problems with the study, with several acknowledged by the authors (St. John, Kobus et al. 2004), including issues with construct and external validity, the statistical methods, significant missing data, and a well-known problem inherent to psychophysiologic research,  noisy data.  Given the number and severity of confounds, the results from this study can be at best considered preliminary and not unequivocal scientific evidence that the reported significant cognitive state gauges can effectively detect change in cognitive activity in a complex human supervisory control task.

A second set of four experiments was conducted in support of the second phase of AUGCOG, which was designed to manipulate an operator’s cognitive state as a result of the near real-time psychophysiologic measurements (Dorneich, Ververs et al. 2005). The experiments took place in a Military Operations in Urban Terrain (MOUT) video-game environment, either at a desktop setting or in a motion capture laboratory. In addition to the primary task of navigation through the MOUT, participants identified friend from foes and monitored and responded to communications. A communications scheduler, part of the Honeywell Joint Human-Automation Augmented Cognition System¹, determined operator workload via a cognitive state profile (CSP) and prioritized incoming messages accordingly. The CSP was an amalgamation of signals from cardiac inter-beat-intervals, heart rate, pupil diameter and microvolt cardiac QRS waveform root mean square amplitude, EEG p300 signal, and EEG power at the frontal (FCZ) and central midline (CPZ) sites.

As in the first phase study, there were very low numbers of participants in each of the studies (16 or less), and construct validity and statistical models were questionable with significant experimental confounds. There is no clear published account of how the neurologic and physiologic variables were combined to form the CSP, so the ability to independently replicate these experiments is difficult, if not impossible. Moreover, the results should be considered very preliminary and viewed with caution due to claims such as 100% improvement in message comprehension, 125% improvement in situation awareness, 150% increase in working memory, and increased survivability of over 350%. In addition, the authors claim, admittedly with anecdotal evidence, that their cognitive state gauges can indicate operator inability to comprehend a message (Dorneich, Ververs et al. 2005).

These results are reminiscent of the famous Kuhn quote, "The man who is striving to solve a problem defined by existing knowledge and technique is not just looking around. He knows what he wants to achieve, and he designs his instruments and directs his thoughts accordingly (Kuhn 1970)." The results from these studies, funded by DARPA and the U.S. Army AUGCOG programs, are likely not an indication of truly revolutionary neuroscience technology breakthroughs, but rather an indication that either the experiments were poorly designed or they were designed to guarantee success². More importantly, because the focus in the experiments was on generating measurable outcomes in a very short period of time to prove program success, most of the technical data on the performance of the actual sensors and signal processing and combination algorithms was not discussed, which could have been useful to other related scientific endeavors.

Even if one assumes that the results from the experiments are valid, unfortunately no follow-on studies have been reported that show how the successful cognitive state gauges could or would be combined in terms of hardware. The engineering obstacles in combining an EEG, fNIR, and eye-tracking devices are substantial, and unless dramatic leaps are made in the miniaturization of these technologies and improved signal processing algorithms in the near-future, the realization of a single headset that can perform all, or even a combination of these technologies is more than ten years in the future. In addition, more near-term engineering problems have not been addressed in the open literature such as how to measure EEG signals in a dynamic, noisy environment, which is critical to the success of the US Army deployment of these technologies. This is also a problem for eye tracking devices, which currently require sophisticated head tracking devices in addition to the eye tracking devices, so encapsulating this technology into an unobtrusive device that can be worn in the field is also more than 10 years away.

In addition to the hardware limitations for the use of neural and physiologic technologies in a future operational field setting, the ability of this software/hardware suite make these predictions reliably in real-time in a highly dynamic, stochastic setting typical of command and control environments is also not at the levels required in realistic settings. The experiments conducted under the AUGCOG program were all highly artificial, and the communications scheduler made changes in information presentation based on gross differences in perceived cognitive state.  Actual battlefield conditions and the amount of and degrees of freedom of information states will mean that much more precision will be needed. This means that not only will the sensors and signal processing algorithms have to improve substantially, but also significant advances are needed in decision-theoretic modeling. In addition, these models will have to be able to accommodate a significant range of individual variability

One final obstacle in achieving the AUGCOG goal of enhancing operator performance through psychophysiologic sensing and automation-based reasoning is determining that, even if the system could change information streams and volume of incoming information, how does the systems know this change was correct, or even helpful? This kind of predictive system assumes that it can determine an optimal cognitive load for an individual in dynamic, highly uncertain systems.  The problem with this assumption is that there is no true optimality in command and control settings. Before any kind of predictive system could be deployed that controlled inputs to a military operator, this system would have to guarantee that it at least did no harm, and it is questionable that given the current limitations in neuro and computer science, that this is feasible.

 

The U.S. Army AUGCOG Program

When the DARPA AUGCOG program formally ended, the U.S. Army adapted elements of the AUGCOG program under the direction of Henry Girolamo at the U.S. Army Natick Soldier Research, Development and Engineering Center. The original goal of the Army’s AUGCOG efforts was to incorporate the DARPA AUGCOG technology into Future Force Warrior by 2007 (Public Affairs Office 2005). This goal appears to have been revised, and currently the primary focus has changed from operational to training applications. Moreover, the focus has been narrowed to the use of electroencephalograph (EEG) and electrocardiograph (ECG) sensors, instead of the full array described earlier under the DARPA effort (Boland 2008).

Under the Army’s AUGCOG program, another experiment was conducted that was an extension of the previous efforts. Under the direction of the same Honeywell team that performed the set of experiments reported above, this third experiment focused on developing a mobile cognitive state classification experimental test bed, and testing it in a dismounted soldier field setting using the previously-discussed communications scheduler (Dorneich, Whitlow et al. 2007). The authors developed an EEG headset connected to a laptop computer worn in a backpack by a subject. This laptop supported the signal processing algorithms, the communications scheduler, and other experimental testing elements.

Eight subjects with no military experience completed a one hour navigation and communication task in this experiment with a handheld radio and a personal digital assistant, and a 35 lb backpack. The authors reported that with the communications scheduler prioritizing messages based on whether the subjects were in a low or high task load, mission performance metrics improved from 68% in the unmitigated condition to 96% with cognitive state mitigation.

As with the other studies, there were significant confounds that limit the validity of these results. The authors reported problems with movement-induced signal noise, as well as significant loss of data that reduced the subject pool to 4 for a portion of the experiment, and there were many experimental confounds. In addition, as the authors discuss, their classification approach was extremely limited in state estimation (i.e., costs of actions were not considered), and their approach depended on relatively short temporal gaps between training and testing. This is a critical operational problem both in terms of time constraints, (i.e., will soldiers have to go through extended training before every mission with this device?)/ More importantly, actual combat never adheres to a carefully-planned script and it is highly questionable whether the a priori classification training will have any semblance to actual world events, thus invalidating the usefulness of such a device. And as mentioned previously, the bigger problem is not that this device might fail to provide accurate interventions, but that it could provide detrimental interventions.

 

Summary

The original goal of the DARPA Augmented Cognition program, to enhance human information processing capabilities through the design of adaptive interfaces via cognitive state estimation, is even more relevant in 2008 with the DoD’s move towards network-centric warfare. However, while the researchers involved with the AUGCOG milestones have made progress in terms of both hardware and software advances, the results are preliminary, and not suggestive that the desired results are achievable in the near-term.

Despite the stated objectives that AUGCOG technologies will be operational in ten years, it is likely that the horizon is much longer. In additions to the limitations discussed earlier, one major hurdle in the realization of any AUGCOG implementation will be the development of a wireless EEG device that is unobtrusive, does not require the use of conducting gel (otherwise known as a dry EEG), and able to process on-board signals, all while soldiers are in motion, under often hostile environmental conditions. While some advancements have been made in wireless EEGs, as well as dry EEGs (see the Leveraging Technologies section), the signals from these devices are substantially weaker than the more traditional EEG devices. Moreover, their ability to detect cognitive states to be used in predictive algorithms in dynamic, uncertain environments is a question of basic research.