Out of the Glass Cockpit: Teaching Legal Analysis in Legal Research

In his article The Great Forgetting,^[1] Nicholas Carr tells the tragic story of an airline pilot who didn’t know how to handle an aerodynamic stall that occurred on a commercial flight. He’d become so accustomed to relying on the sophisticated software that flew the plane that he forgot how to respond to the warning signs. He did exactly the wrong thing, crashing the plane and killing 50 people. This situation, and its tragic outcome, is not unique. The problem is that airline pilots increasingly spend their time typing in data and monitoring computer screens, rather than actually flying planes. The solution is for pilots to get out of the “glass cockpit”—where they are surrounded by screens that efficiently and seamlessly do the work of flying—and practice for themselves the essential skills of flying.

Similarly, our students spend much of their research time in a glass cockpit. The current wave of sophisticated search engines—and each wave that will follow—can turn students into passive computer operators likely to crash when faced with a new, complex issue. Our teaching methods need to get students out of the glass cockpit so that they actually think while doing legal research. Of course, to do so, we have to get out of the glass cockpit ourselves.

Computer Complacency and Bias

Carr’s article explains that working with computers can lead to two cognitive weaknesses: complacency and bias. Complacency lulls us into thinking the computer will work just fine, even if we aren’t fully engaged. Bias means we place too much trust in the information computers provide, sometimes ignoring competing information or assuming that the computer’s information is complete. “When a computer provides incorrect or insufficient data, we remain oblivious to the error.”^[2]

Online legal research can make students prone to complacency and bias. Online research can breed complacency by offering a false sense of security. A high-powered algorithm will return results for almost any search, regardless of how thoughtlessly the researcher types terms into a search bar. Similarly, online research can create computer bias. Students may believe that the powerful search engines will find all—or enough—relevant authorities simply because of the volume of results. If one search from a universal search bar with a few key words produced 894 cases, 231 statutory references, and over 10,000 secondary sources, surely the search was a success and the necessary documents are included! Narrowing these results with a few filters could bring the results down to 37, which any student can easily scan. But the original search might have been so poorly conceived that the resulting list of authorities is incomplete or off-topic.

To counter both complacency and bias, we must teach students how to think throughout the research process, rather than functioning on autopilot. This thinking includes analyzing research techniques, developing searches, evaluating results lists, and scrutinizing individual documents. In other words, research is an analytical process, not an automatic system of document retrieval.

Teaching Students to Think Analytically While Researching

Initially, computers made me complacent. I spent less classroom time teaching research, believing that my computer savvy students would find the sophisticated search engines intuitive. I quit giving quizzes, overwhelmed by the myriad possible answers to straightforward questions. In my complacency, I lost the intensity I once brought to teaching research, especially the focus on thinking and analysis. The following techniques are getting me out of the glass cockpit and refocusing my teaching.

1. Start with civics. Most students need a refresher in court systems, sources of law, and other essentials. If they don’t appreciate the authoritative difference between an enacted statute and a proposed regulation, or who produces those documents, their research is likely to crash.

2. Require pre-class preparation. Students may be inclined to skim over assigned reading about research for several reasons: some might assume research will be intuitive; some might find the details of research tedious or overwhelming; and some may simply be stunned by the first-year workload. “Reading questions” or online quizzes can ensure that students engage actively with each paragraph in their research textbooks.

3. Think first. When online research was outrageously expensive, we encouraged students to construct possible searches and consider various research paths before signing on. Those are still good first steps. We should design in-class research exercises accordingly: no laptops opened until we have thought about what we looking for, where to start, and why.

4. Teach filtering as an analytical step. Reducing from 621 results to 24 is useful only if the best authorities survive the cut. We can teach students to filter results analytically through a class exercise. Together, skim the results of a search, analyze their potential usefulness, and discuss possible ways to filter out unhelpful documents. A student should be able to explain the thought process leading to each click.

5. Vary research approaches. Complacent students may rely on just one approach for every research problem. Working in small groups can counter complacency by encouraging students to think through a variety of approaches. Each group can create a checklist of approaches—throwing words into the universal search bar, conducting a terms and connectors search in a specific content area (e.g., treatises), starting with topical subject areas, using headnote digests, researching news stories—and discuss when each approach might be useful. Later, students’ research logs should include a variety of these approaches.

6. Engage one-on-one. Online research provides endless pathways to the same core authorities. A teacher with a group of even 20 students will be unable to monitor and refine 20 paths simultaneously. Thus, after classroom work, students need one-on-one guidance. Reference librarians familiar with the research project are the best sources for this guidance, working with very small groups in sessions (no more than five students) where students experiment with different research approaches. Amenable vendor representatives and properly trained upper-level students can also engage novice researchers.

7. Repeat. Just as airline pilots need to maintain their flying skills with repeated simulation training sessions, students need to repeat research steps to master them. The first-year research course needs to include multiple encounters with each research technique, resource, and strategy. Ideally, each student should take additional courses requiring advanced research in each semester, reemphasizing research and analytical skills. Better yet, students should have a clinical or externship experience where they work with real clients and research results matter.

By teaching students to think throughout the research process, we can all escape the glass cockpit.