Psychology and Security Resource Page

Ross Anderson

A fascinating dialogue is developing between psychologists and security engineers. At the macro scale, societal overreactions to terrorism are founded on the misperception of risk and undertainty, which has deep psychological roots. At the micro scale, more and more crimes involve deception; as security engineering gets better, it's easier to mislead people than to hack computers or hack through walls. Many systems also fail because of usability problems: the designers have different mental models of threats and protection mechanisms from users. Wrong assumptions about users can lead systems to discriminate against women, the less educated and the elderly. And misperceptions cause security markets to fail: many users buy snake oil, while others distrust quite serviceable mechanisms. Security is both a feeling and a reality, and they're different. The gap gets ever wider, and ever more important.

At a deeper level, the psychology of security touches on fundamental scientific and philosophical problems. The `Machiavellian Brain' hypothesis states that we evolved high intelligence not to make better tools, but to use other monkeys better as tools: primates who were better at deception, or at detecting deception in others, left more descendants. Conflict is also deeply tied up with social psychology and anthropology, while evolutionary explanations for the human religious impulse involve both trust and conflict. The dialogue between researchers in security and in psychology has thus been widening, bringing in people from usability engineering, protocol design, privacy, and policy on the one hand, and from social psychology, evolutionary biology, and behavioral economics on the other. We believe that this new discipline will increasingly become one of the active contact points between computing and psychology - an exchange that has hugely benefited both disciplines for over a generation.

This page provides links to a number of key papers, workshops, the home pages of active researchers, relevant books, and other resources. Complementary pages include my security economics resource page and Alessandro Acquisti's privacy economics page.

The most relevant regular event is The Security and Human Behaviour workshop: see the papers, liveblog and audio for 2009; and the papers, liveblog and audio for the first meeting in 2008. See also the Symposium On Usable Privacy and Security which has been established since 2005 and is the focus for security usability work.

Introductory Papers

• In The Psychology of Security, Bruce Schneier gives a layman's introduction to how heuristics and biases affect the way we deal with risk and uncertainty. Our analysis of security problems can thus draw on the large literature on behavioral economics with its insights on risk aversion, the availability heuristic, mental accounting, and discounting. (See also Bruce's other essays on psychology and security.)
• For a more detailed treatment, see the chapter on security and psychology from my book Security Engineering. The book also has a chapter on terror which discusses the social-scale effects of security psychology, and there is also a paper that discusses the interacation between security, economics and psychology, Information Security Economics - and Beyond (you can download a video of this survey talk and the slides).
• Economics, Psychology and Sociology of Security by Andrew Odlyzko discusses how cultural factors undermine the formal assumptions underlying many security systems, and gives some insights from evolutionary psychology; for example, we have specialised neural circuits to detect cheating in social situations.
• Privacy in Electronic Commerce and the Economics of Immediate Gratification is one of the seminal papers: in 2004 its author, Alessandro Acquisti, first applied behavioral-economics ideas such as risk aversion, optimism bias and hyperbolic discounting to understand privacy (and security) behaviour. It started to explain the privacy paradox by showing that we cannot expect people to behave fully rationally when confronted with typical privacy choices. (For more on this topic, see here.)
• Human Behaviour and Deception Detection, by Mark Frank and colleagues, surveys the state of the art in deception detection. Most people are poor at detecting lies as we get little practice; a small minority are gifted at it. (For more on this topic, see here.)
• If only gay sex caused global warming by Daniel Gilbert is a psychologist's analysis of why we overreact to threats associated with moral norms, such as terrorism, while discounting morally-neutral threats like environmental damage. (For more on this topic, see here.)
• Why Johnny can't encrypt: A usability evaluation of PGP 5.0 was the seminal paper that kicked off the study of security usability. Alma Whitten and Doug Tygar showed that the then most popular encryption product was essentially unusable, even by computer-savvy people: essentially everyone made significant mistakes when driving it. (For more on this topic, see here.)

Deception

• The Psychology of Scams - Provoking and Committing Errors of Judgment, by Stephen Lea and colleagues, is an encyclopaedic study of how people fall for online and other scams, conducted for the UK Office of Fair Trading. The authors give a thorough literature review, describe a large number of scams, and then present four original studies of their own. Scams involve many of the other techniques used by legitimate marketers; people who are poor at regulating their emotions, and perhaps are socially isolated, may be disproportionately vulnerable, but some of them can be educated to resist scammers.
• Understanding scam victims: seven principles for systems security by Frank Stajano and Paul Wilson examines a variety of scams and cons that were documented in the BBC TV programme "The Real Hustle", extracts the principles on which the scams were based, and applies them to system security.
• Jeff Hancock's On Lying and Being Lied To: A Linguistic Analysis of Deception in Computer-Mediated Communication finds a number of statistical differences between lies and truths told in text-based communication. His Separating Fact From Fiction: An Examination of Deceptive Self-Presentation in Online Dating Profiles shows that men lied more about their height, and women lied more about their weight; the latter class of fibs seemed intentional rather than self-deception, and were balanced against social constraints such as the antitipcation of future interaction.
• Fred Cohen has a summary of deception research; this explores psychological phemonena relevant to informatoon system applications from fraud to warfare, and compares deception at different levels between humans, computers and organisations. This may give a framework for thinking about deceptions that combine human, computer and organisational aspects.
• Less than human: self-deception in the imagining of others by David Livingstone Smith makes the point that self-deception is important in war; many people find it difficult to kill unless they can persuade themselves that the enemy is not fully human but somehow a predator or vermin.
• Social Phishing, by Tom Jagatic and others, reports how the percentage of students who responded to a test phishing email was increased from 16% to 72% by including relevant social information about the target (for example by making the email appear to come from a friend of the target, identified via a social networking site).
• Fred Schauer and Dick Zeckhauser's Paltering explores untruths that stop somewhat short of a full lie – the many half-truths, fudges, exaggerations and distortions that oil social intercourse and are the very stuff of politics.

• The Memorability and Security of Passwords - Some Empirical Results by Jeff Yan and others kicked off the empirical study of what sort of advice we should give users to stop them choosing weak passwords.
• Users Are Not The Enemy, by Anne Adams and Angela Sasse, is another classic; it explores why users break security rules, the costs of poor usability, and what sort of system, contextual, and work-prractice changes may help: for example, they advocate shared passwords for shared work.
• A Series of Book Chapters by Peter Gutmann may constitute the most authoritative guide to actual security usability faults in popular systems. They first give a lot of examples of good and (especially) bad practice, then goes on to discuss underlying cognitive biases that maake usability engineering hard, then to present some guidelines for designing security interfaces, then how warnings and other dialogues can be made meaningful to the user, and finally how security usability can be tested.
• It's no secret: Measuring the security and reliability of authentication via `secret' questions, by Stuart Schechter, Bernheim Brush and Serge Edelman, studies the personal questions commonly used as fallback authentication for users who forget their passwords. This methodology doesn't work very well at all; participants forget a fifth of their own answers within six months, while their answers can be guessed by a sixth of their acquaintances with whom they were unwilling to share passwords.
• Jean Camp's Experimental Evaluation of Expert and Non-expert Computer Users' Mental Models of Security Risks explores empirically the differences between the mental models of information security risk used by experts and non-experts.
• A Framework for Reasoning About the Human in the Loop by Lorrie Cranor proposes a framework to help designers understand human limitations, so that people can be used as dependable components in systems. It is based on research on warnings and essentially walks the designer through issues such as attention, knowledge, comprehension and beliefs, highlighting those areas that may need testing before the system is designed or training afterwards.
• The Emperor's New Security Indicators, by Stuart Schechter, Rachna Dhamija, Andy Ozment and Ian Fischer, presents an empirical study of the security indicators on which banks and e-commerce websites rely to inform customers that all's well (or not). These basically don't work well or in some cases at all.
• Toward a broader view of security protocols, by Matt Blaze, advocates treating security protocols as phenomena of human interaction and gives a number of examples of protocols that have evolved over time to give the participants reciprocal benefits.

Social Attitudes to Risk

• Taken Out of Context - American Teen Sociality in Networked Publics, which is danah boyd's thesis, is an ethnographic study of how American teenagers actually deal with risk and information security. Passwords are often shared as a token of intimacy when young people date; some lies (for example, about age) are acceptable while others (about birthdays) are not. The overall picture is of real usage being very different from websites' contract terms, and from the mental models used by security engineers.
• The Myth of the Superuser: Fear Risk and Harm Online by Paul Ohm illuminates the anthropology of computer security fearmongering. The media consistently exaggerate the capabilities of computer experts in order to create bogeymen; the resulting myth is nurtured by many who find it convenient, but it imposes costs from poor systems design to over-broad laws.
• Reacting to Terrorism: Probabilities, Consequences, and the Persistence of Fear by John Mueller studies ways of measuring the actual (rather limited) harm that terrorism does, versus the exaggerates fears that it provokes. Terrorism does not cause panic directly (actual attacks lead to mutual aid); the problem is its secondary effect in causing risk-averse behaviour such as driving instead of flying; stress-related illnesses; and a political climate in which temperate behaviour is harder. Dramatic first impressions do make a big difference.
• In Cars, Cholera and Cows: The Management of Risk and Uncertainty John Adams explores why organisations tend to be more risk-averse than rational economic considerations would dictate. One of the mechanisms is adverse selection: the people who end up in risk management jobs tend to be more risk-averse than average.

• Privacy, Economics and Price Discrimination describes the privacy paradox: Why is privacy being eroded so rapidly, despite many people saying they care about it? Andrew Odlyzko's analysis puts much of the blame on differential pricing. Technology is increasing both the incentives and the opportunities for this.
• The Best of Strangers: Context Dependent Willingness to Divulge Personal Information by Leslie John, Alessandro Acquisti and George Loewenstein describes experiments that show how context-dependent our privacy preferences are. Students were asked to divulge sensitive personal information in neutral setting, or in one that provided privacy reassurances: they divulged less there as privacy had become salient. Others were asked via a frivolous website and divulged much more.
• What Can Behavioral Economics Teach Us About Privacy?, by Alessandro Acquisti and Jens Grossklags, explores how research at the boundary between psychology and economics can cast light on the privacy paradox. Framing, self-serving bias and other effects could all contribute. Their paper When 25 Cents is too much: An Experiment on Willingness-To-Sell and Willingness-To-Protect Personal Information provides a nice exposition of the difference between willingness to pay and willingness to accept, with the latter usually being much higher, and discuss how this can also be applied.
• Who Signed Up for the Do-Not-Call List?, by Hal Varian, Fredrik Wallenberg and Glenn Woroch, analyses the FCC's telephone-sales blacklist by district. Privacy means different things to different population groups, but this raises further questions. For example, educated people are more likely to sign up, as one would expect: but is that because rich households get more calls, because they value their time more, or because they understand the risks better? In Financial Privacy for Free?, Alessandro Acquisti and Bin Zhang apply a similar analysis to credit reporting.
• Social Interaction, Observational Learning, and Privacy: the "Do Not Call" Registry offers further analysis of who subscribed to the FCC's telephone-sales blacklist. Its authors Khim Yong Goh, Kai-Lung Hui and Ivan Png found that the clustering of people who opted out was more due to social interaction than to local news media; that the extent of observational learning decreased with social heterogeneity; and that people who opted out were mostly interested in escaping telemarketers.
• The Value of Online Information Privacy: An Empirical Investigation by Il-Horn Hann, Kai-Lung Hui, Tom Lee and Ivan Png reports testing privacy preferences of grad students in both the USA and Singapore. Most students were concerned to prevent unauthorised secondary use of shopping data ("privacy guardians"); there were minorities of "privacy sellers" and "convenience seekers" willing to trade privacy for cash or convenience respectively.
• Privacy, Trust and Self-Disclosure Online, by Adam Joinson and others, explores the link between stated and revealed prrivacy preferences. They performed two studies of the dispositional and situational aspects of online trust and privacy in an attempt to establish whether trust and privacy are substitutes. It turned out that user actions were governed more by situational than dispositional factors.
• On the Economics of Anonymity by Alessandro Acquisti, Roger Dingledine and Paul Syverson studies why anonymity systems are hard to sell, and points out some of their novel aspects. For example, honest players want some level of free-riding, in order to provide cover traffic. So equilibria can also be novel, and the ways in which they break down can be complex. We also have to consider a wider range of principals - dishonest, lazy, strategic, sensitive, and myopic - than in most of the markets that economists try to model.
• The Effect of Online Privacy Information on Purchasing Behavior: An Experimental Study by Janice Tsai, Serge Egelman, Lorrie Cranor and Alessandro Acquisti shows that by making information about website privacy policies more accessible and salient it is possible to get shoppers to pay more attention to it and even to pay a premium for privacy.
• Conformity or Diversity: Social Implications of Transparency in Personal Data Processing by Rainer Boehme studies whether making information widely available about the bases on which decisions are taken about individuals will lead to more conformity (because, in the absence of information asymmetries and strategic interaction with others, the optimal behaviour becomes mainstream) or diversity (as in the absence of transparency, individuals are herded together by uncertainty and fear). He presents a model of how preferences and signaling behaviour might interact.
• Why we can't be bothered to read privacy policies - models of privacy economics as a lemons market, by Tony Vila, Rachel Greenstadt, and David Molnar, examines why many consumers fail to think of future price discrimination when giving information to merchants.

Miscellaneous Papers

• The Social Function of Intellect by Nick Humphrey was the paper that kicked off the "Machiavellian Brain hypothesis" - the idea that we became smarter than other monkeys not so we could make better tools, but so we could use other monkeys as tools. As group sizes increased, so did the complexity of the "social chess" that we play. Later versions of this theory can be summarised as: monekeys who were better at deception, or detecting deception in others, left more descendants.
• Reason and Rationality, by philosophers Richard Samuels, Stephen Stich and Luc Faucher, discusses the tensions between the heuristics-and-biases tradition and modern evolutionary psychology, with the former being more pessimistic and the latter being more optimistic about the rationality of the average man or woman.
• The nature of collective resilience: Survivor reactions to the 2005 London bombings, by John Drury, Chris Cocking and Steve Reicher, analyses bomb survivors'accounts, and finds more reports of calm than panic, with the latter tending to arise in reports by commentators who were not there, or in victims' reports of feelings of fear, rather than in witnesses' descriptions of crowd behaviour. Many of the victims felt a sense of solidarity and helped each other; the authors hypothesise that a sense of a common fate drove people to be more altruistic than normal.
• Intra-group Regulation of Violence: Bystanders and the (De)-escalation of Violence, by Mark Levene, Rachel Best and Paul Taylor, reports a survey of 42 incidents of night-time violence, taken from UK CCTV camera records. Bystanders calm down fights more often than egging on the combatants; and, contrary to previous beliefs, this bystander effect actually increases as thr group size does.
• Paradigm Shifts in Security Strategy by Dominic Johnson and Elizabeth Madin explores why society is predisposed to accept the status quo until something goes wrong. Reasons range from the discounting of hypothetical dangers through status-quo bias, organisational inertia and the entrenched policy preferences of dominant leaders to the lack of incentive in electoral politics for disruptive precautions against unlikely threats. The upshot is that radical changes often require a disaster. But democratic states with innovative cultures are more likely to adapt than weak states are, and they might be made more adaptable still by measures such as term limits, campaign finance limits, freedom of information, and ensuring that decision makers talk regularly to front-line staff.
• From Spaces to Places: Emerging Contexts in Mobile Privacy by Clara Mancini and others studied the privacy behaviour of mobile technology users; the paper reports a number of different types of boundaries, based on such factors as personal policy, inside knowledge, etiquette, proximity and aggregation, that people used in different ways to regulate interaction.
• The Compliance Budget: Managing Security Behaviour in Organisations, by Adam Beautement, Angela Sasse and Mike Wonham, argues on the basis of a series of interviews with corporate security manaagers that both people and groups within organisations are only prepared to put up with so much regulation, and in consequence managers should aim to make the best possible use of this "compliance budget" rather than just issuing one instruction after another.
• Using Social Psychology to Implement Security Policies, by Mich Kabay, discusses how to get people to pay attention to security policies and the importance of the social schemata by which people frame reality and judge behaviour. These are particularly important when the desired behaviour violates social norms such as holding doors open for people. A number of strategies for changing expectations and norms are discussed.
• Finally, if you're interested in the dark side, The Manipulation of Human Behavior by Albert Biderman and Herb Zimmer reports experiments on interrogation carried out after the Korean War with US Government funding. It's also known as the Torturer's Bible, and describes the relative effectiveness of sensory deprivation, drugs, hypnosis, social pressure and so on when interrogating and brainwashing prisoners.

The Security and Human Behaviour workshop brings security engineers together with psychologists, behavioral economists and others. See the papers, my liveblog (and Bruce's) for 2009; and the audio. There's also the papers, liveblog and audio for the first meeting in 2008.

The Symposium On Usable Privacy and Security (SOUPS) is the workshop for research on the usability of security systems. It has been running since 2005; here are the programs (with links to the papers) for 2005, 2006, 2007, 2008, and 2009, as well as the call for 2010.

The Workshop on the Economics of Information Security (WEIS) has some relevant papers; its focus is the interface between security and economics. Here are the programs (with links to the papers) for 2002, 2003, 2004, 2005, 2006, 2007, 2008 and 2009. WEIS 2010 will be held at Harvard.

Community - Home Pages of People Interested in Security Psychology

• Alessando Acquisiti
• Andrew Adams
• John Adams
• Ross Anderson
• Yochai Benkler
• Matt Blaze
• Rainer Boehme
• Joe Bonneau
• danah boyd
• Bob Briscoe
• Bill Burns
• Jean Camp
• Luke Church
• Richard Clayton
• Chris Cocking
• Fred Cohen
• Lorrie Cranor
• George Danezis
• Roger Dingledine
• Julie Downs
• Ben Edelman
• Mark Frank
• Allan Friedman
• Dan Gardner
• Carrie Gates
• Anindya Ghose
• Rachel Greenstadt
• Jeff Hancock
• Kjell Hausken
• Geoffrey Heal
• Matt Hottell
• Markus Jakobsson
• Adam Joinson
• Richard John
• Dominic Johnson
• Eric Johnson
• Christine Jolls
• Mike Just
• Mich Kabay
• Carl Landwehr
• Mark Levine
• David Livingstone Smith
• Jeff MacKie-Mason
• David Mandel
• Nick Mathewson
• Tyler Moore
• John Mueller
• Shishir Nagaraja
• Helen Nissenbaum
• Peter Neumann
• Bashar Nuseibeh
• Andrew Odlyzko
• Andrew Patrick
• James Pita
• Uday Rajan
• Rob Reeder
• Mike Roe
• Angela Sasse
• Bruce Schneier
• Stuart Schechter
• Adam Shostack
• Diana Smetters
• Frank Stajano
• Mark Stewart
• Terence Taylor
• Doug Tygar
• Hal Varian
• Alma Whitten
• Jeff Yan

Books

• My book Security Engineering might be a good introduction for social scientists to secure systems engineering. It covers not just technologies such as crypto and firewalls, but a number of specific applications from banking to burglar alarms. It brings out the fact that most systems don't fail because the mechanisms are weak, but because they're used wrong.
• Beyond Fear by Bruce Schneier discusses how we're encouraged to think about security by law enforcement agencies, businesses and governments, and how we can become more discerning consumers and citizens by understanding the agendas of the various players and the means they use to influence us.
• I've described The Art of Deception by Kevin Mitnick as "the most disturbing security book ever". Mitnick was jailed after a hacking spree based mostly on social engineering: he became a master at telling lies on the telephone that would cause people to give him passwords or otherwise open systems to attack. This book tells how he did it.
• The Human Contibution by James Reason discusses human factors in the design of safety-critical systems. The predictable varieties of error are rooted in the very nature of cognition. Safety engineers have been studying psychology for much longer than security engineers; we have a lot to to learn from them! His earlier book, Human Error, is also a classsic, having been a key publication in the emergence of safety usability as a discipline.
• Darwinian Security by Rafe Sagarin and Terry Taylor arose out of a research project that brought together life scientists from many disciplines to review the attack and defence strategies used by living organisms in the struggle for survival. (See also the book website and video.)
• John Mueller, Overblown: How Politicians and the Terrorism Industry Inflate National Security Threats, and Why We Believe Them analyses the psychological and social factors exploited by the fear industry, and also explores its political history. It turns out that presidents who downplayed terrorist attacks (such as Eisenhower, Kennedy, Nixon, Reagan and the elder Bush) did better than the presidents who played them up (such as Carter and the younger Bush).
• Stanley Milgram's Obedience to Authority: An Experimental View describes his classic experiment in which he found that most subjects were prepared to administer painful and dangerous electric shocks to a "student" when ordered to by an experimenter in a position of ostensible authority.
• Philip Zimbardo's The Lucifer Effect: Understanding How Good People Turn Evil describes the infamour Stanford prisoner experiment which had to be stopped after students playing the role of pricon guards started brutalising students playing the role of inmates. You didn't need abuse of authority to turn good people bad: social roles could be enough.
• Security and Usability: Designing Secure Systems that People Can Use, edited by Lorrie Cranor and Simson Garfinkel, is a compendium of the early research papers on security usability. With over 750 pages and 30 contributors, it is basic reading for any serious researcher.
• Richard Byrne and Andy Whiten edited two major volumes of papers on the Machiavellian brain hypothesis: Machiavellian Intelligence – Social Expertise and the Evolution of Intellect in Monkeys, Apes and Humans and Machiavellian Intelligence II – Extensions and Evaluations which cover the development of the theory that we evolved our intelligence in order to become better at deception, and at detecting deception in others.
• Steven LeBlanc and Katherine Register's Constant Battles: The Myth of the Peaceful, Noble Savage debuked the idea that we used to live in harmony before technology came along. In fact, at all times in prehistory and in history until recently, perhaps a quarter of men and boys dies as a result of homicide. Warfare was the default means of population control.
• Why we Lie by David Livingstone Smith provides a detailed discussion of a simple, profound truth: to be good at deceiving others, we have to deceive ourselves. Self-deception is at the heart of many social phenomena.
• Risk by John Adams is the classic study of why people and organisations are sometimes more risk-averse than would seem rational, and sometimes more risk-loving. For example, mandatory seat-belt laws did not reduce road traffic casualties overall, but merely shifted them from vehicle occupants to pedestrians and cyclists. Adams explains this by a `risk thermostat': people compensate for an increased feeling of safety by driving faster. In general, behaviour is governed by the probable costs and benefits of possible actions as perceived through filters formed from experience and culture.
• Heuristics and Biases: The Psychology of Intuitive Judgment, edited by Thomas Gilovich, Dale Griffin and Danny Kahneman, is a collection of great papers on behavioral economics – a subject that was founded by the third editor and that lies at the boundary between psychology and economics. It studies how the various heuristics and biases that underlie intuitive thinking make us prone to irrational behaviour and various kinds of decision errors. It updates a 1982 reference, Judgment under Uncertainty: Heuristics and Biases which brought together the seminal papers in the field.
• Gerd Gigerenzer's Gut Feelings: The Intelligence of the Unconscious describes how simple rules of thumb underlie much of our subconscious mental processing and account for decisions that we make instinctively or intuitively. These rules are surprisingly robust in a wide range of applications, as they are founded in evolutionary pressures on organisms to choose adaaptive courses of action in the face of incomplete information.
• Robert Cialdini's Influence: Science and Practice is the standard reference on the psychological science behind marketing. Cialdini worked for several years undercover as a salesman, fundraiser and advertiser; he distilled the experience of this fieldwork into a study of the techniques of persuasion including reciprocation, consistency, social proof, liking, authority, and scarcity. The book is basically a guide to pushing people's buttons.

Other Resources

Here are some suggestions for further reading:
• The Vmyths site is devoted to debunking computer security hysteria
• Snake-oil Security Claims: the Systematic Misrepresentation of Product Security in the E-commerce Arena, John Michener, Steven Mohan, James Astrachan and David Hale
• An article in Wired describes the low ratio of vulnerabilities to exploits and the attention-seeking nature of many vulnerability reports prior to about 2004
• Psychology - a Precious Security Tool talks about social engineering and on what can be done about it
• A Simple Model of Fads and Cascading Failures, Duncan Watts
• The Cyberwar Debate: Perception and Politics in US Critical Infrastructure Protection, Ralf Bendrath
• Immediacy Bias in Emotion Perception: Current Emotions Seem More Intense Than Previous Emotions by Leaf van Boven and others discusses one aspect of people's response to terrorism
• The Economics of Airline Safety and Security by Robert Hahn
• An Economic Perspective on Transnational Terrorism by Todd Sandler
• Reconsidering Public Reactions to Terrorism by Jennifer Merolla and Elizabeth Zechmeister

---