Phishing IQ Tests Measure Fear, Not Ability

Abstract

We argue that phishing IQ tests fail to measure susceptibility to phishing attacks. We conducted a study where 40 subjects were asked to answer a selection of questions from existing phishing IQ tests in which we varied the portion (from 25\% to 100\%) of the questions that corresponded to phishing webpages. We did not find any correlation between the {\em actual} number of phishing webpages and the number of webpages that the subjects indicated were phishing webpages. Therefore, the tests did not measure the ability of the subjects. To further confirm this, we exposed all the subjects to existing phishing education after they had taken the test, after which each subject was asked to take a second phishing test, with the same design as the first one, but with different questions. The number of sites that were indicated as being phishing webpages in the second test was, again, independent of the {\em actual} number of phishing webpages in the test. However, a substantially larger portion of stimuli was indicated as being phishing webpages in the second test, suggesting that the only measurable effect of the phishing education (from the point of view of the phishing IQ test) was an increased concern---not an increased ability.