Today’s Wall Street Journal story “On Web, Children Face Intensive Tracking“, is interesting, and if you go to the website and click around you find it has a fair bit of material that fills in gaps. There’s a decent video explaining cookies, for those who don’t already know how they work.
I went to the web version of this story because the paper version said this:
(Full methodology, as well as previous privacy investigations in this series, at wsj.com/WTK.)
I wanted to know about the “full methodology” for the study, because the article has several precise figures for the number of “tracking tools” placed on the Journal’s test computer. Precise figures suggest a methodology rigorous enough to produce reliable, precise figures:
…Y8 installed 69 tracking files…
…On average, the eight installed 81 tracking tools, close to the 82 average for all 50 sites…
…a games site called Shockwave.com, installed 146; another game-and-video site, nick.com, installed 92…
…the math-games site coolmath4kids.com installed 60 on a test computer…
…Weeworld.com installed 144 tracking tools in the Journal’s test…
Here’s the Critical Thinking 101 question for the day: What do you need to know about the Full Methodology of the research before you can gauge the importance of all those exact numbers?
Here’s what I think are basic, pass-fail, questions the Full Methodology information must answer:
1. What did they count?
In this case, it’s not difficult. We know what cookies are, since they’ve been around for a long time, since Windows 95 days. Flash cookies and beacons, we’ll guess, are different from traditional flat text cookies, but still discrete, countable things. In other words, if you looked at a bunch of them together there wouldn’t be any doubt as to whether there were 10 or only 8.
Principle #1: Whenever we read numbers, where things have been counted, it’s important to ask, What exactly did they count?
2. How many pages were pulled back from each site and how were those pages selected? Did the researchers just click on links randomly or did they have an algorithm for selecting links?
It’s an obvious possibility that the more pages someone clicks through, the more cookies will be collected, and different kinds of pages could serve cookies differently.
Principle #2: Whenever we read numbers, where things have been counted, it’s important to ask, How did they count?
3. What were the browser (and flash) settings?
Cookie intake is largely controllable by the user. With the browser I’m using presently (Firefox 3.6), I can allow or not allow 3rd party cookies; accept cookies in general, but make exceptions for particular sites I want to deny; deny cookies in general but allow them for particular sites; flush them when I close the browser… It’s fairly granular control, and that’s just with basic browser configuration settings; other functionality is available with plugins or 3rd party software, and if I’m especially fastidious there are tricksy things I can do with my home network firewall, or with the hosts file on my computer.
Given that cookies slurped off the web are controlled by the user, how did the Wall Street Journal’s user doing research set the controls?
Principle #3: Whenever we read numbers, where things have been counted, it’s important to ask, How were things that could affect the count controlled?
With those 3 questions and principles in mind, how does the WSJ’s Full Methodology score?
Not very well, I’m afraid. In fact, it flunks.
But, the fact that the WSJ offered something intended to be a “full methodology” for the research at all, and referenced it in the paper edition, is very much to the Journal’s credit. In general interest media, studies are often cited, but information about how these studies are done — study methodology — is so rare that anything at all about the methodology of a study is worth noting, and even celebrating. With that in view, we can give the WSJ a few points for at least showing up, which pulls its grade up from a solid F to, say, a weak D+.
 Yes, I know that Adobe’s ubiquitous flash software has made absolute control of all cookies somewhat more complex than it was 10 years ago.
 In many situations in life, you may not excel, but you can pass if you just show up.