So, how is it that the Wall Street Journal’s Full Methodology flunks the basic, pass-fail test? Why do all the numbers in “On Web, Children Face Intensive Tracking” add up to a zero-value story?
The Full Methodology is disappointing on several counts, but the core problem is related to Principle #3 in the previous post: Whenever we read numbers, where something has been counted, it’s important to ask, How were things that could affect the count controlled?
The Full Methodology tells us this:
Mr. Campbell used Mozilla Firefox 3.5 and Adobe Flash Player 10.0. Following each session, he examined the tracking files that had been placed on the computer.
That’s all we’re told about the browser. We don’t know the main thing: How was the browser’s cookie management configured?
Which browser was used is of relatively minor importance. I happen to have IE6 on the computer I’m using now, and even that relic allows blocking of either third-party cookies or first-party cookies, or both. Firefox 3.6, which I actually use, has much more granular cookie handling functionality. I can allow cookies only from certain sites, or block them only from certain sites. Or block all 3rd party cookies, etc.
Since the research involved collecting cookies from websites and counting them, the main thing we want to know is how Mr. Campbell, the researcher, configured the browser for cookies. What gets caught in a net very much depends on what the net is designed to catch — minnows or tuna or whatever — and what the net is designed to catch is analogous to how the user has decided to configure his or her browser. If you’re browsing a website, what you accept in the way of cookies in under your control, not the website’s.
There is a clue to how Mr. Campbell, configured his browser in the legend for a graph on the WSJ site, “What we found on one site”. The site is Snazzyspace.com. There were 195 cookies and 4 first party cookies…
If they know there were 4 first-party cookies, and there were 195 total cookies, there must have been 191 third-party cookies, and to know that, the browser must have been set to accept third-party cookies. Mr. Campbell could have blocked 191 cookies simply by de-selecting the “Accept third-party cookies” box in Firefox.
But four measly cookies wouldn’t make a sensational story would it? Four cookies from hitting 20 pages on a website just doesn’t rate front-page play. Is it possible that… Did the Journal promote 10 inches worth of filler from inside Weekend Journal to a front-page splash by telling Mr. Campbell to set his browser to accept 3rd party cookies?
 A website might not work if you block all cookies, which doesn’t change the fact that you’re in control. A discussion of cookie management can get quite involved. What I’m considering here are not flash cookies, but the old-fashioned, 90s technology flat text cookies we’re all familiar with, which in recent versions of Firefox are kept in a sqlite database.
 I thought the default in Firefox was to block third-party cookies, which means to get all those 191 cookies you’d need to deliberately change the setting. I need to check this.