LMTV WTF | Performance Testing Quickstart (by Tony Fortunato)
Monitoring network file stores by analysing network traffic! (by Darragh Delaney)

Sampling Error (by Paul W. Smith)

Sampling error

Sampling error is a useful tool in the hands of anyone who routinely gathers data, which is pretty much anyone.  It has great value for two-year olds, who spend nearly all of their waking hours collecting data about the world.  It is equally as valuable to adult scientists whose working hours are generally spent recording data for a living.  Simply put, when the data humans gather doesn’t agree with their preconceived view of the world, the convenient explanation is “sampling error.” 

Sampling error is the natural consequence of the inability to examine everything.  If you ask everyone on the planet whether they prefer red wine or white, you will get a very accurate idea of which is the most popular (Note:  those who don’t drink wine are outliers, and can be eliminated).  If, on the other hand, you ask this question of ten people in line at the fish market, you might get all whites.  Since this is completely unreasonable, you can invoke sampling error, and get on with your Pinot Noir. 

As handy as sampling error can be in some situations, it also has a dark side.  Back in the 60’s,  cognitive psychologist Peter Cathcart Wason was busy trying to understand why people unfailingly make certain predictable mistakes in reasoning.  He concluded that folks always tend to favor information that supports their own personal beliefs and preconceived notions, regardless of whether or not that info is true.  Pete’s term for this partiality?  Confirmation bias.  In the case of the wine example, it’s clear that those who answered “white” either aren’t serious wine drinkers, or else they didn’t understand the question.  These too are “outliers” to be ignored. 

If you are neither an oenophile nor a statistics nerd (and are still reading this) there remains some cause for concern.  At the intersection of real life and sampling error, words like “always” and “never” raise an immediate red flag.  Rationalizing away data that don’t support our personal worldview is what makes relationships work, gets politicians elected and helps to keep our egos intact.  It also causes some vexing problems.

Consider Thomas Edison, routinely believed to be the most prolific inventor in U.S. history (he’s not).  Tom was exceptionally good at coming up with things the world didn’t have, but overconfidence in his own judgment led him to disregard outside counsel and trends.  He saw the phonograph (he called it the speaking machine) as an office tool for dictation, rejecting the idea that it would ever be used for entertainment.  He kept his motion picture machine inside a small personal viewing box, ignoring advice to project images on screens for group entertainment. 

Convinced that he had no equal in the practical application of physics, Tom overlooked the advice of military experts and wasted considerable resources trying to develop electromagnets that could catch enemy bullets in flight and send them back to the source.  If you think that Edison invented the phonograph and the motion picture (or that Steve Jobs invented the Reality Distortion Field), your confirmation bias may be showing. 

Scientists have mathematical tools to help with the sampling error problem, many of which are designed to minimize human bias.  Doing this in daily life is not so simple.  Happy people can always find things to be grateful for, just as depressed individuals readily assemble proof that life sucks.  The more emotionally significant an issue grows, the more firmly established the preconceived beliefs, the better our filters become.  Keep the data that works, blame sampling error for the rest, and life will proceed just as you believe it should.

If this whole sampling error thing has you feeling a bit anxious, try this.  Just sit back, relax and pour yourself a glass of red wine.  That works for me.  Always.

PWSmith-Logo InvenTPM Logo

Author Profile - Paul W. Smith, a Founder and Director of Engineering with INVENtPM LLC, has more than 35 years of experience in research and advanced product development.

Prior to founding INVENtPM, Dr. Smith spent 10 years with Seagate Technology in Longmont, Colorado. At Seagate, he was primarily responsible for evaluating new data storage technologies under development throughout the company, and utilizing six-sigma processes to stage them for implementation in early engineering models. He is a former Adjunct Professor of Mechanical Engineering at the Colorado School of Mines, and currently manages the website “Technology for the Journey”.  

Paul holds a doctorate in Applied Mechanics from the California Institute of Technology, as well as Bachelor’s and Master’s Degrees in Mechanical Engineering from the University of California, Santa Barbara.