One of them, from Steve Haroz, noted that he knew of know evidence that people read non-zero-baseline bar charts any differently than non-zero-baseline line charts. And, furthermore, that we should be careful in talking about data visualization "rules" when our evidence for them is weak or nonexistent.
Rely on evidence. When there is little or none, don't claim that some absolute or universal rule exists. It's fine to say "maybe" or "I prefer".— Steve Haroz (@sharoz) May 24, 2018
This led to a quite spirited discussion about whether data-visualization "guidelines" or "rules of thumb" that don't have any empirical research to back them up can still be valuable, or if we should stick primarily to those things that we have solid evidence for.
Speaking personally, I didn't fully appreciate the gaps in data visualization research until I watched Robert Kosara's excellent talk at the University of Washington, "How Do We Know That?"
The talk is based on Kosara's paper, Empire of Sand, which I now assign to my students at the University of Florida.
As Kosara points out, many of the things we think we know about data visualization have little empirical evidence to back them up. And other well-accepted "rules" may actually be wrong (for example, "chartjunk" may not be so bad after all).
Some rules are based on nothing more than the strong opinions of influential early writers in the field (like Edward Tufte and Jacques Bertin) and have not actually been subject to peer-reviewed research.
So where does that leave us as data visualization practitioners and teachers?
It would seem obvious that we shouldn't teach "rules" that we know to be wrong. But what about the many areas for which there is little or no empirical evidence at all? Can theory replace research in some cases? Is a common practice worth teaching our students even if we don't know it to be true?
Below, I've tried to collect some of my own thoughts on the matter as well as those of others who took part in the Twitter discussion.
First, though, a big caveat about my own tweets: While I teach at a university and have (strong) opinions on how to teach data visualization, I'm an "instructor" not a "professor". I don't have a PhD and I'm not engaged in academic research myself.
Let's get to the tweets!
My argument is based more on encoding consistency, though, and prudence: if encoding is height, then it may be safer if height is proportional to numbers— Alberto Cairo (@albertocairo) May 24, 2018
We don't know what visual cue people use or how they reason with it. Could be length. Could be area. Could be position of the top of the bar. And not clear why a point's distance from the x-axis is different from the length of a bar. There are plausible options, but no evidence.— Steve Haroz (@sharoz) May 24, 2018
That’s the key point! There’s elegance, Bertin-style retinal variables, etc., and then there’s the actual psychology, cognition, etc., and there are still big gaps in our knowledge of the latter.— Robert Kosara 👀 (@eagereyes) May 24, 2018
Why give credence to the former?— Steve Haroz (@sharoz) May 24, 2018
Cause we don’t have enough of the latter?— Robert Kosara 👀 (@eagereyes) May 24, 2018
Medicine doesn't know how to cure every cancer. That doesn't mean that shamans and witch doctors are worth anyone's time.— Steve Haroz (@sharoz) May 24, 2018
It strikes me that both these statements can be true:— Chad Skelton (@chadskelton) May 24, 2018
1. We need more research on how people interpret charts to ensure our guidelines are backed by actual evidence.
2. In the meantime, guidelines can still be helpful, even if they're not based on peer-reviewed research
I'd say that would be a big improvement over the field 5-10 years ago, which basically was 1. We need to always do what it says to do in a couple really popular books on this topic, and 2. Anyone who doesn't is an idiot.— Ben Jones (@DataRemixed) May 24, 2018
I think the key, which to be fair I think @sharoz agrees with, is that we should be upfront about the nature of our advice:— Chad Skelton (@chadskelton) May 24, 2018
- This is backed by peer-reviewed research
- This is based on best practices in the field
- This is based on a hunch I had this morning
I agree it's good advice. Just call it "common practice" instead of best.— Steve Haroz (@sharoz) May 24, 2018
I tend to be a bit conservative ideologically, at least in some areas. I think that traditions and customs often have good reasons to exist (until they are solidly refuted)— Alberto Cairo (@albertocairo) May 24, 2018
Tufte has strong opinions that are largely unburdened by evidence. That’s a huge problem for the field. Bertin at least gave us a good framework to think in, even if some of his ideas might prove to be wrong.— Robert Kosara 👀 (@eagereyes) May 24, 2018
I value internal consistency (more visible in Bertin's work) and how it translates into a style. The more it overlaps scientific evidence the better, but it's not a requirement.— Jorge Camoes (@wisevis) May 24, 2018
Heresy, I know.
I like @chadskelton's "have it both ways" framing - there's value in both. Attempts to reason about best practices based on what we think we understand now are useful since if we waited until we had empirical studies answering every question, we'd wait a zillion years!— Tamara Munzner (@tamaramunzner) May 25, 2018
Of course empirical studies are worth doing to refine our understanding. But the combinatorial explosion of questions will outstrip the number of answers, especially since every answer leads to so many followup questions. We'll never be done. So we continue to need both.— Tamara Munzner (@tamaramunzner) May 25, 2018
Hmm. Maybe a rating & review system for rules of thumb would be interesting. X% of practitioners follow Rule A virtually always, Y follow it often, Z follow it rarely, etc, with comments & links to examples & studies...— Ben Jones (@DataRemixed) May 25, 2018
Great idea! Sounds similar to the University of Chicago economist survey. Interestingly, that survey is *not* anonymous, which I think makes it better as people have to stand by their views. https://t.co/CjyD8sqoHC— Chad Skelton (@chadskelton) May 25, 2018
I believe Min Chen at Oxford is doing something along these lines ...— Enrico Bertini (@FILWD) May 25, 2018
I was curious about the project Enrico mentioned but Chen didn't appear to be on Twitter, so I sent him an email.
Chen sent me a very nice email back directing me to the Visualization Guidelines Repository.
The repository is still a work in progress, but an example on "chartjunk" suggests it could eventually be similar to what Ben Jones was suggesting: Links to where guidelines come from and studies that support or refute them.
There is also a related project, VisGuides, which is a platform to discuss visualization guidelines. (VisGuides was presented at Eurovis this week.)
Chen told me the two projects were setup by four visualization scientists: Alexandra Diehl, Alfie Abdul-Rahman, Menna El-Assady and Benjamin Bach.
It will be interesting to see how the Repository and VisGuides develops.
But I wonder if there isn't also a space for something more like the University of Chicago economists survey, but for data visualization: A place where people can see at a glance what leading practitioners in the field think about different guidelines.
I think this would provide useful information about which guidelines are universally accepted (i.e. "95% of practitioners think bar charts should start at zero") and which are more contested (i.e. "30% of practitioners think line charts should usually start at zero").
With sufficient buy-in, it could also provide a one-stop shop for people to check in with their favourite thinkers in the field when struggling with a chart decision. ("I want to make a pie chart with eight slices. What would Alberto Cairo think about that?" "Would Cole Nussbaumer Knaflic approve of me truncating this axis?")
If you've got thoughts on this topic, please post a comment below or hit me up on Twitter. Because of spam comments, my comments are moderated so don't be alarmed if yours doesn't show up right away. It will within a few hours.
No comments:
New comments are not allowed.