Thursday, March 15, 2018

My next Tableau Training workshop is Apr. 18-19, 2018 in Vancouver

My next two-day public Tableau training workshop will be held on Wednesday, Apr. 18th and Thursday, Apr. 19th at SFU Harbour Centre. You can buy tickets here or by clicking the button below:

Here are some testimonials from people who've attended my earlier training sessions.

If you can't make this workshop but would like to be alerted when the next one is scheduled, just add your name here.

If you have several people at your organization who need training in Tableau, I'm also available for onsite training.

Wednesday, February 21, 2018

There are fewer journalists in Canada than 15 years ago. But not as few as you might think.

There are fewer journalists in Canada today than 15 years ago — but not as few as you might think, according to data from the 2016 census.

Indeed, between 2001 and 2016 — a time of mass layoffs at daily newspapers across the country — the total number of journalists in the country declined by only 7%.

As the chart above shows, there were just under 13,000 journalists in Canada in 2001, according to the census. That went up to 13,320 in 2006, stayed relatively flat in 2011, then dropped a bit in 2016.

Over the whole period, the decline was just 7%. And even if you measure the decline from the very top — the 13,320 journalists recorded in 2006 — the number of journalists is down by just 10%.

Absolute figures can be a bit misleading, of course. The size of the overall labour force in Canada has risen 18% since 2001, so the relative decline in journalists is greater than it might first appear. Indeed, as a proportion of all working Canadians, the relative share of journalists is down by 20%.

Still, when people talk about how much smaller newsrooms are today than a decade ago, they're usually talking in absolute terms: They look around at cubicles and count heads.

It's also interesting to compare the decline in journalists to other "dying professions", which have seen much steeper declines.

In the interests of not boring readers, I've tried to save the really nerdy stuff to the very end. But, before we go any further, it's important to make note of one important thing: What Statistics Canada considers a "Journalist" and what they don't.

Specifically, the way StatsCan defines "Journalist" specifically excludes two groups that most people would consider journalists: photojournalists and editors (they are instead lumped in with the broader categories for "Photographers" and "Editors").

The degree to which this matters for figuring out what's going on with journalism jobs is explained in (a lot) more detail in the Methodology section below.

The bottom line, though, is that it may be more helpful to think of the "Journalist" numbers as really just showing us the numbers for "Reporters" or other journalists whose main job is writing: columnists, critics, etc.

Even viewed in that limited way, though, the numbers are surprising. Ask anyone working at a newspaper, radio or TV station how many reporters they've lost in their newsroom over the past 15 years and I doubt many would say it's only 10%.

So what's going on?

First, let's start with the obvious: Journalists are losing their jobs.

Annual layoffs and buyouts have been a fact of life at newspapers across the country for at least a decade. I took a buyout from The Vancouver Sun in September 2015, and when I left that paper's newsroom was a fraction of its size when I joined it in 1998.

Last year's Shattered Mirror report into the state of Canadian journalism came up with some sobering estimates based on data from unions (emphasis added):
The Canadian Media Guild has tracked layoffs and buyouts for the past few decades. When non-news companies are excluded, the total is in the order of 12,000 positions lost, more than 1,000 of them in the last year alone. Unifor’s 46 media bargaining units had 1,583 members in 2010 but only 1,125 by early 2016. The CWA estimates it had about 400 editorial members in 2016, a decline of about one-third from 2010 and more than two-thirds since the early 1990s.
Unifor Local 2000 — which represents various newspaper employees in B.C., including those at The Vancouver Sun and Province — told me their membership has dropped from around 2,300 in 2010 to just 800 today (a drop of more than two-thirds).

Some of those union figures include people who work for news companies but aren't journalists, like those working in circulation or advertising. Still, these figures suggest mainstream newsrooms may have seen job losses in the order of 30% or more in just the past few years.

If that's true, that means there's only one way to explain an overall drop in journalists of just 10%.

Someone has to be hiring journalists. And some are.

At the same time as Google and Facebook have gobbled up ad revenue that used to go to newspapers, the Internet has also made it easier to create new media outlets and for niche publications to find an audience.

There are several news organizations in Canada today that didn't even exist 15 years ago, like iPolitics, The Tyee, Discourse, National Observer and Canadaland. Each has its own unique business model: grant funding, subscriptions, donations.

Here in B.C., Castanet, a very successful online-only news site in the Okanagan, has 13 reporters and editors. Metro, the free weekly, has plans to hire a bunch of reporters in Vancouver. And, after a rather brutal round of consolidation and closures, it appears some community papers in the Lower Mainland have started hiring journalists again.

There are also all sorts of niche publications that, while small, do employ actual journalists. Like Modern Dog magazine (based in Vancouver) or The Growler, a quarterly magazine all about B.C. craft beer.

An increase in funding to the CBC hasn't hurt either, with many ex-newspaper employees now doing stellar reporting for the public broadcaster.

The good news doesn't completely outweigh the bad, of course (indeed, statistically, it falls short by about 10%). But there is good news out there. It just tends to come in lots of small doses that may go unnoticed when compared against the big layoffs at big newsrooms.

Does that mean we don't need to worry about the state of journalism in Canada? Of course not.

For one thing, I suspect some of these new journalism jobs don't pay that well. I wasn't able to find salary data for journalists for 2016, but a unionized job at a big-city newspaper almost certainly pays a lot better than being a blogger for some online-only publication.

Also, because of that pay gap, a lot of incredibly talented senior reporters and editors at mainstream news organizations who've lost their jobs are probably more likely to move into Public Relations or communications than take a low-paid job at a digital upstart. That's bad for them, as they had to leave a job they loved and were really good at. And it's bad for the rest of us, as we lose their wealth of experience.

Not all journalism jobs are equally important to society, either.

Losing a city hall reporter at The Province and gaining a blogger at a Hollywood North gossip site is probably not a fair trade in terms of the public good.

That said, I don't think we know enough about what all these new journalism jobs are to fully understand what's going on. For example, trading one movie critic at The Vancouver Sun for one dogged reporter at Castanet probably is a good trade, democracy wise.

My loyalties and biases on this are all over the place.

As a citizen, I want to be able to get reliable information about my community.

As a former journalist at a big-city paper, I think big-city papers do important work and I hope they survive (also, a lot of my good friends still work for them).

And as a journalism instructor, I want my students to get good jobs when they graduate.

On that last one, I think I have the most reason for optimism.

Because at the same time as journalism jobs have declined slightly, jobs that require similar skills — like public relations and photography — have grown substantially.

Indeed, for every job lost in journalism since 2011, there have been 17 jobs added in public relations and advertising (-1,230 vs. +21,320).

That's probably not great for democracy — I'd rather have more watchdogs than spin doctors — but it should soften the blow for journalism students looking for work.

The surge in jobs in related fields like PR should also be good news for journalists who have been laid off (or fear being laid off): Rest assured, there is ample demand out there for your research and writing skills.

I still think we should worry about the decline of newspapers, and other "legacy" news organizations, which serve an important role in democracy.

But these figures at least give me some hope that it's not all bad news in the news business. And that there's at least a chance that new business models will help us figure out a way for journalists to continue to do important work in our communities.


In the interest of keeping my main post concise, I've dumped a lot of background information and methodology stuff here at the bottom. Read on if you want to nerd out some more on census data. But if you just wanted the big picture — and don't get excited by arcane discussions of occupation classifications — you may want to close your browser tab now.


In 2013, while a data journalist at The Vancouver Sun, I stumbled across data on occupations in the 2011 National Household Survey and was surprised to find the number of journalists had stayed relatively flat over the past decade.

I wrote up my findings in a blog post, trying to understand how it was possible — during a time of such doom and gloom in the news business — that the data didn't show a drop in the overall number of journalists.

As I noted in that original blog post, however, I was always a bit cautious about drawing too many conclusions from the data as the NHS, rather infamously, had serious data-quality issues as a result of the Conservative government's controversial decision to kill the mandatory long-form census.

So I was eager to see what the 2016 Census — a return to the mandatory long-form census with record rates of participation — would show about the number of journalists in the country. Especially since, if anything, the last five years have been even doomier and gloomier for journalism jobs than the previous five. While this occupation data isn't perfect (see the methodology section below), I think we can have a lot more confidence in it than the 2011 NHS data.

The occupation data for the 2016 Census actually came out at the end of November. I meant to write something up then. But, unfortunately, I had various other projects on my plate so wasn't able to find time to get to the data right away. The data was also trickier to work with than I expected — especially since I wanted to compare journalists to other job categories over the same period.


As I did in my 2013 blog post, I thought it might be helpful to address some of the common questions people often have when presented with this data:

Are more people just calling themselves journalists? This is the idea that people who blog about politics or post YouTube videos reviewing video games, are somehow calling themselves journalists now and that might be throwing off the numbers. It's possible. But they'd have to be making a living doing it. The questions about occupation on the long-form census questionnaire (Questions 36 on) are pretty darn specific and instruct people — if they have more than one job — to list the one they spend the most hours at. So someone who has a day job as a roofer and a side gig reviewing video games on YouTube would (or should) put down their job as "roofer". Or to use a personal example, in May 2016, I would have listed by job as "university instructor" not journalist, even though I was still doing the odd freelance gig. Statistics Canada also has pretty strict rules about what counts as a journalist and what doesn't (more on that below).

Aren't some of these journalists unemployed? So the short answer is: Yes. But the slightly longer answer is: It shouldn't matter that much to the overall trend. The 2016 data actually shows that 560 of the 12,050 journalists (or 5%) were unemployed at the time of the census. That's actually lower than the unemployed share for all workers in Canada, which is 7.7%. Also, as noted above, you only list an occupation at all if you've had some kind of job since January 1, 2015. Otherwise, you leave the occupation section blank. And if you've found another job — any job, even a barista at Starbucks — you should be listing that as your occupation, not "journalist". Which means that while there could be a bit of a lag in the data, for journalists who've just lost their job and haven't found another one yet, it shouldn't make that much difference to the analysis. (I tried finding the number of unemployed journalists for previous Census years but was unable to.)

Are more journalists working freelance? When I looked at the data in 2013, there didn't appear to be any change in the number of journalists who identified themselves as self-employed. Indeed, the figure had gone down from 16.2% in 2011 to 14.8% in 2011. Unfortunately, I wasn't able to find any data on the rate of self-employment for journalists in 2016.

Are journalists making less money? As noted above, I haven't been able to find any data on journalist incomes from the 2016 Census. In 2011, journalists incomes were up 4.8% since the 2006 census. Still, that was less than the average increase for all workers for that period (15%) and less than the rate of inflation. If I had to guess, I would imagine that trend has gotten even worse in 2016 — especially since, anecdotally, I know many newspaper newsrooms have seen 0% wage increases for several years. And journalists in some newsrooms have admirably taken pay cuts and reduced hours to help save the jobs of more junior colleagues.


For people who really want to nerd out, here's the nitty gritty details on how I pulled this data together and some of the caveats that accompany it.

I started with Census/NHS tables breaking down the number of people in each occupational category for 2001, 2006, 2011 and 2016.

I then copy and pasted all that data into one spreadsheet.

The occupational data from the census is based on how people answered two census questions on the long-form census: "What was this person's work or occupation?" and "In this work, what were this person's main activities?" You can fill in whatever you want in response to those two questions and then, based on what you write down, StatsCan assigns you to one of 500+ different job categories.

All sorts of written job descriptions — from "reporter" to "cyberjournalist" — are treated as "Journalist".

As noted above, though, the category of Journalist specifically excludes two groups that most people would consider journalists: photojournalists and editors. Photojournalists are, instead, included under the broader category of "Photographers". And editors — which includes copyeditors, news editors and sports editor — are counted under the broader category of "Editors".

By definition, this means StatsCan is undercounting what most people would consider "journalists". But for the purposes of this post, what's more important is whether that might give us a misleading picture of the rate of job losses.

If the job loss trends are the same for those excluded jobs — if photojournalists and copyeditors have lost their jobs at the same rate as reporters (7-10%) — it shouldn't make a difference that they aren't included in the "Journalist" category. The overall trend would still be the same.

But if photojournalists and/or copyeditors are losing their jobs at a faster rate than reporters, things might be worse than they appear in the charts above. And that seems at least possible.

Anecdotally, copyeditors were one of the first groups to be cut at many newspaper newsrooms. Many papers went from having an army of copyeditors to just a handful. (If you've noticed more typos in your newspaper recently, now you know why.) By not having editors in the "Journalist" category, we may be getting a misleadingly rosy picture of what's really going on. I have less of a sense of whether photojournalists lost their jobs at a faster rate than reporters but I suspect that should have less impact on the overall numbers as, at least in the newsrooms I've worked in, there were a lot more reporters and copyeditors than photographers.

Complicating the analysis is that — as you can see above — the overall categories of "Editors" and "Photographers" have both grown pretty steadily over the past 15 years. But that's probably because the job losses of photojournalists and copyeditors have been outweighed by non-journalism jobs like wedding photographers and technical editors.

As noted above, perhaps the best way to think of the "Journalist" numbers is that it counts those who do reporting or writing. For that group of people, the numbers should be pretty reliable and, I think, still give us a good sense of what's going on.

("Journalist" also excludes "Announcers and other broadcasters" but my read of that category suggests that shouldn't make as big a difference to the overall figures. While "Announcers" includes "news reader", "anchorman/woman" is still explicitly included under the "Journalist" category.)

The other tricky issue with analyzing the data was that occupational categories and codes sometimes change between Census years.

Luckily, the category for Journalist has stayed relatively stable over the entire period, as have most of the other categories compared in this blog post.

The trickiest job category was Public Relations, which had a major shift in definition between 2006 and 2011. After referring to the 2011 Concordance tables provided by StatsCan, I originally settled on using the F024 code for 2001 and 2006 and the 1123 code for 2011 and 2016, as the categories sounded pretty similar:
  • 2006: F024 Professional occupations in public relations and communications
  • 2011: 1123 Professional occupations in advertising, marketing and public relations
However, the more I looked at the chart that created, the more I had second thoughts.

As you can see, the number of people in the job category basically doubled between 2006 and 2011 (when the definition changed), compared to a 34% increase from 2001 to 2006 and a 29% increase from 2011 to 2016. That struck me as suspicious and a sign that I might be comparing apples and oranges.

So I ended up deciding to just show the 2011 and 2016 data above, as the occupation code remained stable between those two years. Not ideal, but probably more accurate.

I really, really wish that StatsCan had kept the PR category alone for all four census years, as I'm more interested in comparing PR jobs with journalism, as the skills for those two occupations are so similar. Having advertising folks included after 2011 muddies the waters a bit. But I had to work with the data available.

If you'd like to play around with the data yourself, you can find the raw data here and my Tableau workbook here. If you notice any errors in my analysis or you have any ideas or suggestions, please comment below your can drop me a line on Twitter.

Tuesday, February 13, 2018

On the housing debate, ask yourself: "What if I'm wrong?"

Photo by MyEyeSees
I'm trying to follow my own advice to jot down ideas on my blog and not just in tweetstorms. Below is the full text of a tweet thread of mine on Vancouver's housing debate.

As with so many debates, I think it would be helpful if those arguing over the impact of foreign buyers on Vancouver’s housing market asked themselves a simple question:

“What if I’m wrong?”

I’ll start. I suspect foreign buyers are much less of a factor than many believe. What evidence we have suggests they are maybe 5% of the market. I think other factors — like domestic speculation and housing supply — are a much bigger cause of high prices.

But: What if I’m wrong?

Maybe the tools we have for measuring foreign buyers aren’t good enough, understating their true numbers. Maybe foreign buyers, while small in percentage terms, create “spillover effects” that inflate the market. What if foreign buyers *are* a huge part of the problem? What then?

Well, then, a solution that focuses only on supply — even a whole lot of supply — probably won’t be enough to make Vancouver housing affordable. Because a big chunk of that new supply will just get snapped up by foreign buyers and prices will continue to rise.

So the best approach, even for a foreign-buyer skeptic like me, is to support smart measures to address foreign demand (like @HousingBC's tax proposal). At worst, they should still help a little bit. And they may be essential to solving the problem.

But the same logic applies to those on the other side. What if foreign buyers *aren’t* the main factor driving Vancouver’s housing crisis? What if, for all the stories about offshore buyers, there just aren’t enough of them to make that big of a difference?

What then? Well, then, even the most extreme restrictions — like banning foreign buyers outright — won’t be enough to bring down prices.

Which means those folks who think the problem is mostly foreign buyers should still support smart measures to address housing supply, such as rental-only zoning and more density. At worst, they should still help a little bit. And they may be essential to solving the problem.

Now, some will say: “Let’s try my thing first. Then, if that doesn’t work, we can try your thing.” There are a couple problems with this. The first, and most obvious, is we then have to wait even longer for things to get better.

The second is that the policies being proposed are probably a good idea even if they DON’T bring down prices. While NIMBY’s hate it, more density in single-family areas should make our communities better: less car dependent, more inclusive and with fewer carbon emissions.

And even if the @HousingBC tax proposal does nothing to bring down prices, it will force rich property owners who don’t pay income tax (whether foreign buyers or drug dealers) to pay their fair share towards government services. That’s a good thing!

Exactly what impact foreign buyers are having on Vancouver’s housing market is unknown. And, more than that — given the limited tools we have for measuring it and the complexities of housing markets — it may be, in a very real sense, *unknowable*.

We know supply and demand are the two factors that determine prices.

We have no way of knowing, for sure, how much of Vancouver’s housing crisis is one or the other.

Should we keep arguing about it? Or get to work addressing both, as quickly as possible?

Thursday, February 8, 2018

What do we mean when we talk about Vancouver's housing "crisis?"

Photo by Adrian Farcas

I'm trying to follow my own advice to jot down ideas on my blog and not just in tweetstorms. Below is the full text of a tweet thread of mine on Vancouver's housing crisis.

I think one reason people can’t agree on what to do about Vancouver’s housing crisis is because they can’t agree on what the crisis actually *is*. In particular, for many, the crisis is: “Upper middle class people can’t afford detached homes in Vancouver.”

You see evidence of this in news coverage. Vancouver homes have been unaffordable for most families since the early 2000s, if not before. But it became a “crisis” when well-paid professionals (like newspaper columnists) got priced out. I don’t think that’s a coincidence.

You see it in the stories people tell. “When my doctor friend moved here from Alberta, he could only afford a place in New West!” “My cousin has an MBA and lives in a condo!” The “crisis”, for many, is that well-off professionals can’t have the lifestyle they feel entitled to.

Expensive housing hurts everyone who isn't already an owner. But not all hardship is morally equivalent. Someone who has to commute 2hrs to their minimum wage job, or a single mom living with 3 kids in a basement suite, is *worse off* than a lawyer who can only afford a townhome.

Housing markets are complex. And anything that reduces demand or increases supply could plausibly help everyone. But I think how one defines the problem affects which solutions they think should take priority. For example...

Cracking down on foreign and speculative demand might soften the market enough to make it easier for well-off professionals to buy a house. But it probably won’t do much to bring up vacancy rates. And an $800k house isn’t any more “affordable” for most families than a $1.6m one.

Similarly, more rental-only zoning and purpose-built rental in single-family neighborhoods should bring down rents and allow renters to live closer to work. But it may not have much immediate impact on the price of detached homes.

I think it would be helpful if those engaged in the housing debate took a moment to ask themselves:

  • What do you mean when you talk about the housing “crisis”?
  • What problem are you most trying to solve?
  • What group of people are you most trying to help?

My take: It’s nice to own your own home. And if we can make that easier for middle-class families, great. But it’s far more important that everyone has a secure, affordable roof over their head. And more rental is a big part of the solution to that problem.

Luckily, we don’t have to choose. We can do it all: Crack down on foreign/speculative demand (ie @HousingBC), allow more density in single-family neighbourhoods AND build more purpose-built rental. However, doing just one likely won’t help everyone.

Analysis of the 2018 BC Liberal leadership race

On Monday afternoon, I noticed a tweet from CBC reporter Justin McElroy that the BC Liberals had released riding-by-riding vote totals for their leadership race.

While the Liberals deserve kudos for releasing this data (the BC NDP famously didn't in 2011),  it was annoying that the data was released as a PDF rather than a spreadsheet. PDFs are fine for looking up individual results for individual ridings. But spreadsheets are best for really analyzing the patterns in the data.

From dealing with PDFs in the past, I knew there were actually some pretty decent tools out there for extracting data from them. Chief among them: Tabula, a great little program designed by journalists, for journalists.

My first attempts to extract the data using Tabula didn't work: When it tried to automatically detect the tables, it misjudged and ended up leaving off some rows of data. Not good! But after some fiddling, I was able to successfully grab the data and then, with a little bit of fiddling in Excel, create a nice, clean spreadsheet for analysis.

My original plan was just to get the data into a nice format and make it public so other folks could play around with it. But then once I had the data I figured: What the heck. I want to play with it!

So my next few hours — with a short break for dinner — were spent crunching the numbers and looking for interesting patterns in the data.

The most interesting finding (and the one that seemed to generate the most retweets): Andrew Wilkinson really benefited from the point system used by the Liberals, in which every riding got 100 points regardless of how many members it had. Indeed, looking at the raw vote totals for each round, Michael Lee was ahead in every single round he participated in. And, indeed, in Round 4, when Wilkinson knocked Lee out by points, Wilkinson was actually in third place by raw votes.

My analysis even got a nice little write-up in my former paper.

I pulled all my charts and maps into a Tableau workbook. It's too big to embed in this blog post, but you can see it by clicking here.

Monday, December 18, 2017

Coming this January: A special "Extended Edition" of my Tableau training

I've been doing Tableau training workshops for a few years now and the feedback I get from the people who take it is often the same: They loved it, but wished there was even more of it.

Many have expressed interest in a second, "Intermediate", training workshop that would cover more advanced Tableau skills. I hope to offer such a workshop at some point in the future.

But in the meantime, I thought I could easily expand my current workshop from three days to four to give participants even more Tableau goodness: to move beyond the basics into some of Tableau's more advanced (and really cool!) features like reshaping data, joining datasets and using calculations in your visualizations.

So next month, for the first time, I'm offering a special Extended Edition of my online Tableau training with four days of training instead of the usual three. For the math nerds, that's 33% more Tableau goodness.

The training will run for four hours on two Wednesdays and Thursdays in a row: Jan. 17-18 & Jan. 24-25, 2018.

You can buy tickets on Eventbrite right here:

If you can't make this workshop but would like to be alerted when the next one is scheduled, just add your name here.

And if you have several people at your organization who need training in Tableau, I'm also available for onsite training.

Tuesday, October 17, 2017

Infographic: How the way we vote makes B.C.'s "urban/rural divide" seem much worse than it really is

I made a little infographic in Tableau Public on B.C.'s "urban/rural divide" and electoral reform. The interactive version is below. You can view the static version here or download it from here.

I'm a bit of a B.C. politics junkie (if you are too, I highly recommend CHNL's "Inside Politics" podcast). And since the 2017 election, there's been a lot of talk about the province's "urban/rural divide". Specifically, how few seats the NDP got outside of Metro Vancouver and Vancouver Island.

There's also been some talk lately about electoral reform, as the new NDP/Green government is planning a referendum on proportional representation next fall.

Which, weirdly, got me thinking of the 1993 Canadian federal election. That's the one where the separatist Bloc Quebecois became the Official Opposition with 54 seats and Kim Campbell's Progressive Conservatives government was completely humiliated, winning only 2 (!) seats.

But the really weird thing about the 1993 election was that the Progressive Conservatives actually got more votes (2.2 million) than the BQ (1.8 million) did.

But because the BQ's votes were geographically concentrated in Quebec, while the PC's were scattered across the country, the BQ got way more seats. The way Canada voted rewarded a party with regional, rather than national, ambitions.

I wondered if something similar might have happened in B.C.: Whether the way we elect MLAs might make our regional divisions seem more severe than they really are.

So I did a bit of number crunching using Elections BC data and map files.

The situation isn't anywhere near as severe as the 1993 election, but the First Past The Post electoral system does definitely exaggerate the BC Liberals' popularity in rural B.C. (and, to a lesser extent, the NDP's popularity in urban areas).

The biggest decision I had to make was where to draw the boundaries for the purpose of my analysis. I've seen some pundits refer to how the NDP didn't get many seats outside the "Lower Mainland" and "Southern Vancouver Island". But those are slightly fuzzy concepts. Where does one divide Vancouver Island between north and south? And while many people think of the Lower Mainland as the same thing as Metro Vancouver, it's not.

So, instead, I decided to lump Metro Vancouver (a well-defined regional district) together with all of Vancouver Island. That seemed to fit the electoral patterns of the province best. After all, the NDP is pretty popular up and down the Island and the Liberals are still pretty strong in Fraser Valley communities like Abbotsford and Chilliwack.

If you don't agree with my choices, though, you're more than welcome to download the Tableau workbook yourself and choose different regions.