Tuesday, February 13, 2018

On the housing debate, ask yourself: "What if I'm wrong?"

Photo by MyEyeSees
I'm trying to follow my own advice to jot down ideas on my blog and not just in tweetstorms. Below is the full text of a tweet thread of mine on Vancouver's housing debate.

As with so many debates, I think it would be helpful if those arguing over the impact of foreign buyers on Vancouver’s housing market asked themselves a simple question:

“What if I’m wrong?”

I’ll start. I suspect foreign buyers are much less of a factor than many believe. What evidence we have suggests they are maybe 5% of the market. I think other factors — like domestic speculation and housing supply — are a much bigger cause of high prices.

But: What if I’m wrong?

Maybe the tools we have for measuring foreign buyers aren’t good enough, understating their true numbers. Maybe foreign buyers, while small in percentage terms, create “spillover effects” that inflate the market. What if foreign buyers *are* a huge part of the problem? What then?

Well, then, a solution that focuses only on supply — even a whole lot of supply — probably won’t be enough to make Vancouver housing affordable. Because a big chunk of that new supply will just get snapped up by foreign buyers and prices will continue to rise.

So the best approach, even for a foreign-buyer skeptic like me, is to support smart measures to address foreign demand (like @HousingBC's tax proposal). At worst, they should still help a little bit. And they may be essential to solving the problem.

But the same logic applies to those on the other side. What if foreign buyers *aren’t* the main factor driving Vancouver’s housing crisis? What if, for all the stories about offshore buyers, there just aren’t enough of them to make that big of a difference?

What then? Well, then, even the most extreme restrictions — like banning foreign buyers outright — won’t be enough to bring down prices.

Which means those folks who think the problem is mostly foreign buyers should still support smart measures to address housing supply, such as rental-only zoning and more density. At worst, they should still help a little bit. And they may be essential to solving the problem.

Now, some will say: “Let’s try my thing first. Then, if that doesn’t work, we can try your thing.” There are a couple problems with this. The first, and most obvious, is we then have to wait even longer for things to get better.

The second is that the policies being proposed are probably a good idea even if they DON’T bring down prices. While NIMBY’s hate it, more density in single-family areas should make our communities better: less car dependent, more inclusive and with fewer carbon emissions.

And even if the @HousingBC tax proposal does nothing to bring down prices, it will force rich property owners who don’t pay income tax (whether foreign buyers or drug dealers) to pay their fair share towards government services. That’s a good thing!

Exactly what impact foreign buyers are having on Vancouver’s housing market is unknown. And, more than that — given the limited tools we have for measuring it and the complexities of housing markets — it may be, in a very real sense, *unknowable*.

We know supply and demand are the two factors that determine prices.

We have no way of knowing, for sure, how much of Vancouver’s housing crisis is one or the other.

Should we keep arguing about it? Or get to work addressing both, as quickly as possible?

Thursday, February 8, 2018

What do we mean when we talk about Vancouver's housing "crisis?"

Photo by Adrian Farcas

I'm trying to follow my own advice to jot down ideas on my blog and not just in tweetstorms. Below is the full text of a tweet thread of mine on Vancouver's housing crisis.

I think one reason people can’t agree on what to do about Vancouver’s housing crisis is because they can’t agree on what the crisis actually *is*. In particular, for many, the crisis is: “Upper middle class people can’t afford detached homes in Vancouver.”

You see evidence of this in news coverage. Vancouver homes have been unaffordable for most families since the early 2000s, if not before. But it became a “crisis” when well-paid professionals (like newspaper columnists) got priced out. I don’t think that’s a coincidence.

You see it in the stories people tell. “When my doctor friend moved here from Alberta, he could only afford a place in New West!” “My cousin has an MBA and lives in a condo!” The “crisis”, for many, is that well-off professionals can’t have the lifestyle they feel entitled to.

Expensive housing hurts everyone who isn't already an owner. But not all hardship is morally equivalent. Someone who has to commute 2hrs to their minimum wage job, or a single mom living with 3 kids in a basement suite, is *worse off* than a lawyer who can only afford a townhome.

Housing markets are complex. And anything that reduces demand or increases supply could plausibly help everyone. But I think how one defines the problem affects which solutions they think should take priority. For example...

Cracking down on foreign and speculative demand might soften the market enough to make it easier for well-off professionals to buy a house. But it probably won’t do much to bring up vacancy rates. And an $800k house isn’t any more “affordable” for most families than a $1.6m one.

Similarly, more rental-only zoning and purpose-built rental in single-family neighborhoods should bring down rents and allow renters to live closer to work. But it may not have much immediate impact on the price of detached homes.

I think it would be helpful if those engaged in the housing debate took a moment to ask themselves:

  • What do you mean when you talk about the housing “crisis”?
  • What problem are you most trying to solve?
  • What group of people are you most trying to help?

My take: It’s nice to own your own home. And if we can make that easier for middle-class families, great. But it’s far more important that everyone has a secure, affordable roof over their head. And more rental is a big part of the solution to that problem.

Luckily, we don’t have to choose. We can do it all: Crack down on foreign/speculative demand (ie @HousingBC), allow more density in single-family neighbourhoods AND build more purpose-built rental. However, doing just one likely won’t help everyone.

Analysis of the 2018 BC Liberal leadership race

On Monday afternoon, I noticed a tweet from CBC reporter Justin McElroy that the BC Liberals had released riding-by-riding vote totals for their leadership race.

While the Liberals deserve kudos for releasing this data (the BC NDP famously didn't in 2011),  it was annoying that the data was released as a PDF rather than a spreadsheet. PDFs are fine for looking up individual results for individual ridings. But spreadsheets are best for really analyzing the patterns in the data.

From dealing with PDFs in the past, I knew there were actually some pretty decent tools out there for extracting data from them. Chief among them: Tabula, a great little program designed by journalists, for journalists.

My first attempts to extract the data using Tabula didn't work: When it tried to automatically detect the tables, it misjudged and ended up leaving off some rows of data. Not good! But after some fiddling, I was able to successfully grab the data and then, with a little bit of fiddling in Excel, create a nice, clean spreadsheet for analysis.

My original plan was just to get the data into a nice format and make it public so other folks could play around with it. But then once I had the data I figured: What the heck. I want to play with it!

So my next few hours — with a short break for dinner — were spent crunching the numbers and looking for interesting patterns in the data.

The most interesting finding (and the one that seemed to generate the most retweets): Andrew Wilkinson really benefited from the point system used by the Liberals, in which every riding got 100 points regardless of how many members it had. Indeed, looking at the raw vote totals for each round, Michael Lee was ahead in every single round he participated in. And, indeed, in Round 4, when Wilkinson knocked Lee out by points, Wilkinson was actually in third place by raw votes.

My analysis even got a nice little write-up in my former paper.

I pulled all my charts and maps into a Tableau workbook. It's too big to embed in this blog post, but you can see it by clicking here.

Monday, December 18, 2017

Coming this January: A special "Extended Edition" of my Tableau training

I've been doing Tableau training workshops for a few years now and the feedback I get from the people who take it is often the same: They loved it, but wished there was even more of it.

Many have expressed interest in a second, "Intermediate", training workshop that would cover more advanced Tableau skills. I hope to offer such a workshop at some point in the future.

But in the meantime, I thought I could easily expand my current workshop from three days to four to give participants even more Tableau goodness: to move beyond the basics into some of Tableau's more advanced (and really cool!) features like reshaping data, joining datasets and using calculations in your visualizations.

So next month, for the first time, I'm offering a special Extended Edition of my online Tableau training with four days of training instead of the usual three. For the math nerds, that's 33% more Tableau goodness.

The training will run for four hours on two Wednesdays and Thursdays in a row: Jan. 17-18 & Jan. 24-25, 2018.

You can buy tickets on Eventbrite right here:

If you can't make this workshop but would like to be alerted when the next one is scheduled, just add your name here.

And if you have several people at your organization who need training in Tableau, I'm also available for onsite training.

Tuesday, October 17, 2017

Infographic: How the way we vote makes B.C.'s "urban/rural divide" seem much worse than it really is

I made a little infographic in Tableau Public on B.C.'s "urban/rural divide" and electoral reform. The interactive version is below. You can view the static version here or download it from here.

I'm a bit of a B.C. politics junkie (if you are too, I highly recommend CHNL's "Inside Politics" podcast). And since the 2017 election, there's been a lot of talk about the province's "urban/rural divide". Specifically, how few seats the NDP got outside of Metro Vancouver and Vancouver Island.

There's also been some talk lately about electoral reform, as the new NDP/Green government is planning a referendum on proportional representation next fall.

Which, weirdly, got me thinking of the 1993 Canadian federal election. That's the one where the separatist Bloc Quebecois became the Official Opposition with 54 seats and Kim Campbell's Progressive Conservatives government was completely humiliated, winning only 2 (!) seats.

But the really weird thing about the 1993 election was that the Progressive Conservatives actually got more votes (2.2 million) than the BQ (1.8 million) did.

But because the BQ's votes were geographically concentrated in Quebec, while the PC's were scattered across the country, the BQ got way more seats. The way Canada voted rewarded a party with regional, rather than national, ambitions.

I wondered if something similar might have happened in B.C.: Whether the way we elect MLAs might make our regional divisions seem more severe than they really are.

So I did a bit of number crunching using Elections BC data and map files.

The situation isn't anywhere near as severe as the 1993 election, but the First Past The Post electoral system does definitely exaggerate the BC Liberals' popularity in rural B.C. (and, to a lesser extent, the NDP's popularity in urban areas).

The biggest decision I had to make was where to draw the boundaries for the purpose of my analysis. I've seen some pundits refer to how the NDP didn't get many seats outside the "Lower Mainland" and "Southern Vancouver Island". But those are slightly fuzzy concepts. Where does one divide Vancouver Island between north and south? And while many people think of the Lower Mainland as the same thing as Metro Vancouver, it's not.

So, instead, I decided to lump Metro Vancouver (a well-defined regional district) together with all of Vancouver Island. That seemed to fit the electoral patterns of the province best. After all, the NDP is pretty popular up and down the Island and the Liberals are still pretty strong in Fraser Valley communities like Abbotsford and Chilliwack.

If you don't agree with my choices, though, you're more than welcome to download the Tableau workbook yourself and choose different regions.

Thursday, October 12, 2017

My Missing Maps from The Vancouver Sun

I haven't been a data journalist at The Vancouver Sun for more than two years now. But still, every couple months or so, I get an email from someone saying they're having trouble finding an old map or chart from my Sun days and wondering if I could help them track it down.

That's because dozens of maps and charts that I created when I was at The Sun have disappeared from the site since I left.

My articles and blog posts are still there. But in many cases, when they refer to an interactive chart or map, there either isn't anything there at all, or there's a weird error message saying a file can't be found.

The problem of old data journalism gradually disappearing from the web is widespread.

In this case, the blame lies half with The Vancouver Sun and half with me.

First, The Sun's share of the blame. Or, more accurately, Postmedia's.

Every so often, Postmedia, the company that owns The Sun and most other major papers in Canada, updates its websites, often adopting a whole new platform (like Wordpress). This never goes smoothly.

I still remember the first time The Sun transitioned to a new blogging platform. We were assured all the old posts would show up in the new system. That was technically true, but in the process many of the URLs changed, so none of the posts could be found in their old spot (and if someone else had linked to a post in the past, readers now got a 404 Error).

The latest updates to The Sun's website seems to have kept all the blog posts in the same spot. But it stripped out all of the "embed codes" — the stuff that makes sure that maps and charts appear in the blog post properly. The result is that all of the blog posts I wrote that had maps or charts in them no longer do. (For some reason, embed codes in articles seem to have survived in most cases.)

Luckily, in most cases, this problem is easy to fix.

Above most of those embed codes, I put a little note that said something along the lines of: "For the mobile version, click here".

The "mobile version" of all those maps and charts are actually not mobile versions at all, but rather the exact same chart, just in a different spot. This was because of issues we had with embed codes not working on our mobile site. The result, though, is that if you click on the "mobile version" of a chart, you'll be taken to a separate website where you can usually find the missing chart (but not the maps; more on that in a moment).

OK, now for the part that's my fault.

Without getting too much into the technical weeds, many of the maps I made while at The Sun were built using Google Fusion Tables. And in order to add extra features to those maps, like colour legends and search boxes, I needed to do a bit of basic HTML. That raised the problem of where to host those HTML files.

For awhile, I hosted them on Postmedia's servers. But uploading files to those servers was a huge pain, requiring me to fill in IT request forms and other nonsense (not great on deadline) and at least once someone in IT, not knowing what the files were for, decided to go ahead and delete them.

What I should have done at that point is gotten some cheap hosting space on Amazon Web Services: something reliable that I knew would be there for the long term.

Instead, I used a slightly hacky technique to host the files for free on Google Drive. I actually had data journalism colleagues at other papers warn me that this was a dumb idea. "What happens if Google stops allowing free HTML hosting?" they'd say.

Which is exactly what Google did last year.

And the result is that dozens of maps I created for The Sun in Fusion Tables are no longer accessible online.

I actually alerted The Sun to this problem last year and, to its credit, they were working with me to figure out a way to get at least some of the missing content onto a Postmedia server and back on The Sun's website. Unfortunately, the two folks I was working with most closely to fix the problem have since left the paper, too, and so things went into limbo.

The problem is that restoring all these broken links and embed codes is a tedious, time-consuming job and most of it involves old stories that most people never see.

In the end, I figured I'd see if there was a way I could revive the content myself without bugging anyone else at the paper.

In preparing this post, I thought the best thing to do was to upload the old HTML files to GitHub Pages (another free, but arguably more trustworthy hosting solution). But when I tried that I got some weird JavaScript errors and the maps didn't load properly.

That said, for reasons I don't fully understand, the HTML files still seem to work fine when loaded up on your own computer. So what I've done is taken the HTML files for several of the maps that are no longer on The Vancouver Sun site and put them in a single ZIP file called SunHTMLFiles.zip which you can download here.

Just open the ZIP file on your own computer and double click on any of the HTML files. The map should then open in your web browser (in most cases you'll still need an Internet connection as the map data is being pulled from a Fusion Table source online).

Most of the maps are just single HTML files and most of the HTML files names are pretty obvious so you should be able to find what you're looking for.

But there are a couple maps that are a bit more complex.

The Unsolved Homicide Map which accompanied a Sun series on the topic requires you to open the UnsolvedHomicide folder and then click on the index.html file. One thing that doesn't work with that map anymore is the photos of the victims (as they, too, were loaded from a Google Drive folder).

You need to follow the same process for one of the Auto Crime maps that accompanied a Sun series I wrote. Open the AutoCrime folder and then click on index.html. The other auto crime maps can be launched from the main directory.

I haven't had time to grab everything I ever made at The Sun so prioritized content that I know got a lot of traffic when it was first posted and which I think may still be interesting to folks today. My main source was looking through my old blog posts and digging into those that seemed to have interesting content.

To make things a bit easier, I've included below the titles of all the blog posts and articles for which I've added maps to the ZIP file, along with links to the original posts:
For what it's worth, I've since moved away from Google Fusion Tables for mapping and I don't bother teaching it to my students anymore.

Instead, if you've got data to map, I highly recommend Tableau Public, which now has robust support for spatial files like KML and SHP.

And if you're still having trouble finding something I did while at The Sun, I recommend you check out my Tableau Public profile. It contains more than 120 interactive charts I created both during my time at The Vancouver and since I left. My favourites are in the "My Tableau Portfolio" workbook  (the first link on the page).

Saturday, September 2, 2017

The case against tweetstorms

Credit: Wikimedia Commons

Proposed: Tweet storms are a bad way to communicate complex ideas. Blog posts are much better and easier to discover in future.

As an example, Kevin Milligan is doing great analysis on tax reforms but reading his feed means some arguments appear backwards or disjointed.

On the other hand, only nerds like me still use RSS, so perhaps tweet storms are the best way to reach a large audience?

I'm also curious: Why have tweetstorms become more popular for writers than blog posts? Even when arguments are really long? Are tweetstorms easier to dash off on a smartphone? Less work to write? Is there less expectation for writing to be polished? Are they more appropriate for "ideas in progress"?

Are tweet storms more likely to go viral than blog posts? I doubt it. Lots of links to articles go viral on Twitter.

My biggest concern about tweetstorms is they're not easily discoverable. For example, someone Googling CCPC reforms won't find Milligan's tweets.

Twitter is also less popular by far than Facebook. Tweetstorms bypass Facebook's audience (blog posts, meanwhile, can be shared on Facebook and Twitter).

That said, if tweet storms are a way to work out ideas for a later blog post/article, I'm less concerned.

My bigger worry is when people with great ideas share them only in a tweetstorm and never crystallize their ideas in an article or blog post. I think that both limits the audience for their ideas and makes those ideas harder to digest.

NOTE: This blog post was adapted from a tweetstorm about tweetstorms. Given the topic, I thought it was appropriate to adapt it into a blog post as well. The text above is almost identical except for cleaning up the language a bit and adding a conclusion that the tweetstorm lacked. This is also an experiment with writing shorter, less polished, blog posts as I think one reason some writers default to tweetstorms is because of the expectation they place on themselves when writing blog posts rather than tweets: both in terms of length and quality. Blogs should be a safe place to dash off rough ideas.

Kevin Milligan, whose tweetstorms on tax policy inspired by tweetstorm, wrote a thoughtful tweetstorm of his own on why sometimes he tweets rather than writing longer pieces.