Metrics Monday 04: Don’t get the wrong impression(s)

What impressions are, what they aren’t, and how they can be misleading.

You may recall from the first post in this series about web analytics that engagement metrics help us understand how many people are visiting our website and what they’re doing while they’re there. If you’re a data analyst working with engagement metrics, your company seeks to learn more about its customers’ — and potential customers’ — behavior in order to create better user experiences and more effective marketing campaigns.

Web analytics aren’t just for organizations, though. Solopreneurs and content creators, like me, also want to understand the behavior of those individuals who interact with our content, product, or service for the same reasons as companies — we want to create better experiences for users and — hopefully — make a living doing it. 

To frame our discussion of impressions and to provide us some real — but not overwhelming — example data to work with, I created a visualization of my Twitter activity over the past 28 days. (If you’d like to view the interactive version, or just see a larger view, it’s on Tableau Public here.) 

Before I started building the charts, I knew I wanted to measure engagement. What does engagement mean on Twitter? (For clarity, I don’t mean their metric called engagements. Yet.) Likes, retweets, replies, and clicks on links to content come to my mind immediately. I also need to know how many tweets I sent (how many tweets were engaged with) in the time period I selected. I’d also like to know which tweets got the most engagement and what times of day people are interacting with my tweets so I can adjust my content scheduler accordingly.  

Twitter engagement dashboard. Interact with it on Tableau Public.

Twitter’s metric called engagements is defined as the “total number of times a user has interacted with a Tweet. This includes all clicks anywhere on the Tweet (including hashtags, links, avatar, username, and Tweet expansion), retweets, replies, follows, and likes.” The metric is expressed as a raw number and as a percentage. In the heatmap/marginal histogram above, I visualized the raw number of engagements. This is the largest chart in the view because engagements are what I am most concerned with — they account for most of what I said I wanted to measure (likes, retweets, replies, etc.) 

But what about impressions? you may be wondering, since that’s the topic I claim to be writing about today. They’re in my viz. Impressions are visualized at the bottom of the group of area charts to the right of the heatmap. Why so little real estate for impressions? When I open my Twitter Analytics page, it’s the first metric I’m presented with. Twitter obviously thinks they’re important! Right?

Screenshot of my Twitter Analytics Tweets page.

My tweets got more than 40,500 impressions in four weeks? WOW! Why don’t I have a blue check mark yet? 

That is the exact reaction Twitter is hoping for. It feels gratifying to see that large number. So gratifying that maybe I make an assumption about what it means, congratulate myself on a job well done, and call it a day. 

But what are impressions exactly? And why do I have so many more impressions than engagements, retweets, or likes?

As discussed in previous Metrics Mondays, companies don’t all define a metric the same way, despite calling it the same thing. Twitter’s definition of impressions is, “times a user is served a Tweet in timeline or search results.” It’s not explicitly stated — probably so I’ll maintain my enthusiasm about how many I got — but this definition of impressions includes when my tweet is in someone’s feed and they don’t even notice it, when the same person has seen my tweet 20 times (that’s 20 impressions), literally any time my tweet shows up on a user’s screen. Not a unique user’s screen. A user’s screen. So the same person may have seen and liked my tweet this morning, but saw it again 5 more times today because we run in the same data circles. That’s six impressions, not one. So we’ve defined the first problem with impressions: they can be interpreted as a gross inflation of the impact of your content.

While I was doing exploratory data analysis (EDA), I made an additional chart that I didn’t include in my final visualization. (I make a lot of charts I know I won’t use as I explore the data — that’s how I come to understand it.) The line chart below shows the general trend of my impressions (blue) and engagements (purple) by day. (My axes are synchronized and my data is pretty representative of a Twitter user of my activity level and number of followers as far as I can tell.)

To make the discussion of impressions a bit easier, let’s first define “user.” For the purposes of this discussion, a user is a visitor to your website (or viewer of my tweets in our example). 

Impressions are the number of times a web page element has been displayed to users. 

Often, the element is advertising content. In the case of Google Ads (which is a different product from Google Analytics, it should be noted), “an impression is counted each time your ad is shown on a search result page or other site on the Google Network.” Impressions do not account for whether the displayed content was clicked on or otherwise engaged with. Recall that a pageview is an instance of a page being loaded (or reloaded) in a browser. So impressions and pageviews are similar, but they’re not the same. 

So why even measure impressions?

Impressions are unreliable at best, so their frequent inclusion as key performance indicators (KPI) is puzzling. Avinash Kaushik counts impressions among “head-fake KPIs” which “barely qualify to be metric[s] because of the profoundly questionable measurement behind them.” They definitely qualify as vanity metrics when presented without any caveats or context regarding their meaning. Tableau defines vanity metrics as “metrics that make you look good to others but do not help you understand your own performance in a way that informs future strategies.” Their post on the topic goes on to point out that any metric can be a vanity metric if presented without robust analysis. 

🤖 Measures of impressions cannot account for bots, which make up about 40% of all web traffic. The United States and Australia are the biggest targets of “bad” bots.

Some analytics platforms have refined their measurement of impressions to try to make them more meaningful. For instance, Google Ads offers an additional data point — the position of the ad on the page. Platforms often count a partial view of the ad as an impression, so a user may not have actually seen the ad at all, so perhaps this is Google Ads’ way of addressing that issue. We’ve now identified the second big problem with impressions — they’re inaccurate, and we can’t say with precision just how inaccurate they are. 

I included the one little area chart showing impressions just to have a very, very general sense of what my reach might be and to see the relationships between impressions and other metrics visually. Hootsuite defines reach as “the total number of people who have seen your ad or content. If 100 total people have seen your ad, that means your ad’s reach is 100.” In other words, the number of unique users who have seen — not necessarily engaged with — the content. I purposely don’t include numeric values on the marks in the area charts in my viz because I’m just looking for trends. I’m making exactly zero decisions based on the associated numbers.

---

I’d love to hear your thoughts, especially if you’ve found a meaningful way to use impressions as an actionable metric.