Session 6: Evaluation & Reporting
Transcript
Robert Hornsby:
Welcome back. Today is the Strategic Communications Planning class focused on evaluation and reporting. I'm Robert Hornsby, the Associate Vice President of Internal Communications, and I'm joined by Cassandra Nathan, Associate Director. We're going to go through a PowerPoint today that will be about 20-25 minutes, and I'll stop along the way so that we can have discussion. And then we'll see how long we have for breakout, and then we'll come back together for just sort of a last bit roundup and send you on your way.
So thanks, you all, for showing up, and let's start off as we normally do: Go back to the PowerPoint for just a second, and then we'll continue talking.
[Screen share begins]
[First slide: title slide]
[Second slide: Review of session 5—Metrics]
Cassandra, would you lead us through this part?
Cassandra Nathan:
Sure. So you guys, we’re going to ask you about how you felt metrics went for you. So would anybody like to share the easiest piece?
[Screen share stops]
Or a difficult part from metrics that you might have experienced?
I will pick on Katie. I'm not even gonna wait [laughs slightly].
Katie McCluskey:
So I thought metrics for the most part was pretty easy. The one obstacle that I ran into was sort of a software shortcoming. So for those who I haven't spoken to, my plan is based around communications for Columbia undergraduate housing. And students repeatedly tell us that their preferred method of communication is email, but the software that we use to communicate with students is called StarRez. It doesn't automatically track certain metrics unless you do certain plugins or you pay for upgrades.
And so that kind of threw a wrench in how we can evaluate, like, whether or not students are actually using this as their preferred method and whether or not we're actually contacting them through that. So I think it's probably going to need some internal discussions about what's a workaround and is it worthwhile to maybe pursue some of those plugins or adaptations that might let us collect that data.
Robert:
Good. Well, if this course leads to any kind of improvement of your day-to-day work life, then that's golden.
Who had trouble with metrics?
No one. Okay. Well, you know, you guys are cruising on. And I think particularly that the work that you all have done with the planning matrix—that had, you know, the goals and then your strategy and tactics—that helped a lot of you, I think, separate out what's what. And that's the hardest part of this class—is trying to come up with a common language and a common understanding about what's the difference between goals, strategy, and tactics. You are now joining a hundred other people who have taken this course.
On an institutional side for our office—Cassandra and I were talking about this before class started—I mean, our main institutional goal for the communications department is to break down the silos between schools, departments, institutes. So one of the ways we attack those silos is by having this course so that we can share a common language about goals, strategy, and tactics so that when you're stuck with something, you have someone to turn to. They know what you're talking about. If you start a new project that involves two or three other people or two or three other schools—and that's going to happen more and more—you can have a common effort that it doesn't run into sort of the immediate roadblocks of, you know, arguing about what is a tactic and what isn’t a tactic.
So let's move on. The quick refresher, you all know.
[Screen share begins]
[Second slide: Review of session 5—Metrics]
Today is the last class of this series.
[Third slide: Evaluation & Reporting—Summary of Key Terms]
We're going to talk about evaluation, reporting. So here's a summary of the key terms that we'll cover today: interpretation, data, effectiveness, alignment, meaning, hierarchical value, learnings, reporting types, and timing.
[Fourth slide: Evaluation and Reporting]
So let's start off with some definitions. Evaluation is the final step of your plan. It's where you interpret metrics and judge your success. So when I'm talking about judging success, you're really evaluating performance. So how did your tactics perform? Did they serve your strategy? Did they help you reach your goals?
You can also evaluate your strategy itself, not just your tactics. Your metrics are the data for evaluation, but they are also subject to your interpretation. So you might have some data that on the face of it looks one way, but then you do some analysis and you realize “Oh, it doesn't really say what I think it says,” or there's something more to it.
So you need to be able to interpret your data. And that helps you talk to your supervisor and other people. As much as possible, you need to show fairly and objectively what was effective.
So let's pause there for a minute.
[Screen share stops]
So is any of that a mystery or needs explanation? Have any of you done kind of metrics reporting to your bosses or your supervisors? And what form did that take?
Katie, you're nodding. What do you do?
Katie:
We do a couple different things. So we've done Google Analytics dashboard reporting. We also sort of—pre-pandemic when things were a little bit more normal—we would do monthly metric sheets that included metrics from social media, metrics from web traffic. And usually we would, like, pick a specific project to focus on, and we'd sort of communicate all the different tactics that went to publicize that and the metrics that supported it. So that might also include like email metrics or like event metrics if that's what it was related to.
Robert:
So did you provide a filter or some sort of analysis of the metrics, or you just give them the raw data?
Katie:
Probably both. Some of it was just raw data. The featured projects tended to be a little bit more analysis and sort of like giving an estimate as to why things might be– Why we might have seen more traffic here or things like that.
Robert:
So Jamie, you raised your hand. How does your reporting compare to Katie's?
Jamie Nash:
I was actually just gonna say that we're on the same team, so we had the same kind of metric sheets. So you just called on her, but I would have said the same exact thing. [Laughs]
Robert:
Okay. So you and Katie and Kristina are covered. Does anybody else have a kind of report that you do about your programs in communications?
Go ahead, Maddie.
Maddie Henry:
Yeah, so we pull together a yearly press report. We feature some of our really big media hits, but then also we list everything that we've pitched throughout the year, and then we do a whole section on, like, metrics reporting where we detail, like, throughout the year when we've gotten the most clips and when we've gotten the highest publicity value and things like that. And how we compare against our competitors as well.
Robert:
Okay. That sounds all good. Go back to the PowerPoint for a second.
[Screen share begins]
[Fourth slide: More about Evaluation & Reporting]
So a little bit more about this. So in your reporting section, you have the opportunity to demonstrate alignment between the plan and the overall communications goals, institutional goals. So evaluation and reporting give you opportunity to talk about three things: the meaning of your metrics, a hierarchy of value, and the learnings. So I'm going to unpack each of those one by one.
So when I say “meaning,” I'm talking about context. So the things that you do exist in a world of people, institutions, and other forces, and we talked about this: what communications can accomplish and what it can't accomplish. You may not be able to overcome a financial hurdle with a press release.
And, you know, your success or failure can be affected by and is relative to such factors, and it's these factors that supply context. So in your reporting, you have an opportunity to explain context. Like COVID has affected the number of people that can come to events, so you had to pivot to online or Zoom events. And you may have seen greater enrollment or participation, or you might have seen less participation. Or that participation might be more superficial or more in depth, but it's just not the same.
[Fifth slide: Learning from Metrics]
So moving on. Learnings from metrics. So a couple of things you can learn: How successful are key messages? Are they being acted on? What's your engagement like? And we talked about this at last class. Is it up, down, the same? What tactics or combination of tactics could account for good or worse engagement? And then the tactics and platforms themselves: Do your metrics indicate that one activity is doing better and is closer to accomplishing your goals? And then you have an opportunity to provide some analysis. What does that mean about future action?
[Screen share stops]
So I'm going to stop share. In the reporting that you've done, or just in conversations, have you decided to change course at some point? You know, set out to do a series of tactics or a strategy and then change your mind, or your boss changed your mind. Has that happened to any of you?
Robert’s nodding. Tell us about that, Robert.
Robert Tulman:
Sure. So we at the Medical Center, at VP&S, we have the Velocity Bike Ride—very metric-based in terms of determining which communication routes and social routes. So one of the things that was a major concern for us were how many unaffiliated users or audience members are we pulling in. People who are really not familiar with Columbia, who are not familiar with the Medical Center. We had put money into display ads for Google, just regular tracking ads, things like that. And what we found is that, you know, they brought in page views, but there were no conversions. There were no subsidies.
Robert:
Okay, that’s what we’re talking about.
Robert T.:
Yeah, exactly. So we pulled away from that. Invested more money in sort of in-house targeting the community and geo-targeting and new exciting things that may or may not work, but we've gone away from what we know did not work.
Robert:
Right. And that's a sort of a evidence-based decision making, which metrics can help you do if you're thinking about it. If you take the time to, you know, to pause and say, “Okay, is this money worth spending again? What do we get out of it? Did this really fulfill our goals?” And if it didn't, you do something else. And you need to be able to make a case with that, and how you make the case is using the data of your metrics.
And you can also use, you know, qualitative or emotional arguments or, you know, how much time it's taking to do something doesn't offset the benefit of it. We could do the same with redirecting our money into something else or we could put in less time and do it in a different way and achieve the same results. So there's other options about this, but this shows you're thinking on your feet, and you're able to adjust the program and you're not just sort of slavishly going through a series of tactics that may or may not be working. This is your opportunity to fix things.
Okay. Anybody else have an example they want to offer up? Okay. We're going to move on.
[Screen share begins]
[Sixth slide: A Hierarchy of Metrics and Discussion Questions]
So hierarchy of metrics. So not all metrics are equivalent. Some can be more valued or less valued, and in your reporting you have an opportunity to, again, provide context. You know, page views aren't as important in the example that Robert just gave. Page views are great, but what you really want is conversions. And if you don't have conversions, then page views are meaningless or not very valuable.
So here's a couple of discussion questions, and you can take these out into your plans and into the breakout. Does any given metric matter more? And could that be for one audience or another? So maybe page views among 18 to 34 year olds really matters, but for people over 65 it doesn't really matter. Or does any tactic have more than one metric? So think about activities that you're pursuing where you could measure it in different ways.
Maybe you have a quantitative measure and a qualitative measure. For example, you could report on the number of people that are seeing a news story, and then you could poll them and find out that 90% of them don't understand it, or 50% of them think it's BS, or, you know, 50% of them think you're a communist. So those are two different types of metrics that require different interpretations.
Also in the hierarchy of metrics, how would it be best to explain that in your report? So do you do a breakout section for your page views and then compare them to your qualitative information? You know, you can present it in a chart. You can present it in a side-by-side comparison. But you have to think about what's the best way to show that in a report.
[Seventh slide: A Hierarchy of Metrics]
So here's an exercise, and you can do this in breakout if you feel like it. On a scale of one to ten, look at your metrics and then rank them by their importance. So for example, here are six different metric categories, and you could decide for yourself within your school, your department, or your unit, like, are some of these more important than others?
Are page views our number one thing because we're working on the web and we just want to show web traffic? How are we doing on newsletters? Are newsletters more valuable? Are those people more engaged than people that are just looking at your website in terms of page views? How are your likes and shares on social media? Is that more important than people coming to a conference? How many people are seeing your news articles? Is that more or less important than people that are signing up for newsletters?
And you'll see sometimes that these tactical things connect to one another. So you may find that news articles lead to newsletter sign ups. For example, if you have an RSS feed and you're using a link from your home page to the RSS feed, your page views of the home page come before people link to the RSS feed. Okay? They got to see one before the other, so one leads to the other. So you can show the connections between things. But also not everything's equal.
So I'm going to stop share for a minute.
[Screen share stops]
Does anybody already have a hierarchy of metrics in their plan? I'd like to hear from two people. What's your top metric, and why is it your top metric?
Come on, someone volunteer. I'm going to call on you. Go ahead, Kristina.
Kristina Hernandez:
I was referring back to my plan to see which I would say would be the top. And it's hard to say. I mean, I think for mine—which is a meal delivery service for undergraduate resident students during quarantine—so it's a really tricky one because there are a lot of obstacles that we've been dealing with that have made it hard to understand. Like is it the audience? Is it the service? Is it the communications?
And so I think one of the things we talked about in the metrics conversation last week is the number of orders missed, so it's almost like, you know, the absence of something. But that's been one of our big problems is that the students have had a hard time getting their orders in at all. And so even though I have a lot of communications tactics that have metrics like, you know, open rate or clicks or views, I think actually for this particular project, that might be one that's, like, on the top of the hierarchy for me.
Robert:
It's sort of referring back to Robert's example about conversions. So orders launched, orders fulfilled. I mean, that's a very common metric for anything, any kind of, you know, I mean Seamless, Grubhub, or any of those. They track how many people abandon their carts, and did they complete the circle. And that's really what you're talking about. Between your outbound communications and then your incoming, how do those match up?
Kristina:
Right.
Robert:
In the world of fundraising and in admissions and other, you know, sort of yield versus broadcast if you want to put it that way in communications terms, this is where everyone lives. That's not a bad hierarchy at all to be connected back to, like, a real institutional goal of delivering more meals. And what can communications say about that, or what can communications do about that?
Does anybody else have an example of a top metric? Ramona, what are you studying or what are you looking at in terms of, like, metric value?
Ramona King:
Yeah, I was just thinking about that.
Robert:
You're totally in a student-facing role, which is why I called on you.
Ramona:
Yeah, and I think this is, like, part of our sort of ongoing conversations and where we need to get some clarity. I would say currently, like, the thing that we pay the most attention to are the social media metrics—follows and unfollows.
Robert:
And why is that?
Ramona:
In particular? Partially because we don't have currently great email metrics, and partially because social media is, like, where our team sort of puts a lot of energy.
Robert:
Sure.
Ramona:
Because that's where the students are.
Robert:
Right. And see, that's a good point. So where is your most important audience? And how do you communicate with them? So your metrics and your evaluation sort of lives in that space. That's not to say that you ignore other things, but the concept of hierarchy of metrics helps you determine. And you'll be able to tell your boss, like, “We're really focused on web analytics because of x because we're looking for conversions.” Or “We're really focused on social media metrics because that's where the primary communications happens between our unit and our audience.”
It doesn't have to be some, you know, big deal philosophical position. It's just being able to give some grounding and sort of explain why when you're presenting a report about what you've been doing.
So just a little bit more, and then we'll go to breakout.
[Screen share begins]
[Eighth slide: Common Reporting types]
So I want to talk about a couple of common reporting types. So those of you that work in the media world, you know, understand a coverage report, which is a roundup of citations. We generate a daily media coverage report, and then we compile that over the course of the year. And, you know, it's not an exhaustive compilation of everything, but at the end of 52 weeks of this, five days a week, we have a pretty good idea of what are our top stories and what are the things have been most popular. We can see where we've put in most of our time, for example.
You can do an analysis report, and you can look at statistics across multiple platforms. You can look at reach in terms of you know website traffic—you know, blogs, social media, video. How big are the audiences? How long are they staying on various platforms? You know, are they watching the video to the end or only watching the first 30 seconds? That kind of thing.
And then you can do a comprehensive report, which is, you know, to pair a coverage report with analysis, and then also look at goals, strategy, and tactics for a given project, like a theme or a campaign.
So I've worked on these kind of projects really a lot over the time that I've been at Columbia. So for WATERLICHT, the big art project that was in front of Lenfest for a series of nights in 2019 before COVID. You know, we had attendance goals, we had media goals, we had social media goals, and we had different strategies to fulfill each one of those. And then we had to capture metrics on each of those things: How many people looked at the website, how many people registered to go to the event, how long they waited to get to the event, how many press people came, how much news coverage. So we had multiple metrics and reporting that hold it all together.
[Ninth slide: Report styles to consider]
So then there's a couple of things to consider in terms of style. You know a narrative report is fine. It gets too long, it's hard for people to follow. Outline with bullets seems to be the thing that most people default to. A general news report is often just, you know, the headline, the author, the first few lines of a citation. You know, we don't put all of an article into a report because each daily thing would be 47 pages long.
Your social media reports come in different styles. Charts and graphics can help if you can break things out so they're not just raw data, You know, not just numbers in a spreadsheet. And some combination of the above is often the best thing to do.
[Tenth slide: Report Timing and Discussion Questions]
So last bit I wanted to cover before breakout is timing. Like everything else, reporting needs a timeline. So here's some discussion questions for your plan. When will various metrics yield something of value? Like is quarterly good enough? Do you need to do it more often than quarterly? When would reporting to your boss be helpful? We all have annual performance reviews, so it wouldn't be a bad thing for you to do a bit of analysis and reporting on your year's progress before you have your annual review with your boss.
And this might be the opportunity or the only opportunity that you have to step back and say, “What have we been doing? Has it been working? What have I been doing? Has that been working? Would I like to be doing something else?”
And there can be other inflection points, like a calendar year is often a thing. You know, year in review. Maddie talked about that. We used to do a 100 top hits for a year in review. You know, January to December, and we present that to President Bollinger every year just for his edification. Your fiscal year, you know, is a financial thing, and so if you have a fundraising goal or something and it's going to fall into those kind of worlds. And of course sure, before your annual review.
[Eleventh slide: Evaluation & Reporting—Additional Discussion Questions]
Here's the last bit of discussion questions. Did your metrics reveal something unexpected, expected? Did your metrics align with your goals? What about the institutional operational goals? What's the best form that could be understood or impressive to your supervisor? What elements are best to include? How does your evaluation suggest changes to the communications plan?
And this is something that can be very powerful at the end of a plan, saying, “At the end of six months, we're going to evaluate our plan. We're going to evaluate our metrics, and, you know, change course as we need to.” And if that's all you say at the end of your plan, good enough.
[Twelfth slide: Breakout—General Discussion]
[Screen share stops]
So I'm going to stop share. And we have nine people plus Cassandra and I. You want to break out into groups of three or groups of four? Three? Three groups of three? Okay. Cassandra, let's do three groups of three. You're going to get mixed up in different groups, but we'll give you about 20, 25 minutes. Okay?
[Time jump in the video]
Okay. Well, I hope your breakout sessions went well. We're winding down, and I just wanted to show you a couple of things and remind you of the Google folders and the LabArchives folders which contain samples of different kinds of reporting as well as the PowerPoints from today and other days and videos from today and other days. I'm going to go back to the share screen for just a second, and then we'll close up.
[Share screen begins]
[Twelfth slide: Breakout—General Discussion]
[Thirteenth slide: Finals Week]
So this is finals week. I covered some of this. Your final plans are due next Thursday. As I said to my breakout group, if you only tell me what you're going to do down the road, I don't need to see a sample of your reporting. You need to tell me, “We're going to evaluate our metrics quarterly and issue a simple report quarterly, and then we're going to do an annual report.” Good enough. Give me some sense of how you're evaluating what you've been doing. Don't labor over it too much.
Cassandra can conduct a makeup class next Friday if you want one, or if you missed a class and you want to go over something, or you just want to talk about plans. We can do that next Friday, but we need to hear from you. So send Cassandra a note saying, “Yes I'd like to join with the class.” And it'll basically be like an hour-long breakout session where you can talk to each other.
And then Cassandra will follow up with a course evaluation. If so, we would be very appreciative if you would fill out the survey and give us some feedback about how it's gone for you.
[Fourteenth slide: Thank you for being a great class!]
And thank you for being a great class. So Cassandra, do you have some final words?
Cassandra:
I think you guys did amazing. And as you can see, you literally blink and it's all over. And the first few weeks can be pretty challenging, but very, very proud of you guys for making it through. I think once you make it past the third session and you get past the overall, you kind of see that, “Okay, there's definitely a bell curve to this, and I'm finally descending.”
[Screen share stops]
“Thank God. We're coming down the hill.”
But I do want to encourage you guys: There can be the temptation to get to metrics and not do the evaluation portion. I want to encourage you and press upon you to get past that last hump, that silent hump where there's no one around. This is you and your paper. This is for you and your unit. This is for the future of your department, so think big, and just commit to yourself that you're going to complete this all the way to the finish line. It won't be perfect. We're not looking for perfection; we're looking for progress.
We're happy to review your paper once it's all– Well, we're going to review everyone's paper, but if you want to add to it, make an addendum, Robert and I will continue to be your allies and we’ll continue to be in your corner. So this is not just a one-stop shop, and I wish you guys all the best.
Robert:
Thank you all. Take care.
Unidentified attendee #1:
Thank you.
Brandon Alexander:
Thank you.
Ramona:
This has been great, thank you.
Unidentified attendee #2:
Thank you so much.
Unidentified attendee #3:
Thank you.
Welcome back. Today is the Strategic Communications Planning class focused on evaluation and reporting. I'm Robert Hornsby, the Associate Vice President of Internal Communications, and I'm joined by Cassandra Nathan, Associate Director. We're going to go through a PowerPoint today that will be about 20-25 minutes, and I'll stop along the way so that we can have discussion. And then we'll see how long we have for breakout, and then we'll come back together for just sort of a last bit roundup and send you on your way.
So thanks, you all, for showing up, and let's start off as we normally do: Go back to the PowerPoint for just a second, and then we'll continue talking.
[Screen share begins]
[First slide: title slide]
[Second slide: Review of session 5—Metrics]
Cassandra, would you lead us through this part?
Cassandra Nathan:
Sure. So you guys, we’re going to ask you about how you felt metrics went for you. So would anybody like to share the easiest piece?
[Screen share stops]
Or a difficult part from metrics that you might have experienced?
I will pick on Katie. I'm not even gonna wait [laughs slightly].
Katie McCluskey:
So I thought metrics for the most part was pretty easy. The one obstacle that I ran into was sort of a software shortcoming. So for those who I haven't spoken to, my plan is based around communications for Columbia undergraduate housing. And students repeatedly tell us that their preferred method of communication is email, but the software that we use to communicate with students is called StarRez. It doesn't automatically track certain metrics unless you do certain plugins or you pay for upgrades.
And so that kind of threw a wrench in how we can evaluate, like, whether or not students are actually using this as their preferred method and whether or not we're actually contacting them through that. So I think it's probably going to need some internal discussions about what's a workaround and is it worthwhile to maybe pursue some of those plugins or adaptations that might let us collect that data.
Robert:
Good. Well, if this course leads to any kind of improvement of your day-to-day work life, then that's golden.
Who had trouble with metrics?
No one. Okay. Well, you know, you guys are cruising on. And I think particularly that the work that you all have done with the planning matrix—that had, you know, the goals and then your strategy and tactics—that helped a lot of you, I think, separate out what's what. And that's the hardest part of this class—is trying to come up with a common language and a common understanding about what's the difference between goals, strategy, and tactics. You are now joining a hundred other people who have taken this course.
On an institutional side for our office—Cassandra and I were talking about this before class started—I mean, our main institutional goal for the communications department is to break down the silos between schools, departments, institutes. So one of the ways we attack those silos is by having this course so that we can share a common language about goals, strategy, and tactics so that when you're stuck with something, you have someone to turn to. They know what you're talking about. If you start a new project that involves two or three other people or two or three other schools—and that's going to happen more and more—you can have a common effort that it doesn't run into sort of the immediate roadblocks of, you know, arguing about what is a tactic and what isn’t a tactic.
So let's move on. The quick refresher, you all know.
[Screen share begins]
[Second slide: Review of session 5—Metrics]
Today is the last class of this series.
[Third slide: Evaluation & Reporting—Summary of Key Terms]
We're going to talk about evaluation, reporting. So here's a summary of the key terms that we'll cover today: interpretation, data, effectiveness, alignment, meaning, hierarchical value, learnings, reporting types, and timing.
[Fourth slide: Evaluation and Reporting]
So let's start off with some definitions. Evaluation is the final step of your plan. It's where you interpret metrics and judge your success. So when I'm talking about judging success, you're really evaluating performance. So how did your tactics perform? Did they serve your strategy? Did they help you reach your goals?
You can also evaluate your strategy itself, not just your tactics. Your metrics are the data for evaluation, but they are also subject to your interpretation. So you might have some data that on the face of it looks one way, but then you do some analysis and you realize “Oh, it doesn't really say what I think it says,” or there's something more to it.
So you need to be able to interpret your data. And that helps you talk to your supervisor and other people. As much as possible, you need to show fairly and objectively what was effective.
So let's pause there for a minute.
[Screen share stops]
So is any of that a mystery or needs explanation? Have any of you done kind of metrics reporting to your bosses or your supervisors? And what form did that take?
Katie, you're nodding. What do you do?
Katie:
We do a couple different things. So we've done Google Analytics dashboard reporting. We also sort of—pre-pandemic when things were a little bit more normal—we would do monthly metric sheets that included metrics from social media, metrics from web traffic. And usually we would, like, pick a specific project to focus on, and we'd sort of communicate all the different tactics that went to publicize that and the metrics that supported it. So that might also include like email metrics or like event metrics if that's what it was related to.
Robert:
So did you provide a filter or some sort of analysis of the metrics, or you just give them the raw data?
Katie:
Probably both. Some of it was just raw data. The featured projects tended to be a little bit more analysis and sort of like giving an estimate as to why things might be– Why we might have seen more traffic here or things like that.
Robert:
So Jamie, you raised your hand. How does your reporting compare to Katie's?
Jamie Nash:
I was actually just gonna say that we're on the same team, so we had the same kind of metric sheets. So you just called on her, but I would have said the same exact thing. [Laughs]
Robert:
Okay. So you and Katie and Kristina are covered. Does anybody else have a kind of report that you do about your programs in communications?
Go ahead, Maddie.
Maddie Henry:
Yeah, so we pull together a yearly press report. We feature some of our really big media hits, but then also we list everything that we've pitched throughout the year, and then we do a whole section on, like, metrics reporting where we detail, like, throughout the year when we've gotten the most clips and when we've gotten the highest publicity value and things like that. And how we compare against our competitors as well.
Robert:
Okay. That sounds all good. Go back to the PowerPoint for a second.
[Screen share begins]
[Fourth slide: More about Evaluation & Reporting]
So a little bit more about this. So in your reporting section, you have the opportunity to demonstrate alignment between the plan and the overall communications goals, institutional goals. So evaluation and reporting give you opportunity to talk about three things: the meaning of your metrics, a hierarchy of value, and the learnings. So I'm going to unpack each of those one by one.
So when I say “meaning,” I'm talking about context. So the things that you do exist in a world of people, institutions, and other forces, and we talked about this: what communications can accomplish and what it can't accomplish. You may not be able to overcome a financial hurdle with a press release.
And, you know, your success or failure can be affected by and is relative to such factors, and it's these factors that supply context. So in your reporting, you have an opportunity to explain context. Like COVID has affected the number of people that can come to events, so you had to pivot to online or Zoom events. And you may have seen greater enrollment or participation, or you might have seen less participation. Or that participation might be more superficial or more in depth, but it's just not the same.
[Fifth slide: Learning from Metrics]
So moving on. Learnings from metrics. So a couple of things you can learn: How successful are key messages? Are they being acted on? What's your engagement like? And we talked about this at last class. Is it up, down, the same? What tactics or combination of tactics could account for good or worse engagement? And then the tactics and platforms themselves: Do your metrics indicate that one activity is doing better and is closer to accomplishing your goals? And then you have an opportunity to provide some analysis. What does that mean about future action?
[Screen share stops]
So I'm going to stop share. In the reporting that you've done, or just in conversations, have you decided to change course at some point? You know, set out to do a series of tactics or a strategy and then change your mind, or your boss changed your mind. Has that happened to any of you?
Robert’s nodding. Tell us about that, Robert.
Robert Tulman:
Sure. So we at the Medical Center, at VP&S, we have the Velocity Bike Ride—very metric-based in terms of determining which communication routes and social routes. So one of the things that was a major concern for us were how many unaffiliated users or audience members are we pulling in. People who are really not familiar with Columbia, who are not familiar with the Medical Center. We had put money into display ads for Google, just regular tracking ads, things like that. And what we found is that, you know, they brought in page views, but there were no conversions. There were no subsidies.
Robert:
Okay, that’s what we’re talking about.
Robert T.:
Yeah, exactly. So we pulled away from that. Invested more money in sort of in-house targeting the community and geo-targeting and new exciting things that may or may not work, but we've gone away from what we know did not work.
Robert:
Right. And that's a sort of a evidence-based decision making, which metrics can help you do if you're thinking about it. If you take the time to, you know, to pause and say, “Okay, is this money worth spending again? What do we get out of it? Did this really fulfill our goals?” And if it didn't, you do something else. And you need to be able to make a case with that, and how you make the case is using the data of your metrics.
And you can also use, you know, qualitative or emotional arguments or, you know, how much time it's taking to do something doesn't offset the benefit of it. We could do the same with redirecting our money into something else or we could put in less time and do it in a different way and achieve the same results. So there's other options about this, but this shows you're thinking on your feet, and you're able to adjust the program and you're not just sort of slavishly going through a series of tactics that may or may not be working. This is your opportunity to fix things.
Okay. Anybody else have an example they want to offer up? Okay. We're going to move on.
[Screen share begins]
[Sixth slide: A Hierarchy of Metrics and Discussion Questions]
So hierarchy of metrics. So not all metrics are equivalent. Some can be more valued or less valued, and in your reporting you have an opportunity to, again, provide context. You know, page views aren't as important in the example that Robert just gave. Page views are great, but what you really want is conversions. And if you don't have conversions, then page views are meaningless or not very valuable.
So here's a couple of discussion questions, and you can take these out into your plans and into the breakout. Does any given metric matter more? And could that be for one audience or another? So maybe page views among 18 to 34 year olds really matters, but for people over 65 it doesn't really matter. Or does any tactic have more than one metric? So think about activities that you're pursuing where you could measure it in different ways.
Maybe you have a quantitative measure and a qualitative measure. For example, you could report on the number of people that are seeing a news story, and then you could poll them and find out that 90% of them don't understand it, or 50% of them think it's BS, or, you know, 50% of them think you're a communist. So those are two different types of metrics that require different interpretations.
Also in the hierarchy of metrics, how would it be best to explain that in your report? So do you do a breakout section for your page views and then compare them to your qualitative information? You know, you can present it in a chart. You can present it in a side-by-side comparison. But you have to think about what's the best way to show that in a report.
[Seventh slide: A Hierarchy of Metrics]
So here's an exercise, and you can do this in breakout if you feel like it. On a scale of one to ten, look at your metrics and then rank them by their importance. So for example, here are six different metric categories, and you could decide for yourself within your school, your department, or your unit, like, are some of these more important than others?
Are page views our number one thing because we're working on the web and we just want to show web traffic? How are we doing on newsletters? Are newsletters more valuable? Are those people more engaged than people that are just looking at your website in terms of page views? How are your likes and shares on social media? Is that more important than people coming to a conference? How many people are seeing your news articles? Is that more or less important than people that are signing up for newsletters?
And you'll see sometimes that these tactical things connect to one another. So you may find that news articles lead to newsletter sign ups. For example, if you have an RSS feed and you're using a link from your home page to the RSS feed, your page views of the home page come before people link to the RSS feed. Okay? They got to see one before the other, so one leads to the other. So you can show the connections between things. But also not everything's equal.
So I'm going to stop share for a minute.
[Screen share stops]
Does anybody already have a hierarchy of metrics in their plan? I'd like to hear from two people. What's your top metric, and why is it your top metric?
Come on, someone volunteer. I'm going to call on you. Go ahead, Kristina.
Kristina Hernandez:
I was referring back to my plan to see which I would say would be the top. And it's hard to say. I mean, I think for mine—which is a meal delivery service for undergraduate resident students during quarantine—so it's a really tricky one because there are a lot of obstacles that we've been dealing with that have made it hard to understand. Like is it the audience? Is it the service? Is it the communications?
And so I think one of the things we talked about in the metrics conversation last week is the number of orders missed, so it's almost like, you know, the absence of something. But that's been one of our big problems is that the students have had a hard time getting their orders in at all. And so even though I have a lot of communications tactics that have metrics like, you know, open rate or clicks or views, I think actually for this particular project, that might be one that's, like, on the top of the hierarchy for me.
Robert:
It's sort of referring back to Robert's example about conversions. So orders launched, orders fulfilled. I mean, that's a very common metric for anything, any kind of, you know, I mean Seamless, Grubhub, or any of those. They track how many people abandon their carts, and did they complete the circle. And that's really what you're talking about. Between your outbound communications and then your incoming, how do those match up?
Kristina:
Right.
Robert:
In the world of fundraising and in admissions and other, you know, sort of yield versus broadcast if you want to put it that way in communications terms, this is where everyone lives. That's not a bad hierarchy at all to be connected back to, like, a real institutional goal of delivering more meals. And what can communications say about that, or what can communications do about that?
Does anybody else have an example of a top metric? Ramona, what are you studying or what are you looking at in terms of, like, metric value?
Ramona King:
Yeah, I was just thinking about that.
Robert:
You're totally in a student-facing role, which is why I called on you.
Ramona:
Yeah, and I think this is, like, part of our sort of ongoing conversations and where we need to get some clarity. I would say currently, like, the thing that we pay the most attention to are the social media metrics—follows and unfollows.
Robert:
And why is that?
Ramona:
In particular? Partially because we don't have currently great email metrics, and partially because social media is, like, where our team sort of puts a lot of energy.
Robert:
Sure.
Ramona:
Because that's where the students are.
Robert:
Right. And see, that's a good point. So where is your most important audience? And how do you communicate with them? So your metrics and your evaluation sort of lives in that space. That's not to say that you ignore other things, but the concept of hierarchy of metrics helps you determine. And you'll be able to tell your boss, like, “We're really focused on web analytics because of x because we're looking for conversions.” Or “We're really focused on social media metrics because that's where the primary communications happens between our unit and our audience.”
It doesn't have to be some, you know, big deal philosophical position. It's just being able to give some grounding and sort of explain why when you're presenting a report about what you've been doing.
So just a little bit more, and then we'll go to breakout.
[Screen share begins]
[Eighth slide: Common Reporting types]
So I want to talk about a couple of common reporting types. So those of you that work in the media world, you know, understand a coverage report, which is a roundup of citations. We generate a daily media coverage report, and then we compile that over the course of the year. And, you know, it's not an exhaustive compilation of everything, but at the end of 52 weeks of this, five days a week, we have a pretty good idea of what are our top stories and what are the things have been most popular. We can see where we've put in most of our time, for example.
You can do an analysis report, and you can look at statistics across multiple platforms. You can look at reach in terms of you know website traffic—you know, blogs, social media, video. How big are the audiences? How long are they staying on various platforms? You know, are they watching the video to the end or only watching the first 30 seconds? That kind of thing.
And then you can do a comprehensive report, which is, you know, to pair a coverage report with analysis, and then also look at goals, strategy, and tactics for a given project, like a theme or a campaign.
So I've worked on these kind of projects really a lot over the time that I've been at Columbia. So for WATERLICHT, the big art project that was in front of Lenfest for a series of nights in 2019 before COVID. You know, we had attendance goals, we had media goals, we had social media goals, and we had different strategies to fulfill each one of those. And then we had to capture metrics on each of those things: How many people looked at the website, how many people registered to go to the event, how long they waited to get to the event, how many press people came, how much news coverage. So we had multiple metrics and reporting that hold it all together.
[Ninth slide: Report styles to consider]
So then there's a couple of things to consider in terms of style. You know a narrative report is fine. It gets too long, it's hard for people to follow. Outline with bullets seems to be the thing that most people default to. A general news report is often just, you know, the headline, the author, the first few lines of a citation. You know, we don't put all of an article into a report because each daily thing would be 47 pages long.
Your social media reports come in different styles. Charts and graphics can help if you can break things out so they're not just raw data, You know, not just numbers in a spreadsheet. And some combination of the above is often the best thing to do.
[Tenth slide: Report Timing and Discussion Questions]
So last bit I wanted to cover before breakout is timing. Like everything else, reporting needs a timeline. So here's some discussion questions for your plan. When will various metrics yield something of value? Like is quarterly good enough? Do you need to do it more often than quarterly? When would reporting to your boss be helpful? We all have annual performance reviews, so it wouldn't be a bad thing for you to do a bit of analysis and reporting on your year's progress before you have your annual review with your boss.
And this might be the opportunity or the only opportunity that you have to step back and say, “What have we been doing? Has it been working? What have I been doing? Has that been working? Would I like to be doing something else?”
And there can be other inflection points, like a calendar year is often a thing. You know, year in review. Maddie talked about that. We used to do a 100 top hits for a year in review. You know, January to December, and we present that to President Bollinger every year just for his edification. Your fiscal year, you know, is a financial thing, and so if you have a fundraising goal or something and it's going to fall into those kind of worlds. And of course sure, before your annual review.
[Eleventh slide: Evaluation & Reporting—Additional Discussion Questions]
Here's the last bit of discussion questions. Did your metrics reveal something unexpected, expected? Did your metrics align with your goals? What about the institutional operational goals? What's the best form that could be understood or impressive to your supervisor? What elements are best to include? How does your evaluation suggest changes to the communications plan?
And this is something that can be very powerful at the end of a plan, saying, “At the end of six months, we're going to evaluate our plan. We're going to evaluate our metrics, and, you know, change course as we need to.” And if that's all you say at the end of your plan, good enough.
[Twelfth slide: Breakout—General Discussion]
[Screen share stops]
So I'm going to stop share. And we have nine people plus Cassandra and I. You want to break out into groups of three or groups of four? Three? Three groups of three? Okay. Cassandra, let's do three groups of three. You're going to get mixed up in different groups, but we'll give you about 20, 25 minutes. Okay?
[Time jump in the video]
Okay. Well, I hope your breakout sessions went well. We're winding down, and I just wanted to show you a couple of things and remind you of the Google folders and the LabArchives folders which contain samples of different kinds of reporting as well as the PowerPoints from today and other days and videos from today and other days. I'm going to go back to the share screen for just a second, and then we'll close up.
[Share screen begins]
[Twelfth slide: Breakout—General Discussion]
[Thirteenth slide: Finals Week]
So this is finals week. I covered some of this. Your final plans are due next Thursday. As I said to my breakout group, if you only tell me what you're going to do down the road, I don't need to see a sample of your reporting. You need to tell me, “We're going to evaluate our metrics quarterly and issue a simple report quarterly, and then we're going to do an annual report.” Good enough. Give me some sense of how you're evaluating what you've been doing. Don't labor over it too much.
Cassandra can conduct a makeup class next Friday if you want one, or if you missed a class and you want to go over something, or you just want to talk about plans. We can do that next Friday, but we need to hear from you. So send Cassandra a note saying, “Yes I'd like to join with the class.” And it'll basically be like an hour-long breakout session where you can talk to each other.
And then Cassandra will follow up with a course evaluation. If so, we would be very appreciative if you would fill out the survey and give us some feedback about how it's gone for you.
[Fourteenth slide: Thank you for being a great class!]
And thank you for being a great class. So Cassandra, do you have some final words?
Cassandra:
I think you guys did amazing. And as you can see, you literally blink and it's all over. And the first few weeks can be pretty challenging, but very, very proud of you guys for making it through. I think once you make it past the third session and you get past the overall, you kind of see that, “Okay, there's definitely a bell curve to this, and I'm finally descending.”
[Screen share stops]
“Thank God. We're coming down the hill.”
But I do want to encourage you guys: There can be the temptation to get to metrics and not do the evaluation portion. I want to encourage you and press upon you to get past that last hump, that silent hump where there's no one around. This is you and your paper. This is for you and your unit. This is for the future of your department, so think big, and just commit to yourself that you're going to complete this all the way to the finish line. It won't be perfect. We're not looking for perfection; we're looking for progress.
We're happy to review your paper once it's all– Well, we're going to review everyone's paper, but if you want to add to it, make an addendum, Robert and I will continue to be your allies and we’ll continue to be in your corner. So this is not just a one-stop shop, and I wish you guys all the best.
Robert:
Thank you all. Take care.
Unidentified attendee #1:
Thank you.
Brandon Alexander:
Thank you.
Ramona:
This has been great, thank you.
Unidentified attendee #2:
Thank you so much.
Unidentified attendee #3:
Thank you.