“How do I improve engagement with my customer education content?” This is one of the most common questions faced by Customer Training programs after they have launched their first coursework. This article will describe some important questions to ask to figure out why your learning content isn’t catching fire with your customers and define the associated KPIs to use as you enact improvements.
The Content Fire Triangle:
What is the Learning Content Fire Triangle?
You might recall from science class that a fire needs three things to survive: Fuel, Oxygen, and a Spark. They call this “the first triangle.” If you remove one of these three elements, you won’t have any fire. It won’t matter how much dry wood you have at your campsite, without a spark, you’ll have nothing to eat but cold beans.
Customer Education content works the same way. Creating an article or course alone doesn’t necessarily make it functional or useful. Often Instructional Designers and Training Program Managers will neglect one or more of the components of the Content Fire Triangle and then find that few (if any) learners engage with the content.
I’ll use an example use case that illustrates this point. I’m sure you’ve been in this situation at least once professionally:
You’re in a meeting. A challenge has been identified and someone announces that the challenge has already been solved in the past [insert a hint of snark in the person’s tone]. The person explains how this solution was built by their team and has been around for years. Despite the existence of the solution, regardless of whether it is useful or not, no one else in the room is aware of its existence.
In this case, the content never “caught fire” because no one knew it existed. Despite the work involved in creating the solution, it effectively didn’t exist, which means that it may well have not been created at all. That is the definition of wasted effort.
To make your content catch fire you need to ensure you have all three elements of the Content Fire Triangle:
Content must exist
Content must be discoverable
Content must be understandable & useful
Unless you can answer “yes” to all three of these conditions, your Customer Education engagement levels will likely be challenged.
Let’s look at these three aspects in detail, as well as the KPIs used to measure them.
1. Your learning content must exist
This is the most obvious part of the Content Fire Triangle and it is unfortunately where many program managers stop. Simply building training content is a great first step, but it isn’t sufficient on its own. This is similar to the example I used above where a resource is created but little or no thought is given on how to make it discoverable to the intended users.
You may be thinking: Okay, this article just got stupid. The author must think I am a pigeon.
First off, this site has been pigeon-free since 2019 (note the pokey things on all the window ledges).
Secondly, there is actually a practical answer to the question “does my learning content exist?” You may be surprised, but In many cases, the answer is actually “no.”
Here is a very important exercise for anyone managing a Knowledge Base or creating content for an LMS:
How to evaluate your knowledge content coverage
Open your company’s application
Click into every publicly accessible page in the application
Screenshot or print out every page
Number every unique feature or functional element on all the pages
Create a spreadsheet with four columns
Write all of these features & functions in the first column, each in their own cell
Write the index number next to the respective feature or function in the second column
Now open your knowledge base & review each article
In the third column of your spreadsheet, put a “1” next to every feature that has an article with specific instructions on how to use that feature or function
Now open your LMS or video library
In the fourth column, mark a “1” in the row associated with each feature or function if descriptive LMS or video content exists for that feature or function
What you’ll likely uncover is that very few of the features that customers potentially interact with have any educational content associated with them at all. For most companies, there just isn’t a lot of wood to throw on the campfire.
So let’s turn this into a KPI. There are a few versions:
Content coverage KPIs
% coverage
The first version is simply to SUM the number of 1’s in column 3, and then divide that by the total count of features in column 1. That is your “coverage percentage.” In a perfect world, you’d have 100% coverage of all features.
% coverage of valuable content
A variant on the first KPI is to add another column, name it high-value, and mark a “1” next to any feature that drives adoption or stickiness with customers (See this article about how to measure Customer Education). Now do the math to determine what percent of these high-value features have content associated with them.
2. Your content must be discoverable
The next part of the Content Fire Triangle is discoverability. The definition of “discoverable” is going to be slightly different depending on the audience, but it tends to be related to a few ways that people find your content:
Search
Browse
Promotion
Let’s start with the first two: Search & Browse.
Search & Browse
Search can mean either “findable when I search Google” or “shows up with I search your knowledge content.” A good example of the latter is when using a Knowledge Base like Zendesk. That platform has a very robust search function that displays results based on the title of the article, subheadings, and any tags you’ve used.
Browse in this case refers to when someone goes to your Knowledge Base or LMS and starts clicking around to find content. You’ve organized your content in such a way that a person will naturally understand where to look and as they dig into your folder structure, they eventually find exactly what they are looking for.
Both of these first two types of discovery are functions of your Information Architecture. That is, the way you have your content organized, titled, and tagged. These are general attributes of good information architecture:
Article & course titles are concise and descriptive
The top-level folder structure is simple & limited
Sub-folder levels are clearly aligned to feature areas or tasks
Little or no duplication of content
Tags or Labels are agreed upon and used consistently
If you don’t have a good Information Architecture, then people likely won’t be able to find your most valuable content. They might visit your Knowledge Base but not find anything of value, even though it may actually exist, for example.
A bad Information Architecture can negate the efforts you put into content creation. You could inadvertently starve your fire of oxygen by making it unable to be found.
Promotion
The most common culprit related to discoverability is a poorly designed Information Architecture. However, not planning any promotion is another related shortcoming that can inhibit people from finding & using your valuable content.
Customer Education should be seen by an organization as a product. Just like when launching a new feature, marketing should be involved in helping get the word out. Too often the training program manager operates as a one-person business, doing everything from content production to driving traffic.
A way to resolve this is by involving more executive stakeholders and planning learning content further in the future. This allows marketing to become involved and add learning content launches to their roadmap. Marketing can’t be an afterthought. Treating marketing as something that happens at the end of a process is a great way to ensure it never gets done.
Discoverability KPIs
To determine if your learning content is Discoverable, try using some of these KPIs:
Pageviews of your knowledge content
This is the most simplistic view of discoverability. The question you can answer with this metric is: Am I able to get more visits to my content. This is effectively asking “are my improvements in discoverability having a positive impact?”
Most content management systems provide page view or visitor analytics in their native reporting solutions. However, you will likely need to set up Google Analytics to get a deeper insight into the discoverability of your content.
Navigate to Google Analytics
Click on Behavior in the left nav
Click on Site Content in the sub nav
Select All Pages
Choose a time period of the last three months
Select “compare to prior period”
When you view this report, it might look like spaghetti if you don’t isolate some of the content first. Look at page views of your Knowledge Base homepage. This can be a good proxy for whether people have “discovered” your content and are returning. If you had a poor Information Architecture before, with improvement, you should see upwards of 200-300% increases year-over-year.
Page depth over time
A slightly more insightful version of the page views metric is page depth. This evaluates how many pages a person views when they visit your site. This is helpful for estimating how easy it is to browse for content. If you have low page depth, then it means either people are finding what they look for on the first try, or they aren’t finding anything and leaving.
Use Google Analytics to set up your page depth report:
Navigate to Google Analytics
Click on Acquisition in the left nav
Click on All Traffic in the sub nav
Select Channels
Choose a time period of the last three months
Select “compare to prior period”
When viewing this report, look at the “Pages / Session” column. At the top of the column will be an aggregate number. If this number goes up significantly over time, it means people are navigating around your site easily and looking at content.
% of successful searches
Another useful metric related to discoverability is to analyze your on-site search behaviors. You can do this for free with Google Analytics. I could go in-depth about this topic and use it regularly in my analysis, but this article does a great job of describing how to set this up & interpret the results: How to Analyze and Measure the Results of Your Site Search
The general idea is: See how often an on-site search results in someone viewing an article. The degree to which people are successful with using your search time to find content will help you understand both what they are searching for as well as how well you are addressing these queries.
If you are using Zendesk for your Knowledge Base, they also have some very insightful reporting as it relates to search query effectiveness. You can read more about that here.
Count of Level 1 Support Tickets over time
There is a direct relationship between people finding content and your level 1 support tickets. In fact, one of the most common goals for eLearning content is “ticket deflection.” This refers to any process or technology that reduces the inclination for a person to create a support ticket. For a more comprehensive description, take a look at this article: What is Ticket Deflection?
Level 1 support tickets are those which should have a self-serve answer. These often make up the bulk of support tickets and can be a real drain on resources. As a Training Manager, you can evaluate the discoverability of your content by measuring the aforementioned metric about on-site search queries and then going a level deeper by seeing how new content on a topic impacts the count of support tickets related to that topic over time.
3. Your content must be useful
The last part of the Content Fire Triangle relates to Utility. You might have the most complete piece of content ever created, optimized for on-site search, and pushed in front of every eyeball scrolling Linkedin … and if it is written for the wrong audience, it effectively has no utility. Existence and Discoverability alone don’t make content functional. Unless it is created with the end-user in mind, it might be undecipherable and therefore, a wasted investment in time & energy.
I’ll illustrate this with a common use case found in Software organizations:
The support engineers answer the same question about their API every week. They are experts in using the API and so they decide they will create a short video describing how to connect the API to customer’s software configurations. In their 2-minute video, they describe in precise accuracy how to complete a GET with the terminal using a test key and how to isolate JSON objects relevant to the query. They add this video to the homepage of the support site & it pops up on the API page in a special alert box. Despite the perfection they have created, the average view time of their video is only 5 seconds. They continue getting support tickets related to this issue.
So what is the issue? They clearly have two parts of the fire triangle completed:
They have created a piece of content for the common issue
They have put this piece of content in places where it is discoverable
Well, what I leave out of this example is that the users experiencing this issue are junior-level graphic designers and most of the customer base is in a region of the world where they speak a different language than the support engineers who created the video. For the most part, they don’t have any technical aptitude related to APIs.
In this example, the content has little to no utility for the learner. What I have described is a not uncommon experience when you have a technical product and the education program originates out of the product team.
Content Usefulness KPIs
There are two ways to evaluate the utility of content: Completion rates & Learner feedback.
Completion rates
When you place your content into an LMS, it helps to review in detail where people get stuck with your courses. You might have a great learning path, but if everyone stops 20% through, then everything past that point may as well not exist. To extend our fire analogy, just because you started the log on fire doesn’t mean it stayed lit.
You should expect that with new content, just like with new products, you’ll need to iterate. Too often learning management professionals focus on the first part of the fire triangle and think their job is done when they have published their coursework. It just ain’t so. It regularly takes 2-3 iterations to work out points in the coursework where people get stuck or disinterested. Maybe that 30-minute webinar recording isn’t very engaging after all and people drop, never to return.
Most LMS platforms will tell you on a lesson-by-lesson basis how far learners get into the content before they drop. Look for these spots and interrogate them fanatically. Expect you will need to iterate with that material to reduce the drop-off rates.
Learner feedback
The other great way to find out whether content is useful is to simply ask your audience. Many solutions have a thumbs up / thumbs down feature on articles or courses. Others allow you to put an NPS score that will display upon exiting a course. Consider setting these rating systems up from the beginning and start collecting feedback.
If you don’t have a platform that does this rating collection automatically, then make it a part of your Go-To-Market plan to collect feedback from your registered users for a few weeks after launching. Carve out time a few days a week and send users an email asking if they will meet to talk about the course they took. This interaction can be a goldmine for your instructional designers and really improve the utility of your courses.
Let the world burn
If you create learning content and no one can use it, then it was a waste of your time. As Customer Education professionals, we should be zealots about measuring what we do and showing the value of training in our organizations. It isn’t enough to simply build content and publish. We need to ensure our efforts catch sparks with our audiences.
If you can’t say Yes to these three statements, then you have not actually created learning content … you have just spent energy without creating any light:
My content exists
My content is discoverable to my users
My learners can understand and make use of my content