Tools for analysing user and product data

As a product manager and all-around data nerd, analysing product and user data is something that brings me joy. This is convenient, as understanding users and analysing data are two of the biggest parts of my job. In this post, I share some of the tools and data sources I like best.

Product analytics

Useful for: quantitative data on user acquisition, engagement and retention

Product analytics tools give me access to quantitative data about where users are coming from, how they are accessing my tools, how they are using the product and how often they return. This enables me to answer questions like:

• how are users finding the product?
• how did that marketing campaign perform?
• are we increasing the percentage of new users who we convert into active users?
• at which stage in the process are most users dropping out?
• how do different segments of users behave on the site?
• how do users behave over time?

Google Analytics is the market leader in web analytics, and every digital product I have ever been involved in uses it. So long as it is free, I will continue to use it because it is the tool most people are most familiar with, and more people on the team having the ability to access analytics can only be a good thing.

More recently, I have primarily been using PostHog as my product analytics tool of choice. I like that it gives fine-grained control over event tracking, the ability to run SQL queries from within the product and powerful capabilities to build up my own analytics dashboards from the ground up.

Cookie-less product analytics

Useful for: understanding the total number of user sessions

The limitation with tools like Google Analytics is that they only work if users accept cookies. The percentage of users who click the accept button varies widely between websites, but it is fair to say that most people do not. This means that a large percentage of users are ignored by cookie-based analytics. This can be problematic if relying on these metrics to give an accurate view of total user numbers, or if users who accept cookies behave differently from those who reject.

To get around this, I like to supplement cookie-based tracking with a more limited set of analytics which works regardless of whether users accept cookies. The big limitation of cookie-less tracking is it does not allow you to identify user activity across multiple sessions. For each new session, you have no idea whether it is from a new or returning user. What you do know is the total number of sessions, and what the user is clicking on in each session. Combining cookie-based and cookie-less tracking gives a more complete view of user activity.

PostHog has support for cookie-less analytics tracking. I have heard that server-side tracking tools can also be useful here, but that is not something I have tried.

Session recording

Useful for: seeing how users are actually behaving on the product

Session recordings show anonymised recordings of how real users are using my products. They allow me to see where users type, click, scroll and move the mouse as they navigate. Session recording software is often complemented with heatmaps which show which parts of the page are getting looked at more and less.

Session recordings are most useful when a trend has already been identified by quantitative analytics, and I want to understand why something is happening. It can help me answer questions like:

• why aren’t many users getting to that stage in my funnel?
• do users understand how to use my new feature?
• are there parts of my product that are causing frustration?
• why is my product performing worse on certain devices?

There is no substitute for user interviews where you can watch how real users are using the application and ask them questions as they go. However, session recordings are a good compliment. In an hour, I can watch as many as 20 session recordings, so it is a great way of generating insights quickly. I can also easily share recordings in Slack channels or presentations to get the data in front of the team.

I like to use HotJar for session recording. It integrates well with Google Analytics and seems to be the market leader in this field. PostHog also have session recording functionality which I have heard good things about but I have not tried it yet.

A/B testing

Useful for: understanding with high confidence which features or functionality work better for users

A/B testing involves showing two or more versions of a product to different groups of users, and seeing which version performs best according to a set of pre-defined metrics. It enables objective, evidence-based and data-driven product decisions, and can reduce the risk of change by testing early with a smaller subset of users. A/B testing helps product managers answer questions like:

• which button colour increases the likelihood the button is clicked?
• which marketing message do users respond to best?
• which content recommendation algorithm is most likely to get more users clicking on related content?
• does this new feature increase the time a user spends on our site?
• are there any unexpected negative impacts caused by this new feature?

There are limitations to A/B tests. For example, short-term metrics may not align well with long-term user goals. It can take time to build up to statistical significance. Tests can interfere with each other. There is also an administrative overhead involved in running and evaluating tests.

One substitute for A/B testing is to keep track of when product changes happen and monitor how different cohorts of users behave. For example, if we monitor key acquisition and engagement metrics for users who use the site in a given week and we see a sudden change, that might indicate that something we have done has had an impact. It might not be caused by anything we have done or might not be statistically insignificant, but it is still a signal that is worth digging into a little further.

SEO

Useful for: understanding which search terms users enter to find the site and which pages they land on the most

SEO (short for search engine optimisation) is the practice of optimising content so that more users can find it via search engines. Although many predict AI tools will imminently replace search engines, tools like ChatGPT search and Perplexity indicate that even if Q&A with an AI do replace traditional search engines, there will still be a place for surfacing the content that these AI tools cite in their responses.

Google dominates the market for search engines with more than 90% market share, so is by far the most important one to track. There are two tools I like to use for this.

Google Search Console is a free tool that helps people understand how their site is performing in Google Search. It helps me answer questions like:

• how have click-through rates, impressions, and average position in search results varied by page over time?
• which pages and keywords are attracting the most traffic to my site?
• which sites are linking to mine

SEMRush is a more general-purpose tool for online marketing tracking. I like to use the free version to help me answer questions like:

• which keywords are users using the most (in general, not just on my site). This helps me understand what terms I might want to target
• how are my rankings for different search terms changing over time?
• what technical and content changes can I make to improve my performance in search engines?

Surveys

Useful for: gathering feedback from users

Surveys are a great way to gather direct insights from users. Unlike all other tools in this post so far, they focus on what users say rather than what users do. They also offer a way to get the conversation started, and are a great source of user research participants.

Examples of surveys I like to run include:

“Did you find what you were looking for” micro-surveys. These are one-question surveys which ask a user a single-choice question, followed up with an optional comment field for those who want to share more. Great for quickly getting a sense of user satisfaction or thoughts on a topic.

Micro-surveys to request or feedback on a specific feature. These can be useful to help decide how users want you to extend a feature, or whether they want you to build a feature at all. Low user engagement with this type of survey can be an indication in itself that the feature is not wanted.

Survey to access a feature. Features such as data downloads often require users to fill in a survey first. Willingness to fill in such a survey and the data provided when they do can help product managers understand what problem users are trying to solve with the feature.

Jobs-to-be-done surveys. A JTBD (Jobs to be Done) survey helps organisations understand the types of tasks that people are doing and how difficult they find them. This helps product managers to decide which problems to focus on solving for users. These surveys are quite long-form surveys and usually take a few minutes to fill in.

My favourite survey tool is JotForm. It enables all of the survey types listed above, as well as bespoke emails/confirmation pages depending on how questions are answered. I have heard good things about SurveyMonkey and TypeForm too. Google Forms is also a good choice for simple surveys.

User conversations

Useful for: building a rich understanding of users – their goals, problems to solve, and how they are solving those problems today

Last but most definitely not least: I have conversations with people who actually use my products. There is no substitute for conversations with real-life users, and if I had to pick one thing from this list it would be this. I have blogged before on the importance of user research and how I am doing user research at a start-up, so I will keep this last one brief.

Calendly is my tool of choice for scheduling user feedback conversations. It allows users to schedule time with me and automates the collection of all the relevant consents. I liberally share this link to my calendar at every opportunity.

Dovetail is a tool I only started using two weeks ago but I am already loving it! It makes it easy for me to analyse recordings of user research sessions, extract insights and make those accessible to my team.