Using AI for competitor analysis

I just finished The Complete Agentic AI Engineering Course by Ed Donner on Udemy. It was great! I thoroughly recommend you give it a try if you are interested in learning how to build AI Agents, and prefer to do so via hands-on projects.

Instead of building hobby projects, I focused on building AI agents to help me be a better product manager.

This post covers my very brief journey trying to build an AI agent that could automate competitor analysis, why I ended up deciding to use the Perplexity app instead, how I am using this capability to help me get more done, and how I hope to improve this capability in the future.

Context

Competitor analysis is an important task for a product manager. Since it never has deadlines attached, it never has the same urgency as most of the other things I do.

I have a Notion database of products I like to keep track of. The most relevant products I try to look at every few months, the less relevant ones I like to look at every year. Whenever someone in the team identifies a product to take inspiration from, we add it to the database for investigation.

There are two types of competitor research I like to do:

1. Researching a product for the first time
2. Checking to see what has changed since the last time I checked

This post focuses mostly on 2 – checking on a product I know well to see what has changed since my last round of checks. But I am also using AI to help me research a product for the first time.

Evals

Most good AI projects start with evals. Evals are the dataset used to objectively test whether AI models have given a good response or not.

I started by focusing on a product that I know well. I took an hour to read up on everything they had done in 2025, and played around with their product. I came up with a list of 10 interesting improvements they had made in the past year that looked noteworthy to me. I evaluated my model performance against how many of these 10 improvements were successfully identified.

This dataset of 1 is alarmingly small for an AI project. If I was building an AI tool for a user facing product, I would expect to have thousands of these types of test in place. But it beats relying on vibes alone. And for an internal tool where I am the only user, it feels proportionate.

Starting out manual

Before jumping into code, I decided to try to get good results using the best AI models available at the time of writing. Knowing that I need models which are good at web based research, I decided to try Deep Research mode in both Perplexity and Gemini.

After a bit of experimenting with prompts, here are the results:

• Perplexity: 9/10
• Gemini: 6/10

Perplexity was also a lot faster.

Off I went to try to build an agent myself in CrewAI!

Design

CrewAI was one of the AI tools I most enjoyed from the Agentic AI course. A “crew” consists of agents who do tasks, which felt very intuitive to me as someone with a background in project management. So I decided to start with CrewAI.

I set up a new crew using UV. If you use Python and haven’t heard of UV, do yourself a favour and take 5 minutes to check out uv is the best thing to happen to the Python ecosystem in a decade. One of the great things about the AI agents course was learning about some of the tools Ed uses when he is working with AI. Tools like UV.

Before starting to write my agents and tasks, I decided to get some design advice from Gemini. I asked Gemini to:

1. Identify three agent architectures that could be used to solve my problem
2. Do a SWOT analysis of each, and give them all a score
3. Identify a fourth agent architecture that draws upon the context of the first two questions
4. Re-do the SWOT analysis of each

The end result was an approach involving:

• 3 agents
• 3 tasks
• 1 tool connection to allow the agents to search the internet for information

Implementation

Result of first attempt: 1/10.

That means it only found 1 of the 10 improvements that I had found manually. Not very good!

To debug the issue, I examined the traces. Traces document the requests made to AI tools and their responses, and are a great way of debugging generative AI systems when they go wrong. I quickly discovered the issue. My first agent – the one who goes out and collects relevant information from the internet – was only finding 9 snippets of information. To get round this issue, I instructed this agent to perform multiple web searches.

Result of second, third and fourth attempt : 2/10

It took me a few tries to get multiple instances of my agent to go out and do web searches, so that more context could be brought back overall. Still, it struggled. Not enough results were brought back, and not enough context from each. Upon further investigation, I realised that the free version of SerperDevTool I was using only allowed me to retrieve limited information from each source. I had an Exa.AI account for a different project, so I decided to give that a try.

Result of retrieving context from web using Exa.AI: 4/10

Getting better! However, there was still a lot of good context being missed. AI tools can only give the right answers if they have the right context to draw on, so unless I could get this right, the rest of the agent would struggle.

Pivoting to Perplexity Deep Research

I had spent almost a day on this project by this point. Whilst I was learning a lot from my tinkering and experimenting, I couldn’t help but think it was all bit of a waste of time. Perplexity Deep Research had already achieved 9/10 from my earlier checks. Even if I did succeed in building my own agent to equal Perplexity’s standard, it was likely to take me a long time to achieve. Given how fast the newest models are improving, I was realising that I may as well just use Perplexity Deep Research Mode for this job.

Fortunately, as an OpenRouter user, I can access API keys from most AI model developers, including Perplexity Deep Research. I decided to create a script that would allow me to quickly generate Deep Research reports on similar products, and output the results to markdown so that I could get them into Notion with a few clicks. If this worked, I could automate the script to run every 3 months, giving me quarterly updates.

So I decided to create a very simple crew. A crew of one super agent equipped with Perplexity Pro. I gave that agent one task – the task I had given Perplexity and Gemini right at the beginning.

Result of working with Perplexity Deep Research via CrewAI: 8/10

This was not quite as good was what I was able to achieve with Perplexity via their app. But the generative nature of generative AI means the results will never be the same every time. And 8/10 is really not far from 9/10.

It was great to generate 8 markdown files that I could easily upload to Notion in just a few clicks. However, of those 8 documents:

• One only generated a one paragraph response
• One ignored my formatting guidelines, and created 6x the text of the others
• The average cost was $1 per report. That’s pretty high by AI standards

My solution (for now)

I decided to stick with using the Perplexity Pro App to generate my competitor analysis. Why?

• It’s free for me to use the App
• The time savings from having all the reports generated automatically for me isn’t that high if some of them randomly don’t work
• I have other Agentic AI projects in progress that Deep Research models are less likely to solve any time soon, so I will focus my Agentic AI efforts on those instead. More about those in another post!
• A no code solution makes it easier to share this capability with non technical members of my team
• Prompting via the app seems less temperamental than prompting via the API

How does this help me do my job?

Right now, according to my alarmingly small evaluation dataset, I am better at doing competitor analysis than AI is (10/10 beats 9/10). However:

• AI is much faster than me
• 10/10 isn’t much better than 9/10
• AI is getting better all the time
• It is not a competition. Alan + AI is better than Alan or AI alone

Competitor analysis is also not something a good product manager should outsource entirely to AI. The scores out of 10 above don’t capture the extra context I get from using similar products myself.

So I take a few minutes to read these reports. I click on the links that look interesting, and spend a little time trying out the product for myself. I add screenshots of interesting things I find to the report, and add comments as I go so that I can refer back to the bits I find most interesting at a later date.

TABLE
Alan: Relevant improvements found: 10/10. Time taken: 1 hour
AI: Relevant improvements found: 9/10. Time taken: 1 minute
Alan using AI: Relevant improvements found: 10/10. Time taken: 15 minutes

Next steps

Here are some extra capabilities that I plan to explore in the future:

Auto publish reports to Notion

I have not tried the Perplexity/Notion integration yet, but one thing I would like to try soon is automatically publishing research to Notion, and using the context from previous research I have done to minimise the sharing of duplicate information. Be sure to turn off AI data retention in Perplexity if you give this a try, as I can’t imagine any employer wants the contents of their file management system used as training material for AI tools.

Using context from previous reports for future reports

Every 3 months when I read these reports, I expect to come across lots of duplicate information. It would be nice if new reports automatically excluded information that I had read before. Or at minimum, tagged it as potentially duplicate information.

Screenshots of key features

Agents are increasingly able to browse the web and click around as a human would. I would love to see the reports I generate include screenshots of key features. I am tempted to try to build an agent myself that does this. But I do not think it is long before this capability comes to Deep Research mode, or one of the AI powered web browsers like Perplexity’s Comet or OpenAI’s Atlas that have been launched recently.