AI product management in high stakes domains

I have spent most of the past decade developing digital products in domains where truth and accuracy really matter, including government services and public policy. More recently, that has involved a lot of working with AI. This blog post summarises some of the approaches to AI product management that I have seen work well and that have worked well for me in high-stakes domains

1: User value comes first

This one comes #1 in the UK Government Design Principles, Google’s People & AI Guidebook and pretty much everything Marty Cagan has ever written. In the AI domain, this means only considering AI solutions when they have the potential to help solve problems for users better than non-AI alternatives. It also means thinking very hard about how your AI solution offers value over and above what a user can get on existing general-purpose tools like ChatGPT. AI tools are not cheap to develop, and users need good reasons to continue using a product.

For example, decision-makers in the law and policy domain need very accurate data, and they want to be able to verify for themselves where that data has come from. Adding a ChatGPT-powered chatbot on top of a long text document would not have offered users a meaningful improvement over existing AI tools they have access to. Instead, Climate Policy Radar has focussed on building capabilities that address the core needs of decision-makers in the climate domain. These include mapping unstructured data from PDFs into a knowledge graph of structured climate law & policy data to make it easier to find relevant content. It also means showing the user the exact passages of text from documents that support claims made by AI models to help users verify the claims made by AI. These capabilities help users to discover insights from long climate documents, faster and more reliably.

2: Use cross-functional teams

Supervised learning is one of the main techniques used to teach AI models what they know about a topic. This technique relies on labelled data for both training and evaluation. Domain experts are needed to create the labelled data. Data scientists develop the models which consume the labelled data to predict the meaning of the text. Data and application engineers productionise these models. Product and UX experts incorporate these models into user experiences that are usable and valuable and plan how to take them to market.

It is accepted wisdom that good teams are diverse teams. This is especially important for teams developing AI products because AI is a fast-moving complex domain where design and technology conventions are only just beginning to emerge. I learned a lot about what good cross-functional teams look like during my time at the UK’s Government Digital Service. It is good to see the tradition continuing at the Government’s AI Incubator, and that the list of specialist roles has expanded to include more AI-specific roles.

*Cross-functional AI team roles being advertised for by the UK Government*

3. Augmentation over automation

Automation is when AI completes an action without human intervention. Augmentation is when AI and a human need to work together to complete an action. When people are responsible for something, they want (and sometimes need) to maintain control over it. So AI in domains where the cost of failure is high tends to require a human in the loop.

A great example of AI augmentation can be found at Full Fact, a fact-checking organisation based in the UK. They use AI to help their fact checkers work faster – not by trying to replace them with AI. Similar approaches can be found in the medical domain, with AI helping specialists do MRI scans faster.

Automation can go spectacularly wrong in high-stakes domains. Mistakes by automated systems in the US have left people bankrupt, deprived of healthcare services and rejected from jobs.

*This poor guy went bankrupt because of a mistake by an automated system*

4: Avoid garbage in, garbage out

High-quality machine-readable data is critical for AI models. After all, if data can’t be read by machines then it cannot be used by AI. And if data is read incorrectly by machines, then it is unlikely to generate a useful result.

This makes high-quality structured data a top priority for any digital product or service. This was true in digital transformations I have worked on like those at the Land Registry where they are trying to digitise the Land Register, and move from Tetris to LEGO in the process. It is particularly true for AI products. Climate Policy Radar works mainly with Government laws and policies, which are usually long text documents in PDF form. PDFs are notoriously difficult to extract data from. Actively measuring the quality of text extracted from PDFs and improving accuracy improves every area of the product where the machine readability of text from PDFs matters (which is pretty much all of them). ChatGPT won’t be of much help if it can’t detect any text in the document you upload to it.

Good data quality enables users, search algorithms, transactional services and AI models to generate better insights from data, faster and at lower cost.

5: Move fast and break things

AI is a game-changing technology. But since ChatGPT 3 burst onto the scene, it has earned a bit of a reputation for inaccuracies, biases and hallucinations. People in high-stakes decisions have low tolerance for these types of issues. Move fast: yes. Break things: no thanks.

Here are some of the things that can go wrong with AI models:

Misunderstanding intent. If an AI tool misunderstands a user’s question, it is very unlikely to give a useful response at all.
Equity & inclusion. AI models can amplify biases that affect certain groups. There are some examples going around about AI models being used to screen job applicants, unfairly screening out candidates from underrepresented groups. You can easily imagine the ways in which this could go wrong if used to decide who qualifies for Government support from critical services.
False positives occur when results are included when they should be excluded. Minimising false positives is important in domains like fraud detection or spam detection where inaccurate predictions have a high cost. Precision is the name of the metric that measures false positives.
False negatives occur when results are excluded when they should be included. For thorough research where it is important that no evidence is missed, minimising false negatives can be more important. Recall is the name of the metric that measures false negatives. It is important to measure and minimise both precision and recall but at a certain point, they start to trade off against each other. Which one to optimise for varies by the type of product being built.
Domain switch occurs when a model is used to give results in an area which it was not trained to. This increases the likelihood of inaccurate results. GOV.UK get around these problems by triaging out users who want to ask questions which their experimental chat tool is unlikely to answer effectively.

*GOV.UK chat is making sure I plan to ask it questions about a domain it has been trained on before allowing me to access their experimental AI tool*

6: Measure performance

Measuring performance is one of the key tenets of a good and responsible AI strategy.

Expert human labellers create evaluation datasets to measure model performance. This allows the accuracy of AI classification models to be measured. Climate Policy Radar’s Queried prototype is experimenting with ways to surface these scores to users to help them understand the accuracy of AI responses.

*Climate Policy Radar’s Queried prototype automatically checks AI answers for quality and accuracy*

Equity and bias can be understood by ensuring that a good amount of evaluation data comes from underrepresented groups. In the domain of written content on the internet, that means things like ensuring there is a good amount of data from underserved regions of the world or content that is not written in English.

Measuring data quality and experimenting with ways to improve it is also important, as none of the above works without good data. For text data, this includes making sure text is extracted from documents and that the text extracted retains its original meaning and structure.

Metrics like these help teams to understand where they are doing well, and where they can do better. As a product manager, this helps me decide where to prioritise future improvement efforts.

7: Make things open

This is another one I have shamelessly borrowed from the UK Government Design Principles. Being open means several things here.

Making code and data more open accelerates impact, enabling others to learn from, reuse and build on top of good work.

Being open isn’t just about doing good in the World. With a section heading “Move fast ~~and break things~~” I poked a bit of fun at Meta. But Mark Zuckerberg’s focus on open-source AI makes commercial sense. Being open means that data and models are open to others to improve and scrutinize, which helps make them better. (In Meta’s case, it also helps them undercut competitors.)

Being able to verify information in order to understand how much it can be trusted is a core need of users working in high-stakes domains. Features that really excite me in this space include:

A. Highlighting relevant text. Displaying citations for all claims made by AI models and enabling users to find the exact passages of text in documents that match their query helps users quickly judge the accuracy of AI responses.

*NotebookLM identifies the passages of text in a document that support claims made in their AI response*

B. Transparency. Being transparent with information used to construct a response helps users figure out how accurate an AI response is

*The Consensus Meter (by Consensus) helps users understand how much agreement there is in the answer to scientific questions*

8: Continuous feedback

Last but not least, it is important to build in ways to gather continuous feedback from users. This makes it easier to identify improvements and can provide useful training data to help fine-tune models.

Thumbs-up/thumbs-down features on key page elements help gather structured feedback. A user labelling search results, metadata or answers as good or bad could be used to help improve the underlying algorithms.

*Gemini make it easy for users to feed feedback on results via thumbs up or down, and then prompts users to give extra feedback*

Qualitative data is also critical in fast-moving domains like AI. User behaviours are evolving fast and this is a great way to understand why users are behaving the way they are. I like to sprinkle lots of micro surveys throughout the products I am responsible for to get a sense of what matters to users. This also helps me get lots of conversations started with users. I liberally share a Calendly link for users to sign up for a feedback conversation everywhere I can.

Overall

None of the 7 points in this post are unique to AI products. But the power, complexity, novelty and data-driven nature of this new technology make them especially important. Especially so in domains like government, news and policy where there is a premium based on accuracy and truth.

Not for the first time, I also want to give a shout-out to Henry Franks, Kalyan Dutia, Harrison Pim, Matyas Juhasz and Michal Nachmany from whom I have learnt so much about AI over these past few years.