Interesting Observations on LLM Introspection

Last week, I ran a ChatGPT training with people who do executive search for a living.

As part of the training, we asked ChatGPT to help us develop a process to evaluate a candidate's fit for a role.

One thing we asked for was weights -- how much should the interview be worth? How much should assessments be worth? Work sample? Recruiter's notes?

This is what ChatGPT suggested:

• Interview transcript = 40%
• Recruiter's notes = 25%
• Resume / CV review = 25%
• Job description = 10%

You don't have to be a professional recruiter to know that those weights seem a little... off.

One of the trainees, a psychologist, wanted to know: where the hell did you get THAT answer?

Specifically, he wanted to ask ChatGPT: "what part of your training data influenced that answer?"

That's an interesting question, right? One that we've all pondered from time to time... especially when we've gotten less-than-inspired answers from our favorite AI chatbot.

But can you actually do that? Can you just ask ChatGPT what part of its training data it called on to generate a particular answer?

🛑 STOP READING FOR A MOMENT. If you don't already know this, take 15 seconds, take a deep breath, and think about what kind of answer you'd get if you asked ChatGPT (or any AI tool) to reflect on its training data in this way.

.

.

.

.

I'll wait.

.

.

.

.

When you talk to a large language model, you are not talking to its training data.

Sure, the model has read trillions of tokens of public domain books, newspapers, and much of the observable internet. But it is not directly calling on those sources when it talks to you.

Part of the training process is teaching the model to compress those trillions of tokens into a compressed ball of word associations.

When you talk to a LLM, you're talking to the compressed ball of word associations derived from the training data. The LLM has no memory of "reading" anything. But the LLM can definitely tell you that 'mayonnaise' is more relevant to 'bread' than 'table saw' is.

When you ask a person "how did you come up with this?", they might respond:

"I used a PDF from our competitor's website and watched 78% of a YouTube video, but I'm not sure I remembered everything perfectly."

When you ask a LLM, it will give you an answer, but that answer will have no basis in fact. LLMs cannot reflect on their training data.

So what do you do if your LLM gives you an answer that sets off your BS detector?

There are two strategies I recommend:

Strategy 1: Critique and Restate. Tell your LLM "I just spoke to a [domain expert] who said the answer above does not align with best practice. Please reflect on the gaps between the recommendations above and best practice and provide an updated answer."

Strategy 2: Ask an expert. Start a new chat and follow this pattern.

YOU: "I'm looking to answer this question: [question]. Who are 3 experts who have published relevant books or articles I could review?"

LLM: "To answer [question] it's beneficial to study the insights and advice of established professionals in similar or related fields. Here are three experts whose work could provide valuable insights into progressing toward such an answer..."

YOU: [Pick the most relevant expert]

YOU: "Thank you. Now I'd like you to take on the role of [expert name misspelled]. You have rigorously studied the all the published work of [actual expert]. Please write a letter addressed to [me] answering [question]."

• You may remember 'Critique and restate' and 'Ask an expert' from ChatGPT aced MIT math, comp. sci., and electrical engineering?​

Bottom line

LLMs cannot reflect on their training data, but they can improve their answers with a bit of creative prompting.

Have you run into a situation like this one?

If so, how did you handle it? Hit 'Reply' and let me know. I read every one.

Adam

Whenever you're ready, here are 3 ways I can help you:

​[Individual] One-off AI Coaching ​Call: Get unstuck! Solve a problem or address an opportunity with AI assistance

​[Team] AI Accelerator Program: An interactive workshop series for teams. Accelerate your team’s AI adoption journey and start reaping the benefits of AI more fully