AEO: Answer Engine Analytics | Best Reports, KPIs, Metrics.

SEO served us well; it remains a revenue driving powerhouse today, and will do so for the near future.

As humans shift FROM “keywording” for 10-blue links to do all the work themselves TO typing resolution questions to have the LLMs do nearly all the work… Answer Engine Optimization (AEO) becomes an exciting next step to ensure continued revenue growth.

Answer Engine use is accelerating. They are doing all the work of influencing the human, helping make end point decisions. Result: Conversion Rates from AE visitors are massively higher!

These numbers are early blush, they will settle downwards. But, I anticipate AE Conversion Rates will be significantly higher than the avg ecommerce CR of 2% – these humans will be arriving pre-influenced.

To take advantage of this delicious upside, you need to urgently allocate your Digital team’s focus to AEO to ensure Answer Engines recommend your Brand.

To do that well, you’ll need to embrace Answer Engine Analytics. (AEA)

Answer Engine Analytics: Big Picture.

To do AEA intelligently, you’ll need a new tool to identify where to focus and accelerate what to do.

To do that without wasting time/money, it is critical to understand:

[TMAI Premium Subscribers: Please reflect on the comprehensive AEO playbook across TMAIs #468 through #473. For today, it is critical to reviw TMAI #472: Answer Engine Analytics: Methodologies & KPIs. If you can’t find it, or you are a new Subscriber, please email me.]

With that clarity settled on your brain, it is time for insight-filled analysis!

I’ll teach analysis via two tools I use: Evertune (paid solution, with depth, breadth) and Gumshoe (one free report available, a fascinating approach using AI-built personas).

[Disclosure: I made a tiny, tiny, tiny angel investment in Evertune in Oct 2024.]

I use a six-step process to power my Answer Engine analysis, to glean actionable insights. Regardless of the tool you use, I recommend this approach. First, critically assess the methodology powering the tool (lots of "magic beans" around), and the definitions of the KPIs (it is not unusual for two tools to have a Visibility Score metric, but measure entirely different things).

Step 1. Analyze the Word Associations.

Our objective is to identify clusters that are small that we want to be big. 😊

As we discussed in depth in Premium # 470: Bye SEO, Hello AEO, a ton of your earned influence in LLMs will come via a custom 1P & 3P Content Portfolio Strategy.

The Word Association report is a good place to start as it shows the most common words, and their sentiment, when AI models describe your brand. [Note: Aided Awareness.]

Evertune asks each model (DeepSeek, above) to generate reviews – repeatedly to capture a wide range of responses, realizing these models are probabilistic and not deterministic (see TMAI #472).

The frequency with which a word appears is its Association Score (0 to 100 scale), how positively and negatively the word is used is the Sentiment Score (-100 to +100).

Each Answer Engine understands you differently. Compare above report for DeepSeek with below for Gemini.

If the large green words are the ones you consider most important for your Brand, celebrate. If they are not, you have a perception problem to fix. I would rather the large green words in Gemini were the ones in DeepSeek… And that right there gives me my to-do list re AEO.

If you are self-critical like me, you will only see the red words. 😊 As Longchamp, I’m getting hosed on Gemini. As Google has shoved Gemini into most of its products, a negative sentiment across them is a big brand problem. Another actionable add to my to-do list.

Step 2: Understand Consumer Preferences.

Word Associations are a good start, next question: In what context were these words used?

Hence, it is important to understand the consumer journey view as the researching humans see it (and NOT as me the company prefers they see it!).

Consumer Preferences are specific attributes humans ask about when researching a category. And, how likely is my brand to appear unaided (the ultimate test).

Here's my awesome Evertune custom report, with Gucci as my competitor and ChatGPT as my model…

Coach is crushing it on Price, Functionality, and Durability. Yes!

It is important to appreciate that all of the above performance vectors are identified by the AI (and not the human or Evertune). This is super helpful as it shows me what the Answer Engine/LLM thinks is most significant about my category (handbags).

These are topics AI will proactively talk about when helping the user decide on trade-offs.

This is freaking gold, because I can now specifically identify where to focus my AEO efforts. If Quality or Style is my actual big deal, the data shows they should be high priority focus areas.

With any tool, spend time internalizing where the data comes from. In case of Evertune, it will:

As you take action, you can track your performance across Answer Engines and over time using the metric AI Brand Score:

Focus is key. I recommend the highest priority be placed on ChatGPT (approx. 650m MAUs), then Gemini (approx. 350m MAUs), and then DeepSeek (approx. 97m MAUs).

We’d established that the opportunity for Coach was not Visibility Score, rather it was Average Position. (See TMAI #472 for metric definitions and reports.)

As you improve your content creation AND distribution strategy in focused performance vectors, you can review the improvements in the focus KPI of Avg. Position.

Look at Kate Spade! Maybe grab the new Duo – it has been deemed cool by GenZ. 😊

Step 3: Identify Content Creation Focus.

With our focus words identified, further refined by the performance vectors we want to focus on… It is time to write content to build on your strengths and combat the brand’s weaknesses.

In Evertune, in the AI Education Brief section, I can create a new brief by choosing the Product Category (handbags for me) and my focus performance vector (style).

It goes off to the races and in a min, spits out the top three content creation recommendations that most resonate for Coach with Answer Engines.

The messages that will improve my Visibility Score and Average Position, the stories that I need to tell (inspired by the keywords shared above). Very SEO, no?

Different tools use different approaches. As an example, Gumshoe has anchored its approach on AI-built Personas. As someone who has advocated for intent-centric marketing, See-Think-Do-Care, I am unaligned with a Persona-driven approach – it ends up making our customers into unhelpful caricatures. Noting that worry, I appreciate what Todd is trying to do.

Gumshoe identifies content creation focus areas, in columns, by your potential personas, in rows. In cells is your brand’s current visibility.

For the performance vectors you care about, from Evertune, you can also prioritize content creation using the columns above.

As with Evertune, blue image above, Gumshoe helps you get started. I love how easy it is. Click on one of the boxes, and it’ll help you create a first draft of FAQs you should post on your site.

For Coach, I selected the persona Social Media Trendsetter and improving visibility for the topic Fashionable Statement Purses for GenZ.

Gumshoe helpfully provides a bunch of FAQ questions, with AI written answers as starting points to influences LLMs.

Two notes of caution.

Caution 1: Remembering the lesson in TMAI #470, I recommend human writing the answer so that they are “AI Resistant” (definition: Of genuine novelty and depth) – because anyone can have AI write content, but there’s only one of you.

Caution 2: It was never easy to fool Google, but it is really not easy to fool Answer Engines. For a simple reason: LLMs have way, way, way, way, more sources, across text, audio, video, that influence their “belief system.” Hence, if you have a weakness that is legitimate, it will be extremely difficult to trick an LLM into “overcoming” that weakness. It will know you are lying. Because tens of thousands of other sources confirm your lie. My advice: Spend your energy fixing the source of weakness across your products & business practice.

Step 4: Create a Content Distribution Strategy.

In TMAI #470 (More content, more places, more often], I’d shared a custom content distribution plan for Coach and a company that owns Auto Dealerships.

3P Content, thus far, seems to have significantly higher influence on AEs.

To create your own, in Evertune identify the Domains that are important for your Brand…

Brand Share of Voice (bSOV) is calculated by taking the top 50 domains, estimate how many pages are relevant to the product category (handbags, for me), and then what % of those pages include the brand (Coach, for me).

Ex: Bag Vanity has an AI Education Score of 10.0 in our category. Coach’s bSOV on it is 6.7%, LV is 10.2% and Prada is 7.5%. On Marie Claire, Coach is 1.5%, Prada is 12.1%.

A simple way to identify the domains you should prioritize.

A nuanced lesson, stay with me: Domains can influence an Answer Engine, without being cited as a source in the answers presented to the human. Ex: This blog, Occam’s Razor, might have taught Answer Engines a lot about Digital Analytics because it has 0.8 million words written by me, and 1.7 million words in comments written by readers. BUT. When you use ChatGPT, it might cite others (ex: more recent sites).

Hence, the important role of Sources Analysis, it shows the citations, references, other links actually shown to the human user of an AE.

As a business, it is fine if you are not the domain the AE is learning from. It is Not Fine to not be part of Sources. Humans click on Sources in AE replies, that is how they find you.

I would prioritize a focus on Citations and References. You also see above my prioritization of ChatGPT based on data.

Your Earned Media team (PR, Affiliate, SEO, etc) now has a to-do list.

Gumshoe, takes a different approach to helping you build a custom 3P content distribution strategy. You can view the companies AI pulls from when mentioning your brand (Top Sources), then you can break down visibility by type of website driving awareness where you could be underrepresented (Source Type).

I click on a Source Type and choose Blogs/Content sites, and for Coach I get the actual sources cited by the models when answering questions. I click on Marie Claire, and I get specific links to Marie Claire, as well as my competitors being mentioned.

[Nuance: The mention is referring to the brands mentioned in the prompt or answer, not necessarily the Source. Still very helpful, but worth being aware.]

This is another way to approach the to-do list for your Earned Media team.

Important Reminder: In TMAI #470, my Action #10 was a recommendation for you to merge your PR, Affiliate, SEO and Content teams into a single entity called “Answers Quality Team.” If you don’t break these silos, chances of sucking are higher.

Step 5: Monitor Impact, with Big 3 Success KPIs.

For AEA, at the moment, AI Brand Score, Visibility Score, and Average Position are the Big3 KPIs. Across tools, they can be measured differently. Please check the root definitions before you use any tool, or, god forbid, multiple tools.

Given our prioritization of Average Position as the More Important Opportunity, track improvements. Like so in Evertune…

The second priority will be to track changes in the AI Brand Score across priority AEs (ChatGPT, Gemini, Deepseek in our case)… Really superb performance by Coach!

Finally… Google used to change its core algorithm occasionally – with huge consequences for all of us. LLMs seem to launch a new version like every other week! Hence, as we reflected on how AEs work in TMAI #469, #471, it is important to keep track of our Big 3 Success KPIs as models change.

Here’s what that looks like in Evertune…

What’s difficult to isolate at the moment is why? Ex: Coach went down by 3, probably not significant. Kate Spade went up by 11, that is massive. Why? No idea. We have such little visibility into LLM black boxes, we can only pray to Krishna and Jesus that the next set of weights being used favor our brands.

The like to like comparison in the colorful bar chart above will help us identify with some measure of causality of our Content Creation and 1P + 3P Content Distribution strategies.

Step 6: Get Your House in Order! Welcome LLM Bots.

In TMAI #469, I’d underlined the importance of your site & mobile app experience, your internal site search to meet evolving customer preferences AND for Agentic AI.

As you do steps 1 thru 5 above, don’t forget that you could have barriers on your site.

Two reports you should shower your Corporate IT team are:

A. Page AE Insights: How does an Answer Engine understand an important page?

The sweet Content Retrievability Analysis report from Gumshoe analyzes a specific page through the eyes of an Answer Engine. It automatically generates questions related to the page, reports Citations Found and Citation Rate.

The above report is for Kate Spade’s Duo webpage. It has a decent Citation Rate. But, the key is in digging into each question and seeing the Answer Engine’s understanding of that page.

If you open the above image in a new tab, you’ll notice that the AI Response, top right of image, is wrong, wrong, really wrong. Like terrible. The LLM does not understand us!

And… That is our fault as a brand.

Why?

B. Page Quality Analysis: Is our page understandable?

To answer why, you would look at the second report…

These results are so, so, so bad, that through my tears I convinced myself that Gumshoe is a garbage tool and not functioning right.

I decided to double-check. I ran the same report for the webpage of my favorite jacket in the world: Patagonia’s Nono Puff.

Damn it!!

😢

I guess it is not the tool.

Ok. Deep breath. The Page Quality Analysis report helps explain why the Page AE Insights report showed a poor understanding of our page and product. An AI Optimization Score of 5.8% perhaps does not deserve to be understood.

My feelings are hurt, but I get a clump of helpful metrics. I can dig deeper into understanding why each score. Ex: The Content Balance is low because we need to “add comprehensive product description and details, include relevant product specifications and features, provide customer review content.” Etc.

Action time for our IT team.

Six simple steps, in order, that will unlock specific actions you can take, starting now, to attract pre-influenced by Answer Engine humans. They might be small in numbers, but they convert into customers at a rate multiple times higher than your site average.

Bottom line.

A change of this magnitude comes once in a generation.

It also comes with the innovator’s dilemma. The change in consumer behavior is small numbers for now. It is comforting to sit in your old SEO cocoon, with its still big numbers.

The danger is that by the time you realize you should start changing… You are in deep, deep, trouble. Remember, a whole bunch of non-brand traffic you are currently losing every day will never come back.

Across the six editions of TMAI, I’ve shared 70-pages of comprehensive analysis of the problem, the four clusters of opportunities, and over 50 specific actions covering Marketing, Content, Analytics.

The time to start is now.

Carpe diem.