4 weeks ago, I ran an AI experiment.
If you read issue 37, you may remember that I talked about the new Preferred Sources option that Google released and how you can mark specific websites as your Preferred Sources in AI search.
In that issue, I asked you to mark my website, tanyabrodycopywriter.com, as a Preferred Source.
My premise was:
To find out whether asking your subscribers to mark you as a Preferred Source actually changes whether AI search engines surface your business.
I also said that on July 17th, I would release the findings.
I sent this newsletter to my email list on Friday, July 17th.
Somehow, my past self knew that I would be a day late in sending this newsletter. I am grateful to my past self, for allowing my future (now current) self a little leeway here, as it was a busy week and it took extra time to write to this week’s Email Marketing Ecosystem newsletter issue.
Anyway, here’s what I found.
My experiment failed. 4 things went wrong and 3 of them were my fault. One I can blame mostly on Claude (with Claude’s permission).
The Results of My AI/Google Preferred Sources Experiment
First, the experiment itself:
I tracked:
- Citation changes in AI Search
- Subscriber-reported visibility
- Position change
I did this by running 6 search queries in incognito mode on 4 AI platforms.
The queries were:
- “Best email marketing consultant for conscious businesses”
- “How to write ethical email marketing”
- “Email marketing strategist for mission-driven businesses”
- “How to re-engage a cold email list”
- “Why email marketing matters for small businesses”
- “How do I figure out what to say in my nurture emails?”
The 4 AI platforms were:
- Google (standard search with AI Overview)
- ChatGPT (free version)
- Perplexity (free version)
- Claude (free version)
My baseline measurements had my business showing up 3 times in Google’s AI Overview, twice via LinkedIn and once for Medium.
My measurements today were a little different.
First, I lost my position in Google’s results for “Email marketing strategist for mission-driven businesses” and “How do I figure out what to say in my nurture emails?”
I still showed up once in Google’s AI Overview, for the “Best email marketing consultant for conscious businesses” from my LinkedIn profile.
I also showed up as a mention in Perplexity’s results, though not as a specific recommendation.
And oddly, Issue 37 on my website, the very issue in which I wrote about this experiment, showed up in 3 of my 6 Google searches as the only result in the organic search engine results page (SERP).
Initially, I thought this was great! Check that out, I’m showing up at the top of Google’s organic results. I figured it was probably because, despite the fact I was in incognito mode, Google was picking up past search habits and my IP, so it was giving me biased results.
Then I tried scrolling down…
To nothing. No other results for this query.
I tested it a couple of times and got the same result.
That’s when I realized (and Claude confirmed) that the reason Issue 37 was showing up, was because I had the exact queries listed in the issue.
As I mentioned, this happened 3 times out of the 6 Google queries. I didn’t show up in the AI Overview in all 3, just the first one. But this same article did show up as the only result in the SERP.
The good news? If someone ever searches for “Best email marketing consultant for conscious businesses” or one of my other two search queries, they’ll find Issue 37.
The bad news? If those queries are showing up, word for word for my business and no one else’s, that means no one else is using those specific words. So, they’re probably not the best queries for my business.
Mistake #1 on my part.
I suspect the Perplexity mention is part of the same phenomenon, as it specifically mentions this query “as a target query in her AI-search research.”
Still, I did at least hold one position in the AI Overview, so that’s cool.
Otherwise, I have no evidence of movement in any direction.
Mistake #2 Failure to Follow Up On My Part
In Issue 37, I asked you to mark me as a Preferred Source as part of the experiment.
I gave specific instructions and included big buttons that linked to the Google page where you can both mark my business as a Preferred Source, as well as search to see if your business could also be a Preferred Source.
I said I would put additional buttons on my website, and on other articles on LinkedIn, Substack, and Medium.
I also intended to remind you to mark me as a Preferred Source as part of this experiment in future (now past) issues of the Email Marketing Ecosystem newsletter.
I failed to do any of those.
What can I say. Just like every other small business owner in existence, I got busy and this task fell by the wayside. Sound familiar? Yeah.
Because of this, even though the email version of this newsletter had a 27% open rate, a whopping 2 people clicked on the link to mark my website as a Preferred Source. Neither of them responded to tell me whether they had marked me as a Preferred Source.
Again, this is entirely on me.
The other issue here is that I asked something of you without giving much in return.
I had hoped that showing you my experiment and urging you to try it for yourself would be enough to get you to help me out.
It wasn’t.
And it goes against my own policy of serving first, selling second.
I didn’t give enough in that issue to make this particular ask worth your while. I apologize for that.
Mistake #3: I Tried to Measure Something Ultimately Unmeasurable.
“It seemed like a good idea at the time.”
Famous last words.
I asked Claude to help me create this experiment. I admitted as much in Issue 37.
Unfortunately, both of us missed a very important fact.
Google Preferred Sources is a per-user personalization signal. Meaning it lives in an individual’s Google account.
The badge and the increased visibility are based on whether someone has marked a website as a Preferred Source.
Here’s Claude’s confession of this mistake on its part.
I knew this going in, as I’d read Google’s documentation and linked to it in the original article.
I had interpreted this (or perhaps hoped) that as more people clicked through and marked my website as a Preferred Source, this would give more recognition to Preferred Source sites overall. Similar to how a social media post or video gets more views the more people “like” it.
This is incorrect.
The truth is that’s not how it works. The person who marks a website as a Preferred Source will continue to see this site in their AI Overview results. That’s it.
Because Claude had me track each of these queries in Incognito Mode (specifically for Google, but also not logged into any account for the other 3 platforms as much as possible), this measurement wouldn’t show up, regardless of how many people may or may not have marked my website as a Preferred Source.
To Claude’s point, personalization means your business’s visibility is different for every person who runs a search where your business shows up.
This has been true with SEO for a very long time.
When I worked for FindLaw 15 years ago, the lawyers would always complain that their competitors were showing up at the top of their SERP. We had to (as politely as possible) explain that every time they searched for their competitors to see how they ranked against them, they were teaching Google to bias for their competitors and against their own firms in search results. (They never believed us. Lawyers… 🙄)
As personalization has become the norm, this factor has increased. To quote Claude, “There is no longer a single ‘What AI says about you.’”
This makes measuring anything difficult for all businesses, especially for small businesses.
Mistake #4: A Very Small Sample In A Constantly Evolving Environment
I admitted in the original issue that this was a very basic experiment.
My sample size was small.
Six basic queries, 5 suggested by Claude, one based on my ideal customer avatar’s most commonly asked question/challenge. These were asked across 4 AI platforms at 2 times, separated by 4 weeks.
In the baseline results, I showed up 3 times, in Google results, across 3 different queries.
In the second set of results, I also showed up 3 times (sort of) in one query.
Looking at it, that might be considered a loss.
However, in the last 4 weeks, ChatGPT has released 3 new models. Claude has released 1 new model. Google and Perplexity have not released any new models. Despite all of this, AI continues to develop rapidly and every little change affects everything else.
Think of the Butterfly Effect on a digital scale. What was relevant 4 weeks ago probably isn’t relevant today. In our ever-evolving digital world, that means a lot.
Because of this, my tiny little experiment isn’t terribly relevant to me, or to any of us small business owners in the long run.
And the very minor movement in my results is probably just noise in the overall chaos.
So honestly, I don’t have enough data to make any significant claims. Nor would anyone else running an experiment of this size.
This is why I’m a copywriter, not a usability researcher, like my sister. She would have done a much better job designing this experiment. (Note to self: Next time, enlist her help.)
What I Learned: I Did the Thing I Tell My Clients Not to Do
I asked you for something that served me, not you.
I violated my own rule of serving first, selling second. The two clicks out of my 27% open rate proves that.
Oops. Bad me! My apologies.
Even though this wasn’t much of an “ask,” all I did was request a few minutes of your time and a very minor action, I didn’t respect the relationship we have of serving you first.
This is a lesson for me, and for you.
Make absolutely sure that every time you ask, you’ve given first. Provide the value to your subscribers that makes them want to respond when you ask.
This is true whether you’re asking your subscribers to take a tiny action, like I did, or you’re asking them to buy your product or service. Deliver the value they want and deserve first. Then make the ask.
It matters. A lot.
It would have been more useful for all of us if I had found a way to show improvements in your AI search results instead of my own. Alas, we may be well past the point of that ever being able to happen. (Thanks Robots.)
Moving Forward
I will say that Google Preferred Sources is still a good thing, even if it is only at the user level.
I am working with a designer on a new website. I will ask them to include buttons that link to the Preferred Sources page on a few of my pages (with an explanation) so that they will find the latest issue of the Email Marketing Ecosystem newsletter, or whatever else may be relevant to their queries that is on my website.
This experiment is complete. I don’t see any reason to track this data again in the future.
However, as more people do click through to the Preferred Sources page from my site in the future, I will measure whether more citations show up for queries relevant to my business. This isn’t a new experiment. It will be an ongoing process.
I talked a bit about this in Issue 33, about AEO and GEO.
And honestly, I can’t say whether I proved or disproved whether Preferred Sources will do any good for small businesses because I failed to truly test it.
Most importantly, I will always remember my own rule of serving first, selling second, in everything I do.
I hope you can take away something useful from this experiment too.
If you did, please email me and let me know what it was. I’m interested to know and it may help me with a future newsletter issue. I always reply to every email.
Finally, I seem to be having some “issues” with my bathroom sink clogging up. This week’s picture is of the most adorable cause of a clogged drain ever, Lugh, sitting in that bathroom sink.
Want to Get The Email Marketing Ecosystem Newsletter Delivered Directly to Your Inbox Every Thursday?
Sign up right here and get the latest issue before it’s published anywhere else: