SEO Riddler by Sara Taher

I’m Building a Search Engine (Part 1): Crawling

Sara Taher — Tue, 22 Sep 2026 13:18:15 GMT

If you find value in my newsletter and want to support the work that goes into it consider —> [☕Buy Me a Coffee☕]💡 Want more from me? [SEO Strategy Course] | [Build Your Own SEO Tools (with Python & Agentic AI)] | [Premium SEO Mastermind]

Alternatively you can become a paid subscriber:

Intro

So you’ve heard… I’m building a search engine from scratch. Here’s the tea: I want to build a small search engine for all my content specifically, so you can use it to search for my input about a specific topic.

I’ll be sharing my journey of building this in my newsletter. Every time a new milestone is done, you’ll read about it here first!

I think is is a fun project but also building a search engine is far from simple. One way to simplify the process is by being very specific like in my case, the goal is my content specifically, although there’s still details that go into this.

So how to build a search engine, eh?

Components of a search engine

We are not re-investing the wheel here by any means, but there are design decisions that allow for creativity.

A search engine is primarily made of 4 components:

A crawler: of course we need some time a of a system to go find the URLs for our search engine and define the rules this crawler will follow.
Indexer: we also need a system to process the content on those URLs and produce something that’s simpler and searchable and store it in a database that we may call our “search index”
Query processor and ranking system: these are two sides of the same coin. What happens to the query when a user searches? how do we interpret it? for example finding the user intent behind the query, so we can match this query against our search index, that will be sorted and filtered based on the “ranking system” we have built.
Search Interface (SERP): our search results page, where user enters their search and get back results.
So this is generally speaking the 4 components you need to build a search engine. Now let’s talk about my search engine! I don’t have a name for it yet but I’m open for ideas.

My search engine: the crawler

I started planning with Claude Code. There are few design decisions and the steps I chose to build my crawler:

The crawl depth: this is a small search engine, that will rely on two sitemaps for seoriddler.com and sara-taher.com. It will also crawl any specific URLs (not the domains) I have linked to in my content. But that’s as deep as it will go. So the maximum crawl depth is 1.
Pre-fetch URL cleanup: before a URL is fetched, it get normalized. This means the URL gets reduced to it’s canonical format by removing tracking parameters for example.
Link filtering: I have decided to exclude links to .pdf, .jpg, .zip, .css, .js, etc. from my crawls and only fetch html pages. However, those non-html links are stored and labelled as such.
Storage architecture: For each URL we decide to fetch, we store two things: information about the URL and the actual HTML of the page.
Information about the URL includes the URL itself, page title, description, crawl depth, crawling errors, status code, and other metadata. This information is stored in an SQLite database.
The actual HTML of the page is stored separately as a compressed gzip file. The name and location of that gzip file are stored in the SQLite database entry for the corresponding URL, so we can always connect the URL’s metadata to its raw HTML.

Link Graph: I’m also storing which page links to which other page, the anchor text used for the link, and, if we didn’t fetch the target URL, the reason it was skipped. Together, these rows form a graph of how the pages are connected. Right now that’s 2,830 rows from 744 pages. Here’s how this looks in the DB. I’m using
https://sqliteviewer.app/ to view the database.

Crawl controls: I also decided, with the help of Claude Code, that we would not crawl pages sequentially, because that could take a long time. Instead, we crawl multiple URLs concurrently, with a maximum of 8 requests running at the same time. We also make sure that we do not send multiple requests to the same domain at the same time, and we maintain at least a 1-second gap between requests to the same domain.
Respecting robots.txt: because I’m polite 😂 so before crawling a website, we check its robots.txt file to see which pages the site allows crawlers to access. We fetch and cache robots.txt once per domain, then check it before each page request. If a URL is blocked by robots.txt, we don’t request it and record it as blocked_by_robots.
If we can’t access a site’s robots.txt, our crawler currently continues rather than blocking the site. This is a deliberate choice for my small, low-volume personal crawler. The crawler also identifies itself with a User-Agent and provides a contact email. If you see sara-taher-seo-crawler as user-agent, that’s me 😉
Errors: because every crawl will have errors. My idea is a failed page should not be left out from the dataset, nor it should interrupt the crawl. Every URL we attempt to crawl gets a record in the database, whether the crawl succeeds or fails. If something goes wrong, such as a robots.txt block, HTTP error, network error, or extraction error, we record the reason in the error field instead of silently dropping the URL.
Stack choices: I chose a relatively simple stack that gives me everything I need for this project. Here’s everything:

Limitations

Ok so I built my crawler and I ran it. I have the database ready for the next system “indexation” but there is something that I want to include but won’t right now as it will complicate things:

I want to include my video content in this database such that when you use the search engine to search for a topic, you can get videos back in the results. While video pages are included in the crawl, I didn’t download the transcripts for example.
I also want to include content I have written on third part websites, this is not included in the current version of this crawler.
Another thing is, how often should this crawler re-crawl the sitemaps to discover URLs? right now, it’s a one and done thing and to update the database with new URLs I have to manually run a crawl.

So these are few things that come to mind when working on this.

And That’s a Wrap (Almost 😄)

I really think this is a fun project and I will be recommending it moving forward to anyone who wants to learn SEO. We’ve always recommended things like build your own website, or audit online websites, but building a search engine should definitely come at the top of SEO learning tactics!

I know this is way technical that some of my readers would like, but it’s also a good change of pace, and it may open up ideas and inspire you to build things of your own. If you ever want to vibe code, reach out I’m happy to help!

That’s that for today folks and see you in the next newsletter!

Support the Riddler!

Sign up for my newsletter if you’re not already. (Pssst, you can also become a paid subscriber)
Share the newsletter and invite your friends to signup. Help me reach 2k signups on Substack by end of 2026 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

How Much Thinking Goes Into Your Content?

Sara Taher — Tue, 15 Sep 2026 12:46:47 GMT

Alternatively you can become a paid subscriber

Intro

Creating good content is not easy for many reasons, one of them is that we don’t know what good content looks like 😄

For example, some SEO news comes out, my entire timeline is talking about the exact same thing… sometimes I just hold off… do I need to say something just because everyone is saying something? would that be good content really?

And honestly, Googlers have been trying to help clarify what is good content. Just earlier this year I attended Google Search Central Toronto, and out of this event came the famous slide shared by Danny Sullivan about commodity vs non-commodity content:

And there’s also this checklist from Google that you can use to measure the quality of your content:

But still, I feel something is missing. That’s why today I want to introduce you to a new concept… new in SEO but definitely not new in general:

Bloom’s Taxonomy

what’s that? read below!

What’s Bloom’s Taxonomy?

In simple words, Bloom’s Taxonomy is a way of organizing different levels of thinking and learning, from simple to more complex.

It’s used by teachers and educators, but we’re not going to dig too deep into it today.

So what does this have to do with content creation?

Well, there’s a quote by Paul Graham that I will probably quote for life (quote to quote... sorry 😄):

“Good writing is just good thinking.”

And I think this is where Bloom’s Taxonomy and the concept of organizing different levels of thinking, from simple to more complex, becomes interesting for content creators.

Understanding the level of thinking required for different types of writing can help us create better content.

You know what? Let’s not talk about Bloom’s Taxonomy.

Let’s see it in action.

The Cognitive Domain Pyramid

Bloom’s taxonomy defines six levels of thinking. We can use those levels to define our very own six levels of writing! Also note the triangle shape, it’s because level one of thinking is easy, so it is more common, same with content: if it’s easy to produce, there’s probably a lot of it out there.

Level 1: Remember

If you’re writing at level 1, it’s basically recalling facts. You are retrieving established information whether from another source or from your own knowledge and writing about it.

I’d say this is similar to the skyscraper technique in SEO: you find what is already known, gather the information, and put it together in one place.

Example of writing at this level: “Meta descriptions are snippets that summarize the content of a webpage and can appear in search results.”

Level 2: Understand

If you’re writing at Level 2, you’re making sense of information you already know. You’re interpreting it, explaining it, comparing it, or translating it into a different way of thinking.

You’re not just repeating the fact. You’re showing that you understand what it means.

Example of writing at this level: “think of internal links like signposts on a highway: they help search engines discover and navigate between the different pages on your website.”

Level 3: Apply

If you’re writing at Level 3, you’re taking a concept, rule, method, or process you already know and using it in a new situation to solve a specific problem.

Example of writing at this level: "For this local plumber’s landing page, we can apply buyers journey principles by focusing the page on emergency plumbing services to attract conversions rather than writing a general article about plumbing."

Level 4: Analyze

If you’re writing at Level 4, you’re breaking something down into its parts and looking at how those parts relate to each other.

You’re not simply reporting what happened. You’re looking for patterns, relationships, differences, and possible reasons behind them.

Example of writing at this level: "This website lost traffic not because of AI Overviews. A YoY comparison shows a reduction in the number of pages and content created."

Level 5: Evaluate

If you’re writing at Level 5, you’re making a judgment based on data and specific criteria or standards.

This is where you move from “here’s what the data says” to “here’s what I think we should conclude from it, and why.”

Example of writing at this level: "Based on our latest Search Console data, our current content strategy is failing because, while overall traffic increased, the average clicks/page is still very low”

Level 6: Create

If you’re writing at Level 6, you’re combining knowledge, ideas, and information to produce something new.

This is the highest cognitive level, so it generally requires more original thinking and effort, which aligns with the content-effort measure Google uses to assess content quality.

Example of writing at this level: "After studying how LLMs retrieve and cite information, we built an interactive SEO tool that lets users analyze the sources behind an LLM answer."

Note: I asked chatgpt to assess this article and assign a bloom taxonomy level, here’s what it came back with:

There are elements of Understand, Apply, Analyze, and Evaluate throughout, but the overall intellectual work is synthesis: you took concepts from education, SEO, Google’s content guidance, and your own experience and combined them into a framework that didn’t previously exist in this form. That makes the article itself a strong Create-level piece, which is also a nice real-world example of the argument you’re making.

And That’s a Wrap (Almost 😄)

Ya so when writing content, it’s good to see what level of thinking you are putting into your content. Here’s a short summary that you can save somewhere to always assess yourself!

Remember: I know this.
Understand: I can explain this.
Apply: I can use this.
Analyze: I can break this down and find relationships.
Evaluate: I can make a judgment about this.
Create: I can use what I know to make something new.

I hope I gave you some inspiration on what is good content, you can also integrate this framework into your content automation workflows if you’re automating your content production.

That’s that for today folks and see you in the next newsletter!

Support the Riddler!

Sign up for my newsletter if you’re not already. (Pssst, you can also become a paid subscriber)
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2026 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

I Opened the Network Tab. Here’s What ChatGPT Was Doing.

Sara Taher — Tue, 08 Sep 2026 12:38:31 GMT

Alternatively you can become a paid subscriber

Intro

SEO is not dead, but AI is definitely a big part of it now. Clicks are not dead either, but they do not really fully reflect/capture the value both traditional search and AI platforms bring to your businesses.

I mean look at brand mentions. Your brand can be recommended, users can see it, later go to google and perform a navigational search to find your website. Or if they’re already familiar with your website (e.g amazon) and realize there’s no better place to get what they need, they can just go back via direct traffic. You may also end up getting traffic as a referral from one of the cited sources.

For example:

I monitor my newsletter and brand status on different AI platforms. I recently checked ChatGPT for “top indie seo newsletters” and I came out on top with two cited sources, none of them is my website. People will see my website name and either go to one of those sources and look for a link to my newsletter, or manually search Google for “SEO Riddler Sara Taher” for example to find my website. So people will come to your website via organic channels in new ways that mere clicks may not account for.

I think this is a big reason why we have not yet seen many people talking about how much ChatGPT is sending traffic to their websites.

So yeah, definitely fun times in SEO! And because of how important getting mentioned/cited in ChatGPT is becoming, this brings us to today’s newsletter topic: what happens when you prompt ChatGPT?

Let’s dissect that… not in a theoretical way, but what I actually saw happening!

The Network Tab

I really want everyone to play around with this, so here’s what to do. Before prompting chatgpt, right click and select “inspect”.

Next you write your prompt, it can be anything really for me I typed “best crochet yarn for beginners in canada”

You’ll see things starting to populate in the inspect window that you opened, click on network, find the conversation, then click on EventStream.

You can always copy and paste this to chatgpt or claude to explain to you everything! This object captures the moment chatgpt decides it needs to search the web for your query.

The Conversation Anatomy

When you look at the text in EventStream, you get something like this:

{
  "v": {
    "message": {
      "id": "2cc91a19-092d-4735-a0d4-3c6581736af9",
      "author": {
        "role": "tool",
        "name": "web.run",
        "metadata": { "real_author": "tool:web.run" }
      },
      "create_time": 1788831562.9684362,
      "content": {
        "content_type": "text",
        "parts": [""]
      },
      "status": "finished_successfully",
      "metadata": {
        "search_model_queries": {
          "type": "search_model_queries",
          "queries": ["best yarn for crochet beginners Canada worsted weight acrylic cotton Bernat Caron"]
        },
        "reasoning_titles": ["Searching best yarn for crochet beginners Canada worsted weight acrylic cotton Bernat Caron"],
        "reasoning_status": "is_reasoning",
        "tool_icons": ["globe"],
        "reasoning_start_time": 1788831561.9985478,
        "model_slug": "gpt-5-6-mini",
        "model_adjustments": ["auto:smaller_model:reached_message_cap"],
        "parent_id": "d5410723-6ee6-4435-938e-1558808447f0",
        "turn_exchange_id": "6a85u2bb-37cc-1155-a3b4-1625056f1268"
      },
      "recipient": "all"
    },
    "conversation_id": "6a9f673e-e3b8-83e9-8d7d-5b48e139ae34"
  },
  "c": 9
}

so what is happening here? This conversation happens everytime chatgpt decides it needs to search the web during an interaction with the user.

This conversation shows:

The author: can be the user (you), the AI, or a helper tool that the AI can ask to do things for it like searching. This shows that the AI doesn’t search the web itself. It basically says “hey search tool, go look this up for me” and then waits for the answer. Why is this important? it shows that information retrieval (IR) is a separate algorithm than the LLM, optimizing for IR can be different and Google/Bing-like compared to LLMs.
search_model_queries: THIS is the actual useful information, because it’s the exact words the AI typed into the search engine tool it is using. In our example, the AI typed “best yarn for crochet beginners Canada worsted weight acrylic cotton Bernat Caron” so it included most popular brands and yarn types as well.

Interesting note: I tried prompting “best icecream stores near me” and do the same analysis, turns out chatgpt has a separate search tool for local queries called “map_search_model_queries” instead of the “search_model_queries” I mentioned above.

And That’s a Wrap (Almost 😄)

I really hope this newsletter edition helps you explore ChatGPT and how it works closer. And it’s not just about the query fan outs.

As you can see, the query sent by ChatGPT to search the web can include pre-decided brand names, or specific product types for example.

I always knew that search is a separate functionality/algorithm than the LLM itself. Getting cited in an LLM relies on whether this system can find you. Definitely something we should keep in mind when thinking of optimizing for this new era!

That’s that for today folks and see you in the next newsletter!

Support the Riddler!

Sign up for my newsletter if you’re not already. (Pssst, you can also become a paid subscriber)
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2026 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

Is Your SEO Strategy Agile?

Sara Taher — Tue, 01 Sep 2026 13:52:18 GMT

If you find value in my newsletter and want to support the work that goes into it consider becoming a paid subscriber!

Intro

I’ve been super vocal about what a good SEO strategy should look like. And one of the things you’ll see often is a 12 months SEO checklist shared with stakeholders as an SEO strategy.

[checkout my SEO strategy course]

I guess stakeholders want some stability in what to expect is coming next, but also, it needs to be super clear that SEO is dynamic with new things happening and Google updates to search itself and to websites ranking criteria happen all the time.

Not to mention a lot of experimentations are happening to improve visibility in LLMs and we’re basically learning new things every day.

So whatever you present as an SEO strategy, there needs to be an understanding that this strategy is agile, and will adapt to whatever new thing comes our way. Or else, we will be living in the past.

A new report

I was reading a recent report saying that B2B buyers have started to embrace LLMs. But not just any B2B buyers, small and medium size buyers. The use LLMs to create a vendor list, and then take this list to do their own further research on other platforms [source].

What does that look like?

Bob, VP of Operations at a 50-person SaaS company, asks ChatGPT: “What are the best project management platforms for a growing B2B software company?” The LLM recommends five vendors, and Bob narrows his shortlist to three. He then visits their websites, checks G2 reviews, and watches YouTube demos before booking a few sales calls.

What does that mean for SEO?

If you’re optimizing your blog and landing pages pages content hoping this would help new users discover you and learn about your products, you maybe missing out a lot. Because users at the awareness (or discovery stage) may not be starting at google search at all, but rather on LLMs.

I mean here’s another simple example to demonstrate a real life scenario:

I was playing around with ChatGPT’s “fan-out” searches and noticed something interesting: the brands it recommends are already part of the search—not something it discovers from the websites it finds.

So, if you ask ChatGPT for “top SEO newsletters,” it already has a set of brands in mind. The additional searches are mostly gathering information about those brands or validating its choices.

In other words, the shortlist comes first; the search helps justify or verify it. That shortlist is influenced by what the model has already learned from its training data.

Adjust your strategy to the new funnel

The buyer journey funnel for b2b buyers changed a bit 😄

The buyers in the study were evaluating solutions with an average annual value of approximately $250,000, and 94% of those buyers reported using LLMs during their buying journey. [source]

So this means you need to yes continue thinking of search, but optimizing for LLMs visibility should take most of your attention. Ranking in bing is actually more important for you than ranking in Google, since chatgpt uses Bing.

The good news is Bing gives us a bit more data in it’s Bing webmaster tool dashboard, so that should be fun!

The next thing that should be a solid part of your strategy is how you get mentioned on third party websites, your social presence and if you’re doing enough of those to begin with. Because that’s were b2b buyers go to verify and do their own research, so you gotta be ready for them.

While these two things seem simple to understand, execution is where you separate the men from boys. It is a complete directional shift, and a good SEO will come up with a good list of tactics to execute on those two new directions.

If your SEO is still doing SEO for your B2B thinking only about Google, they’re still using the old playbook, in this new new world (yes new new 😄)

And That’s a Wrap (Almost 😄)

Agility is essential to success as much as doing the SEO right tactics. The last thing you want to be doing is sailing your ship at the best speed in the best possible way, only in the wrong direction.

I always say understand who the customer is, in-depth. Only then you would be able to create a tailored SEO strategy for them.

Ya… a mix of agility, and knowing your customer well would take your SEO strategy a very long way!

That’s that for today folks and see you in the next newsletter!

Support the Riddler!

Sign up for my newsletter if you’re not already. (Pssst, you can also become a paid subscriber)
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

Is Meta Description Important for AI Search?

Sara Taher — Tue, 25 Aug 2026 12:15:56 GMT

If you find value in my newsletter and want to support the work that goes into it consider becoming a paid subscriber!

Intro

I had plans to talk about a research paper today, but got excited for an SEO experiment and I think you will too!

A lot of SEO research is being done right now which I really appreciate, and some of it is around meta descriptions. So I decided to contribute to the discussion and run my own quick test.

So bring your coffee and lock in!

The setup

I asked chatgpt (of course) to create content for a fictional coffee shop called “the open book”. Thanks chatgpt for saving my time into writing copy!

Then I tool this content, went to my other dear friend “claude code” and asked it to create a landing page with this text and push it live - done✅

Then, I asked claude code again to add a fictional name, address in Japan, and a random phone number information in the meta description. This is the only place this information is available on the page - done✅

I also added a title tag and a canonical.

You can check out the fictional bookstore here: https://www.sara-taher.com/the-open-book

Now let’s see if different LLMs can read the NAP (name, address, phone number) information in the meta description!

What Chatgpt said

I asked chatgpt “where is this book store located and who owns it: https://www.sara-taher.com/the-open-book” but of course it had to act out like smarty pants and say this is a fictional bookstore, bla bla bla because it created the content and I was logged in.

So I logged out and asked it again, here’s what it said:

“I checked the page you linked, and there’s an important wrinkle: I can’t verify that “The Open Book” is an actual physical bookstore with a publicly disclosed location or owner.”

It also said that:

“Location: Not disclosed on the page, and I couldn’t find a reliable external listing establishing a physical address.

Owner: Not identified as the bookstore’s owner. The website is Sara Taher’s personal site, but that alone isn’t evidence that she owns an actual bookstore. “

Verdict: Chatgpt seems to not read or consider content in meta descriptions.

Note: apparently chatgpt was trying to load an image or a map but couldn’t

What Claude said

I asked claude the same question, and claude said it could not find this information on the page:

“I fetched the page, but I want to flag something: the visible page content for The Open Book doesn’t actually mention an address or an owner’s name anywhere — it’s just marketing copy about the bookstore/café concept (menu, events, FAQs, etc.).”

However, it said that this information is available in the meta description!

Verdict: Claude seems to read/consider content in meta descriptions.

Note: this is an interesting finding since just recently I came across research that Claude - while being perceived as more trustworthy - has the second highest level of inaccuracies. Well, it definitely is thorough though.

What Gemini said

I asked Gemini the same question, and here’s what is said:

Basically Gemini did not check the meta description for this information, AND it seems like it’s trying to connect the “open book” bookshop to existing entities it already knows about. I think this is super interesting!

Verdict: Gemini seems to not read or consider content in meta descriptions.

What Google search & AI Mode said

I searched for the same query on Google (try it yourself and let me know if you get any different response): where is this bookshop located and who owns it: https://www.sara-taher.com/the-open-book

I- of course - got an AI overviews, when I expanded it, it took me to AI mode and overall I think this is the worst and more irrelevant answer! It started looking for books I mentioned in another irrelevant article 🤷‍♀️.

Verdict: Google search/AIO/AI Mode seems to not read or consider content in meta descriptions.

What Perplexity said

I asked perplexity the same question, and I’m very concerned at the confidence level it responded with 😂😂😂 It just assumed this is the same bookshop as another bookshop it has in it’s statistical prior or something.

and you can see it referenced 27 sources, when I checked, none of them is the URL I provided, although it did check my website homepage 🤷‍♀️

Verdict: Perplexity seems to not read or consider content in meta descriptions.

And That’s a Wrap (Almost 😄)

I really had so much fun working on this small text. It seems that different models have different approaches to how they handle meta descriptions and reading the HTML. Something I have been saying alone in this industry (or so it feels).

Back to our question, is meta description important for AI search? overall no, maybe if you’re optimizing for claude. But It’s not a matter of importance, it’s really a question of do ai search bots read the meta description tag?

So that also brings us to another interesting thought, some models maybe considering the entire html of the page and some may not, some extract specific elements (e.g schema or meta descriptions) and others may not be doing that at all.

Would that work if the html is broken? that’s a totally different question that cannot be answered by this simple test but I have an answer and so many related ideas that I’ll probably discuss in one of my upcoming newsletter editions (ya! more to come).

That’s that for today folks and see you in the next newsletter!

Support the Riddler!

Sign up for my newsletter if you’re not already. (Pssst, you can also become a paid subscriber)
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

Built to Collapse!

Sara Taher — Tue, 18 Aug 2026 12:50:35 GMT

If you find value in my newsletter and want to support the work that goes into it consider becoming a paid subscriber!

Intro

Publishers are fighting back in a battle of humans against bots...

They have been complaining extensively about how AI platforms and Google’s own AI features are using their content and, ultimately, taking their traffic.

There is also a clear divide among publishers on whether they should allow AI bots to access their content at all. Some are blocking them, while others are allowing access, and so far, there is no clear strategy or agreed-upon path for how publishers should tackle this.

And if you haven’t heard, just last week, more than 300 French newspapers filed a complaint against Google over AI Overviews, less than a month after the feature was rolled out in France.

Seems like publishers are hoping for governments to protect the industry. But things are not that simple, are they?

The challenge isn’t just about LLMs stealing content and serving it to users. Journalists and reporters are also competing against AI-powered newsrooms, some of which are completely human-free and fully operated by AI.

An AI news publication called RuntimeWire beat WIRED and other popular news publications by posting stories THREE HOURS faster! And in the world of news, time is of the essence… so you can imagine how big a deal that is.

That should be a fun and interesting one to look into, so buckle up!

The Tea

At the recent Black Hat security conference in Las Vegas, OpenAI shared new details about a hacking incident involving its own rogue AI agents. They have been talking about those incidents very proudly, by the way 😄

Anyways, as usual, after the event, reporters raced to publish, and WIRED was among the fastest. But an outlet called RuntimeWire beat it by more than three hours.

THREE WHOLE HOURS.

RuntimeWire had no reporter at the event. Actually, they have no reporter at all. It’s an AI newsroom, where AI bots scout the internet to find stories they can write about.

In around 3 months, RuntimeWire posted more than 2,000 stories by scanning court databases, forums, company filings, and social media for tech news. This AI newsroom costs $100/day to run.

On the day of the Black Hat security conference, Ryan Merket, the founder of RuntimeWire, spotted an OpenAI executive posting about the talk on X. Merket fed a transcript of the livestream to his AI agents while the session was still underway, and RuntimeWire published about six minutes later.

Voila! Or is it? Is this the death of real journalism? Should we be worried?

The Truth

It’s easy to see news like this and say, “Oh great, one more job category AI is going to take over”…. but that’s far from the truth.

While this is a successful experiment to some extent, let’s not forget the facts:

The quality is not comparable to real journalism.
There will be incidents where real journalism that is present live covering an event cannot be simply replaced by an AI reporter.

Then there’s also the data… as a curious SEO, I decided to have a look at how RuntimeWire performs in search. It all comes down to two questions:

1# Will people like AI reporting?

The data says there’s just momentary interest in the brand, probably related to news about the brand rather than stories reported by the brand itself!

2# Will bots like this content?

Will fellow bots like the content generated by one of their own? 😄 The irony…

We know that Google isn’t against AI-generated content per se. However, we also know that this content is low effort to generate with minimal human intervention and real-world experiences (bots cannot live life 😄)… it’s the definition of commodity content… you know, the type of content that is just canned and mass-produced, that looks good at face value, but offers no actual value over any other page on the internet.

A simple search for the brand name shows that there aren’t even sitelinks for this website. If anything, it goes to show that none of the pages come off to Google as important or popular to users.

Compare that to WIRED in search? There’s a Knowledge Panel, there are sitelinks, there’s “Latest from WIRED,” and more…

If you also look at the traffic, the website received around 100k visitors in the last month, despite having over 2,000 articles. That’s 50 visitors/article. Also, it does not seem to be successful in search. Most of its traffic is from organic social (Reddit and Facebook). All data is from Similarweb.

The top referring website is news.ycombinator.com, which is a platform where you can submit your own stories, really, so no magical tricks here or anything that cannot be replicated or, even better, outperformed…

My 2 cents is, the website idea is cool, and is good enough for people looking for a very high-level surface read into the news. It’s a battle of brand and distribution, not quality at all.

While these kinds of websites can see some success, I don’t see them winning big. They’re definitely an annoyance to big news publishers, but nothing that cannot be addressed.

And That’s a Wrap (Almost 😄)

I think the biggest lesson here is to always pause and think, and understand the play you’re up against.

Creating awesome content is great, but relying just on that is like saying, “Create great products and people will come.”

In news, I don’t see AI reporters outperforming real, experienced reporters or publishers that have been in the game for a long time. Yeah, sure, AI reporting can be faster, so they have a few tricks up their sleeves. However, that is something all publishers can learn from and use.

Any traffic AI newsrooms are capturing today is because of gaps big publishers have left. Are you fast enough? Are you distributing on the same platforms? Are you branded right for your audience?

Nope. AI is still far from replacing humans in journalism.

That’s that for today folks and see you in the next newsletter!

Support the Riddler!

Sign up for my newsletter if you’re not already. (Pssst, you can also become a paid subscriber)
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

The Simple Way to Check AI Overview Visibility in Bulk

Sara Taher — Tue, 11 Aug 2026 13:20:57 GMT

Alternatively you can become a paid subscriber

Intro

We talk a lot about optimizing for better visibility in AI, and there’s tons of content being shared, but one of the questions I get asked in my community calls is how to understand where you stand today in AI?

Personally, every once in a while I have someone reach out to me, and they say they found me from an LLM platform.

I just don’t share those very often so not to jinx it 😄😄😄

Anyways, for today I decided to put together a simple process for you to understand where you stand today in your AI visibility in AI Overviews.

Note: this is not an audit to find fixes, it’s an audit to observe how and when do LLMs cite you, and recommend you.

I did that before

I recently created the data study “Who Does Google Trust in AI Overviews for SEO Advice?”

and it all started with a simple data export of a bunch of related keywords (thousands)

Then I did some cleanup, used an API to get AIO for each of the keywords, and analyzed those answers in bulk.

I really recommend you check out this study so you can imagine what you can do if you follow my process. Works wonders when you begin with the end in mind.

Let’s get to it

So there are two thing you want to do:

how often, and in what situations is your brand cited and/or mentioned
who does AIO cites and mentions in general (competitor analysis)

The steps are simple.

The keywords list

First, pull a list of keywords that you want to analyze your visibility for. You can get those keywords from GSC, or you can get them from any of the third-party SEO tools.

Once you have your list looking something like this:

The cleanup

If you pulled a large list of keywords, it’s always a good idea to sift through it to remove any noise. For example, you may want to remove competitor brand names (or not depending on what keywords are important to you to analyze), keywords with Non-Latin letters for example. API calls can be expensive if you’re doing them for a large number of keywords, so cleanup is important to reduce the cost, but also, bad or noise keywords will reduce the quality of your report.

The best way to do this cleanup is a mix of both manual checks and using claude or chatgpt for more large scale automated changes. But always ask them to show you examples of any keywords they removed. Double check, double check, double check.

The API calls

There are a lot of SEO APIs out there, and I have been using SERPapi and the good news is that you can create a free account and get 250 free search per month (not sponsored)!

250 searches are not a lot, but can be a good place if you’re working on a small audit for your website.

I created this free SEO python script for you, once you run it, it will ask you for your API key, which you can get from SERPapi as I mentioned above. Then it will ask you for a CSV that has your keywords. It needs to be called “sample_keywords” and the keywords are in the first column that’s called “keywords” all lowercase. here’s the sample input and output file that I used for the purpose of this blog.

For each of the keywords on your list, the output will look like this

You have the keyword, the AI Overviews, and all the URLs in serps as well. Looks good right?

Note: before running the API on thousands of urls, make sure you run the code on a small set of keywords to test it’s doing what you would like it to do.

The analysis

This is what separates men from boys, or great SEOs from just SEOs. Using chatgpt or claude (I personally used claude code) I would ask it to analyze this output data. For every analysis it should save the output, as we will need to create an HTML report in the end (similar to the report I created).

Note: If you think this is a quick and easy task, you’re probably doing it wrong. You need to double- and triple-check the output for every question before moving on to the next one.

I also recommend asking Claude or ChatGPT to write code to analyze the data, rather than having them analyze the dataset directly. This will give you more consistent results, save you money, and, perhaps most importantly, make it much easier to audit the analysis when you understand exactly how the results were generated in the first place.

Some of the analysis questions you can perform:

How often was my brand cited compared to my competitors (provide the list of competitors and of course your brand name)
How often does being cited or mentioned overlaps with ranking and in what positions.
When your brand is cited, what was the sentiment? perform a sentiment analysis for each time your brand was cited or mentioned in AI Overviews to understand was is cited in positive, negative or neutral tone.
What type of queries trigger my visibility (is it informational? is it when a specific feature is mentioned, etc…)
What type of queries are we lagging in visibility (same question as previous one but looking to the other half of the glass)

the sky is really the limit… and this is where you shine!

Finally combine all this data into an hmtl report because it’s easier to format and you have a very cool report that you can use to drive optimization initiatives in your organization.

And That’s a Wrap (Almost 😄)

I may have made this task sound simpler than it actually is, I know. But that’s not necessarily a bad thing.

Well… I hope I made it simple enough for you to test out and have fun with!

That’s that for today folks and see you in the next newsletter!

Support the Riddler!

Sign up for my newsletter if you’re not already. (Pssst, you can also become a paid subscriber)
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

How to Learn SEO in 2026?

Sara Taher — Tue, 04 Aug 2026 13:23:03 GMT

Hello Riddlers!

📌 If you find value in my newsletter and want to support the work that goes into it consider —> [☕Buy Me a Coffee☕]💡 Want more from me? [SEO Strategy Course] | [Build Your Own SEO Tools (with Python & Agentic AI)] | [Premium SEO Mastermind]

Alternatively you can become a paid subscriber

Intro

I run monthly live SEO townhalls to chat about anything and everything SEO, you can also bring your own questions or tips that you’d like to share. You can watch recording of previous sessions here, or signup and join us for the next one!

These sessions always inspire me, and last time someone asked, “How do you learn SEO in 2026?” That really got me thinking...

We’ve come a long way. SEO has evolved into a vast ocean that requires continuous learning if you want to keep sailing (such a dramatic analogy, right? That's totally me, not the AI helping me write this post 😂).

AI is also a big part of this evolution, so I thought it was a great question to answer in this newsletter. So, hold tight!

What is SEO today?

First, I want to set the stage to what learning SEO includes today and the areas of knowledge you should pick up. I, by no means intend to overwhelm you, we are all still learning. But it’s important to begin with the end in mind.

SEO imo covers a lot more area than we used to. Auditing reddit and finding brand interaction opportunities for your social team, or auditing your YouTube Channel and optimizing the content and descriptions, not just to rank on YouTube but also to rank those videos better in search, and get your brand cited in AI answers using those videos (YouTube gets cited 25% of the time in AI answers).

You may be wondering at this point… OMG do I need to learn all of this? not right away… you’ll build your knowledge overtime, with every audit and every client and every new SEO article you read… but you gotta cover the basis.

1# Cover the basis

People in this industry would eat me alive if I said anything before saying this: the fundamentals are still the fundamentals. If you want to learn SEO in 2026, you still need to master many of the same concepts you had to learn years ago.

Now that I’ve got that out of the way, here’s what I recommend:

If you can build your own test website, please do. It costs little more than a domain name and about $4/month for GitHub, and you can build your own website from scratch using Claude Code. This is the best learning environment—better than any course I can recommend.

If you don't want to build a website, find a small website you'd like to audit instead.

Next I recommend you go through Google’s SEO Starter Guide. As you read and learn, keep going back to your website and apply what you've learned.

For example, the documentation talks about the site: search operator (e.g., site:wikipedia.org). Go try it in Google Search and see what happens.

Don’t understand something? The good news in 2026 is that you can ask ChatGPT, or join one of the many SEO communities and ask your questions there. I also run an SEO community, apply to join here.

In my opinion, Google's documentation is the best place to start. If you're looking for something more comprehensive that will set you up well, I'd also recommend the SEO specialization certificate on Coursera. By the end of it, you'll have the knowledge expected of a junior SEO.

2# Quality content

Google recommends creating “helpful content”... but what exactly is helpful content?

One of the best places to start is Google’s own “Search Quality Rater Guidelines: An Overview”

I highly recommend reading it and creating your own content audit checklist as you go. It’s a great way to turn what you’re learning into something practical that you can reuse.

You can also upload the document to NotebookLM. It can generate a podcast based on the file, and you can ask questions about the document and get answers grounded in its content.

3# Uncle Tom

What about doing SEO for platforms like ChatGPT and Claude?

Well, some people call this GEO...

And if you haven’t heard, using the term GEO is very inflammatory for some people in the SEO industry. Is SEO the same as GEO? Is that even the right question? The debate goes on and on...

I don’t really get into that debate anymore. Call it whatever you want—you can call it Uncle Tom for all I care. 😂

I highly recommend you devour this resources to get on top of this field:

The iPullRank AI Search Manual

That’s it. That’s my advice!

4# Signup for the right resources

LinkedIn is great, but newsletters are even better. Here are a few newsletters I recommend. I’m not affiliated with any of them (except, of course, my own 😉).

SEORiddler.com: a newsletter by yours truly, it’s not really a beginner level newsletter, but it opens your eyes to new ideas, and shared actionable tips as well.
seroundtable.com: that’s where I go for news, you’ll find every single thing that ever happened any day in SEO there, that’s a given.
Mike King’s iPullRank newsletter: big topics, in-depth AI/SEO research, and so many gems in there.
Tech SEO Tips Newsletter: this one is by Nikki Halliwell, I had her on my podcast one time and she’s awesome and her newsletter is no less if you want to build a good technical SEO foundation.
SEO Notebook by Steve Toth: short actionable tips on SEO and GEO, definitely worth a spot in your inbox.

5# Wash, rinse, repeat

If you feel the resources I recommended above are not enough, you’re right—you need practice. That’s where you truly grow and learn.

Audit more websites. Audit big websites too: Amazon, Wikipedia, CNN, and others.

Learn to find patterns. Every time you use Google, ask yourself: Why is this URL here?

And That’s a Wrap (Almost 😄)

I hope I didn’t overwhelm you—please don’t be! We’re all learning, and anyone who says otherwise is simply not being honest.

Take baby steps, learn as you go, and try to find a mentor who can guide you if you can.

Good luck!

That’s that for today folks and see you in the next newsletter!

Support the Riddler!

Sign up for my newsletter if you’re not already. (Pssst, you can also become a paid subscriber)
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

Google Might Be Learning the Facebook Lesson

Sara Taher — Tue, 28 Jul 2026 13:30:11 GMT

Hello Riddlers!

Alternatively you can become a paid subscriber

Intro

Google has been facing a lot of criticism and, in many cases, deservedly so.

First came AI Overviews. Then AI Mode.

Features that few people were asking for, but Google had to prove it wasn't falling behind in the AI race after OpenAI launched first using the Transformer architecture that originated from Google's own research.

And the data reflects the sentiment.

Since May 2024, searches from people looking for ways to turn off AI Overviews have been steadily increasing. Just facts. The trend peaked around April–May 2026.

Maybe the people actively searching for ways to remove AI Overviews are a minority. But I suspect many more are simply quietly dissatisfied.

You know who else is dissatisfied? Publishers.

I don't think I've ever seen this level of audacity toward publishers. AI platforms are using content they didn't create to train their models and perform live searches, then assembling answers that compete directly with the very sites that produced that content, taking away the clicks those publishers depend on.

Google says it’s putting users first. I’m not convinced.

To me, the goal isn’t just to help users find information really. It’s to keep users inside Google’s own ecosystem for as long as possible. That’s what this is really about.

And let’s not forget...

Google spent years lecturing all of us about E-E-A-T.

Publishers, SEOs, and content creators have been told to create “helpful content” that demonstrates Experience, Expertise, Authoritativeness, and Trustworthiness. We audited pages, rewrote articles, added author bios, cited sources, and jumped through every hoop because that’s what Google said good information should look like.

Then Google turned around and gave us AI Overviews.

A system that has repeatedly produced inaccurate or misleading information.

I could pause here and start talking about OpenAI and Anthropic. They aren’t exactly better when it comes to how they use other people’s content either.

But to keep this conversation focused, I’ll stick with Google today.

I think Google is up for a wakeup call.

Even tech giants can stumble, and history has a funny way of repeating itself in unexpected ways.

A challenging echo system

A lot of businesses have been impacted over the years with Google updates and a lot of websites lost traffic but none of it is equivalent to what we are seeing today.

Reddit is reportedly considering not renewing its $60 million deal with Google, which allowed Google to access Reddit content to train its AI models.

Reddit’s stated concern is that Google isn’t sending enough traffic back to the platform.

That argument is a little hard to take at face value, considering Reddit has become one of the most visible domains in Google Search results. Reddit threads appear everywhere now, often ranking prominently for product reviews, recommendations, and real user experiences.

And then we have publishers. Google said that it “sends billions of clicks to websites every week through AI search.” but also does not provide the data to verify this in GSC, and publishers are seeing declines in their traffic, the traffic they rely on to generate revenue through ads, subscriptions, and other business models.

Hence, a lot of publishers are considering blocking Google.

And just as Google is facing all of these challenges, SerpApi out of nowhere gets a Google lawsuit dismissed.

You see, Google had the audacity to sue SerpApi for copyright infringement because SerpApi was crawling Google.

Yes, you read that right… do not laugh, please. 😂

Google literally crawls every single website on the internet.

At least when SerpApi crawled Google, they weren’t taking clicks and revenue away from publishers.

Justice was served, and the court ruled that there was no copyrighted material to begin with. 😂😂😂

So some users are not happy, publishers are definitely not happy, Reddit is pushing back, and Google SERPs have been declared as non-copyrighted.

My 2 cents: as these incidents continue and more pressure builds on Google, something has to give. Google will eventually have to find ways to compensate publishers… right?

You may say, “Sara, you’re a small content creator. What do you know?”

Fair enough.

But I do know a thing or two from my startup days.

Hear me out.

Once upon a time

Do you remember when Facebook reduced the organic reach of pages from around 25% to somewhere between 1%–5%?

The goal was clear: push more businesses toward paid advertising and increase ad revenue.

But what happened over time?

Content creators slowed down. Pages became less active. The platform became far less engaging and, at times, almost felt like a digital desert.

I remember back in 2015 suggesting that Facebook should allow people to make money from creating great content on pages.

Without active publishers and creators, platforms risk becoming less engaging and less valuable because the content that keeps users coming back depends on those creators.

and turns out, that wasn’t such a crazy idea.

It took Facebook years to realize it, but eventually they started paying publishers and creators to keep producing content.

Facebook paid out nearly $3 billion to creators in 2025. Of those $3 billion in payouts, 60% went to Reels while the rest went to Stories, photos and text posts.

The lesson? Platforms need creators just as much as creators need platforms.

Maybe Google isn’t there yet, maybe they need to go through the same journey as Facebook to realize that. But they will eventually get there, if publishers and creators are not happy, the platform slowly starts to die.

And That’s a Wrap (Almost 😄)

If you’re a publisher, try to navigate this challenging period.

Because there will come a point when Google and other platforms realize that publishers are not just content suppliers, they are essential partners in their success.

And at some point, they will have to find a way to give back as much as they take.

That’s that for today folks and see you in the next newsletter!

Support the Riddler!

Sign up for my newsletter if you’re not already.
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

First They Came for the Clicks. Now They Want the Checkout.

Sara Taher — Tue, 21 Jul 2026 13:14:09 GMT

Hello Riddlers!

Alternatively you can become a paid subscriber

Intro

Everyone seems to be angry at Google right now. Just yesterday, I came across the latest piece from The New York Times titled “Google Is Building an A.I. Fence Around the Internet It Once Championed.”

And, of course, even before this article was released, we had already heard statements from Google representatives emphasizing that Google continues to send significant amounts of traffic to the web. (yes Google we believe you 😂 and that’s exactly why you don’t want to show us the proof in GSC 😂)

Just last Friday, Google’s Head of Search, Nick Fox, wrote on LinkedIn that Google is “sending billions of clicks to websites every week through AI features in Search alone.” He added that these AI features were designed with this goal in mind: “We’ve designed our AI features in Search to connect people to websites.” Overall, he stated that Google “continues to send billions of clicks to the web every day through Search” and that people are using Google Search “more than ever.”

Similarly, last year, Google Search VP Liz Reid said that Search was still sending a “relatively stable” number of clicks—again, in the billions—and argued that reports showing declining search traffic relied on flawed methodologies.

The interesting part is that the conversation is no longer just about how many clicks Google sends. A billion clicks can be a very reassuring number until you start asking for the missing clicks data in GSC.

Well, in this sea of pessimism, I might actually have something potentially good news for ecommerce websites.

So buckle up!

First they came for the clicks, now they’re coming for the conversions

Publishers are upset because Google is taking away clicks from their websites, clicks they earned by creating great content and playing by the rules, clicks they need to generate ad revenue.

Ecommerce businesses, on the other hand, may have different concerns: how their products get recommended across AI surfaces and Google’s recent efforts to capture purchase clicks by offering users the ability to complete checkout directly within Search.

Just last year, ChatGPT announced that users could buy directly from U.S. Etsy sellers within chat. And in January, Google announced tools through the Universal Commerce Protocol that allow shoppers to complete purchases directly within AI Mode and the Gemini app.

I can imagine many concerns around this. One of them is that when people purchase directly from your website, they may discover other products they weren’t originally looking for and add them to their cart. Moving the purchase experience to a third-party platform — whether ChatGPT, Gemini, or AI Mode — gives merchants less control over the overall shopping experience.

For many retailers, this could be a significant challenge. The traditional ecommerce experience allows brands to cross-promote products, recommend complementary items, and encourage customers to explore more of their catalog. In a third-party checkout experience, products may be listed and purchased individually, without the same opportunities or control to guide customers toward other items or offers from the same seller.

This could also be seen as another example of what the New York Times article I shared earlier described: “Tech companies have built empires that locked users in and were focused on maximizing profit instead of enhancing public utility.”

So, as you can see, the concern has evolved. First, they came for the clicks. Now, they’re moving toward controlling the conversions by keeping users within their own platforms which raises an important question: how much control will merchants have over the customer journey in the future that Google and other platforms are building?

I can hear you saying… Sara, where’s the Good news you promised?

Quick historical analysis

I’ve been digging into the history of innovation in ecommerce trying to find patterns for an innovation strategy. Here’s one thing I noticed and I’m sure glad I did:

Does anyone remember Pinterest’s “Buyable Pins”?

Pinterest originally launched in-app purchasing through “Buyable Pins” in July 2015. However, the feature was officially retired globally in October 2018. Instead of keeping purchases within the platform, Pinterest shifted toward Product Pins, which redirect users to the retailer’s website to complete their purchase.

Then there was “Checkout on Instagram.”

Instagram launched its in-app checkout feature in March 2019, allowing users to complete purchases without leaving the app. However, Meta retired the feature globally in April 2024, moving back toward product tags that send users to the retailer’s external ecommerce store to complete their purchases.

So historically, we have two examples of major tech companies attempting to own the checkout experience and both eventually moved away from it.

Why?

Because users don’t just want to buy a product; they want to experience the brand they are buying from. They want to understand shipping times and costs, learn the brand’s story, decide whether they trust the company, explore other products, discover promotions, and so much more.

If there’s one thing this history teaches us, it’s that when users have an established way of doing things on a platform - like shopping on the web- changing that behavior is incredibly difficult.

The better opportunity may not be replacing existing habits, but building solutions around them: reducing friction, improving discovery, and making the existing customer journey easier rather than trying to completely reinvent it.

If you’re an ecommerce website, congratulations it seems you are safe for now. Your focus should be on optimizing your platform for conversions, removing any friction and simply making every click to your website count!

And That’s a Wrap (Almost 😄)

History is not boring! It’s actually one of the most fascinating places to look for insights, if you’re open to learning from it.

There’s a lot of negative news circulating as we navigate this AI boom: layoffs, declining clicks, concerns about the future of the web, and even broader conversations around how technology is affecting the way we think.

But we shouldn’t simply give in, passively consume, and accept everything we read as fact… right?

Speaking of being “open to learning”…. that’s a much bigger discussion for another day. There’s so much I want to say about what it takes to build a better SEO industry.

Hopefully. 🙏

That’s that for today folks and see you in the next newsletter!

Support the Riddler!

Sign up for my newsletter if you’re not already.
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

Where Did the Clicks Go?

Sara Taher — Tue, 14 Jul 2026 13:04:45 GMT

Hello Riddlers!

📌 Join my in my next webinar on Thursday July 16 · 1 PM UTC / 6:30 PM IST where I will be discussing how to navigate the zero click era. Sign up here.

Intro

Google is getting billions of searches every month, but we’re also learning that only 32% of all searches end with any type of click and of those, 66.6% go to the open web (non-google non PPC ads).

We’ve been blaming Google’s AI experiences, and rightfully so. Google, on the other hand, has been trying to soften the blow by saying websites are only losing low-quality clicks from users who would’ve bounced anyway. Just last year, Google’s Elizabeth Reid said AI Overviews drive “higher-quality clicks.”

This year, there’s a different story.

Perhaps more data is now available (like Google needs more data 😂) but Liz said in a recent interview that “the decline [of traffic to publishers] isn’t solely caused by AI Overviews, but rather by broader shifts in user behavior”

So I decided to analyze this statement.

Is it true? because if so, we maybe missing out on so much opportunity, and holding on to optimizing for the wrong type of content.

So buckle up!

As always, I'm going to build the case step by step before getting to my final conclusion.

Platformisation

A recent Reuters study has some very interesting findings.

When it comes to news consumption, the study found that “social media and video networks are, on average across the markets covered, more popular than both TV and owned news websites and apps as sources of news,” and that “people increasingly like to watch rather than read the news, often drawing on a wider range of sources and voices.”

And video content isn’t synonymous with YouTube.

The study shows that:

YouTube is used by 34% of people for news.
Instagram follows at 26% and continues to grow.
TikTok, while starting from a smaller base, is growing the fastest and is now used by 20% of people for news.
Facebook remains the largest platform overall for news consumption, with 43% of respondents using it, reversing recent declines.

But here’s another interesting question...

Who’s creating all this content?

Not necessarily large, trusted news organizations.

The study found that:

27% of respondents globally get some of their news from news-focused creators or influencers.
46% get some news from creators of any type.

Why?

Respondents said creators are more entertaining, easier to understand, and more relatable than traditional news outlets.

Ironically, those same respondents also believe creators are less trustworthy and less impartial.

So despite recognizing that creators are generally less trustworthy, people still choose to consume their content because it’s more engaging, easier to digest (likely because of simpler language), and more relatable.

The following chart shows how trust varies across different news sources.

So... is traditional media dead?

Actually, no.

The study found that people are consuming these newer formats alongside traditional news rather than completely replacing them.

Another interesting finding is that more than half (52%) of respondents aged 18–24 now say that social media, video platforms, and AI chatbots are their primary way of getting news.

To me, this tells us three things.

First, user preferences are shifting toward consuming information on social media and video platforms rather than traditional text-based content that ranks in search.

Second, the age of your audience matters. Millennials—and especially Gen Z—are increasingly choosing these platforms as their preferred way to discover and consume information.

Third, people want to hear from people, not necessarily large brands.

Interesting Expertise Content

For so long, our focus has been on creating “helpful content.”

And naturally, helpful content is not always built to be entertaining.

But it seems that to be successful with your content strategy in 2026, and appeal to wider audiences, you need to think about the entertainment factor as well.

Now, Liz Reid didn’t say that exactly. What she said was:

“To the extent that you produce really interesting expertise content, we still see that people are interested in that.”

May I second that?

I’ve been seeing a lot of misinformation in our industry being amplified by the same voices over and over again. And somehow, those voices become very popular.

What’s interesting is that, almost every time I see this content being shared, because it’s entertaining. It’s funny, provocative, or creates a strong reaction.

And that’s the point: attention and trust are not always correlated.

So if your content is not getting the attention or user engagement it deserves, it may be missing that element.

I’m not suggesting you become a clown 🤡. I’m simply pointing out an observation, and it’s up to you to decide whether you want to adapt your content strategy.

But keep in mind: the more intellectually sophisticated your content and concepts are, the narrower your audience will naturally become. And you can be ok with that by the way.

This is not a new finding. It has always been the case.

However, it may be more visible than ever today because users have an endless number of sources competing for their attention.

Content Length

Liz is also arguing that:

"To the extent that you might just read an AI review and you wouldn’t click, you might have previously just read the headline and not clicked through… You were going to go spend 10 minutes reading the article, you’re probably not going to decide that an AI Overview is a substitute for the 10-minute article."

She attributes this to the fact that sometimes people just want a quick answer—a sports score, for example—and they were never going to read the full article anyway.

So, according to this argument, AI Overviews are not necessarily taking clicks away from publishers. People will still click when they want something beyond basic information: an in-depth analysis, a unique perspective, an opinion, or expert commentary.

Ok Liz, but here’s the thing...

AI Overviews can also summarize those unique viewpoints and opinionated analyses. So that argument isn’t necessarily a strong defense for Google either 😂

However, this does point us toward a bigger question.

At some point, we have to face reality.

And reality says: people are consuming less long-form content.

As dramatic as this title from The Atlantic sounds “The End of Reading Is Here”,it doesn’t feel entirely disconnected from what we’re seeing.

Research at the University of California, Irvine, highlights how our attention patterns have changed over time. By tracking digital activity, researchers found that the average time spent on a single screen before switching tasks dropped from around 150 seconds in 2004 to approximately 47 seconds today.

The reality is that people’s attention is becoming more fragmented. With constant notifications, infinite scrolling, and an endless supply of content competing for attention, users are increasingly looking for faster and easier ways to consume information.

That’s why short-form answers like AI Overviews, as well as alternative discovery platforms like social media and video networks, are becoming increasingly attractive for quick and passive information consumption.

AIO and Few More Things

Yes AIO is taking clicks away, but also we need to acknowledge that people are preferring video content, shorter content (a lot of the times), content from creators and social media, content that makes them smile or laugh 🤷‍♀️

That’s the reality of where we are today.

And while I will remain a strong supporter of long-form content, I don’t think the answer is to ignore these patterns.

My recommendation is to branch out.

Take the valuable insights, research, and expertise from your long-form content and repurpose it into different formats and distribute it across different platforms.

And That’s a Wrap (Almost 😄)

Next time someone asks where the clicks go (I asked Google ppl that in search central live 😄) I guess now you know the answer…. Google took a lot of it, users want different things, and we have not adjusted to this new era yet.

That’s that for today folks and see you in the next newsletter!

References

Support the Riddler!

Sign up for my newsletter if you’re not already.
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

Why Content Chunking Was Never Really Up for Debate

Sara Taher — Tue, 07 Jul 2026 11:52:10 GMT

Hello Riddlers!

Alternatively you can become a paid subscriber

Intro

You’ve probably seen the term content chunking circulating across SEO social media, with some people recommending it as a tactic to improve AI visibility while others dismiss it as unnecessary.

And, of course, Google had to weigh in. On multiple occasions, they have made it clear that you don’t need to chunk your content for AI. They even added this in their official “Optimizing your website for generative AI features on Google Search” guide.

But… there are a couple of issues at play here.

First, a lot of SEOs are still relatively new to LLMs and AI—myself included—and I don’t think we always acknowledge that. If we did, we’d probably be a little more cautious about immediately rejecting new ideas.

Second, Google’s guidance is, to some extent, self-serving. Google doesn’t want site owners restructuring their content because of every new trend that popsup through the industry. From Google’s perspective, that’s a reasonable position to take. They said verbatim “We don’t want people to have to be crafting anything for Search specifically.” So their concern or objection is not the concept of chunking itself; rather, it’s trying to change your content specifically for search and visibility with disregard to users.

Chunking is also misunderstood in my opinion.

Of course it is, as I said we’re all new to this.

So when people try to interpret chunking into actionable approach, they may get it wrong, but then the nay-sayers would latch to those misinterpretations and label the entire concept as flawed.

So…

What is chunking, actually? And why was it never up for debate to begin with?

Buckle up!

What is chunking?

Chunking is the process of breaking a large text document into smaller, coherent sections called chunks. Each chunk represents a complete idea or segment of the text while preserving the original meaning and context.

Chunking is essential for large language models because they cannot process an unlimited amount of text all at once.

Makes sense right?

LLMs need to go through vast amounts of unstructured texts, and by dividing the content into smaller chunks in a size that LLMs can process (this size is called the context window) LLMs can process text more efficiently and therefore search for and retrieve information better.

Without chunking, LLMs are searching for a needle in a haystack.

If you ask an LLM, “What is an atom?”, it does not need to process every chemistry and physics resource ever created. Instead, a retrieval system can identify and provide the LLM with the specific sections of content that are most relevant to the question, such as passages explaining the definition and properties of atoms. This approach improves efficiency by reducing the amount of information the model needs to process, while also improving accuracy by allowing it to focus on the most relevant context.

This isn't just theoretical. Here's a simple example showing how splitting text into coherent paragraphs changes retrieval relevance.

Totally copying iPullRank here, I created this chunking simulator to show the relevance difference of keywords and text, before and after splitting the text into 2 paragraphs:

When all the text was one paragraph the relevance scores were 45.3% and 55.3%:

When the text was split into 2 paragraphs the relevance scores went up to 58.4% and 68.6% and that’s just by splitting the text into 2 proper paragraphs:

I’m still working on this tool and will probably launch it soon on my website for you to play around, stay tuned!

So….

We’ve established that chunking content is a fact of life used by RAG systems to feed relevant information for LLMs instead of giving LLMs the entire haystack…

But this raises an important question: Should you manually chunk your content, or is chunking already handled by the LLM’s retrieval system?

The answer is that you still should structure your content into clear, logical sections. While retrieval systems automatically chunk documents before passing them to an LLM, they can only work with the content they’re given.

So having well-structured content with clear headings, focused paragraphs, and distinct ideas, makes it easier for retrieval systems to identify accurate chunk boundaries and isolate the most relevant information.

And the result is, the chunks presented to the LLM are more likely to contain complete, self-contained answers, increasing the chances that your content is retrieved and used correctly.

But things like using good headers has always been part of what we do, right?

Absolutely, but that’s not the chunking process I’m proposing 😊

Introducing… the proper paragraph 🤷‍♀️

As I said in the title of this blog “Content Chunking Was Never Really Up for Debate” not only because that is actually what RAG systems already use when feeding information to LLMs, but because chunking happens to be how proper writing is!!!

Let me introduce you to….. 🥁🥁🥁

THE PROPER PARAGRAPH

According to the English language, there's such a thing as a 'proper paragraph if you’re writing formally, academically or professionally.

So unless your content is in conversational slang format like I’m writing right now - which we all know is not optimal for search - you should chunk your content using the proper paragraph method.

I can hear you in my mind saying “What do you mean Sara? what is that?”

A proper paragraph in English has the following structure:

--> Topic Sentence (The Opening): The first sentence that introduces the main point or controlling idea. It tells the reader exactly what to expect in this paragraph.

--> Supporting Sentences (The Middle): The body of the paragraph. These sentences provide explanations, facts, data, statistics, or concrete examples to back up your topic sentence

--> Transitions (The Connectors): Words or phrases (e.g., however, furthermore, for example, in contrast) used to link your sentences together smoothly.

--> Concluding Sentence (The Closing): The final sentence that summarizes the main idea, analyzes the evidence, or provides a logical transition into the next paragraph

writing your content in this format, makes each paragraph an individual entity that covers a specific point.

Here’s an example of this concept in action:

This has always been the proper way to write a paragraph. It’s not a new concept I’m introducing. And it’s very beneficial for users because just by reading the first sentence of a paragraph they can tell what the rest of the paragraph is about, making the content skimmable no matter how long it is!

Are we already using this format? I can argue that most of the content online is not written like that, nor do the average copywriters structure their content this way.

I tested the proper paragraph format using my simulator, and it did increase the relevance score.

Why? because it helps you group relevant information in one paragraph and have logically coherent content. So when RAG chunking comes in, it already has a solid foundation to work on.

Now I must be transparent, that I’m still testing more and more of this to provide solid numbers, but the concept still holds, this is how you should write your content. It is good writing which would naturally mean good chunking.

And That’s a Wrap (Almost 😄)

My argument is pretty straightforward, chunking is an established process that’s used by RAG systems to find and feed LLMs the most relevant information in a way an LLM can process.

But the more I read about chunking and what it’s trying to achieve, the more I found it similar to just proper English writing structure that has been established for a very long time.

I ran a few tests with my simulator tool, and early results showed improved relevance scores when using the proper paragraph format. So it’s a approach worth exploring.

Chunking is not just about the headers as some think and I hope by discussing the concept from both an LLM standpoint, and the English/user standpoint I showed you that, there’s really nothing to talk about… chunking your content is a fact of life 😄

That’s that for today folks and see you next newsletter!

Sources Worth Checking

Support the Riddler!

Sign up for my newsletter if you’re not already.
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

Agencies Need to Rethink the SEO Project Manager

Sara Taher — Tue, 30 Jun 2026 12:39:46 GMT

Hello Riddlers!

Alternatively you can become a paid subscriber

Agency Life

“AI won’t replace project managers, because there’s nothing to replace” 😂

I saw this joke on twitter the other day and it inspired this post…!

Throughout my career, I’ve worked at several agencies and experienced different models of working with SEO project managers. They also go by other titles—SEO account managers, SEO coordinators, client success managers, and more.

In one setup, the project manager was the primary client contact. They managed timelines, assigned tasks, followed up on deliverables, removed blockers, and handled most client communication. The SEO specialists focused on strategy and execution.

Many of these project managers had little or no hands-on SEO experience. They were excellent project managers, but not necessarily SEO practitioners.

A large portion of the role involved coordination: chasing updates, writing meeting notes, updating project boards, preparing status reports, scheduling meetings, documenting action items, and making sure nothing slipped through the cracks.

In other words… SEO clerking.

But that was years ago… today a lot of things have changed, my perspective for one as well as AI tools becoming integral part of our work.

So buckle up, as in this blog I intend to share my 2 cents on the role of SPM in 2026!

SEO depth

I really think we need to reconsider how we select people for the SPM role. After being in this industry for over 12 years, I believe an SEO project manager should have some hands-on SEO experience.

Some of the they should have done before:

Build or contribute to building a strategy
performed technical audits, even simple ones
collaborated with developers to get things done
created content recommendations for SEO
worked directly with content writers to update/create new content
worked with external third party agencies on top of an internal SEO team

….. and more just to name a few!

I’m not undermining people with pure project management backgrounds. Traditional PM skills are still important.

But SEO is one of those fields where you can’t fully “coordinate” your way through complexity.

Without hands-on exposure, there’s a real risk of slowing things down, misinterpreting priorities, or becoming an unnecessary layer between execution and decision-making.

If you’re currently in an SEO project manager role without SEO experience, the fix is simple: start getting closer to the work.

Play around with the tools, learn, try things even minor things here and there….

You’re not trying to convert to a full stack SEO, you are trying to build enough fluency to not be a blocker, and ideally to be able to step in when needed.

AI leverage

Not all agencies even have a dedicated SPM role. In many cases, SEO managers inherit the coordination and “clerical” side of delivery. That works too.

So whether you have an SPM or not, the person handling delivery just got a massive unlock.

Because AI is already handling a big chunk of the administrative workload:

Meeting summaries
Turning conversations into structured tasks
Drafting client updates and follow-ups
Organizing project documentation
Rewriting or refining status reports
Converting notes into actionable briefs

And more….

A lot of this admin work was never the “value” part of the job to begin with, it was just necessary overhead.

If you’re an SPM or doing SPM as part of your role, here are two additional areas I recommend you explore:

Optimize the project plan. Once you have a strategy in place, use AI tools to stress test it. AI can help you take it even to the next level, make it better, suggest changes here and there in priorities to drive more value to your clients.
Agility management. Changes happen as you go in any project. SEO is far from being a static field. Instead of maintaining static documents that quickly become outdated, use AI to continuously reframe plans based on new inputs.

Not to mention how I think Claude code should be your best friend.

Familiarize yourself with the idea of “Claude skills” or reusable AI workflows. For example, I recently built a simple SEO Claude skill that turns a single issue name into a structured SEO user story.

For example, here’s the output using this skill for the issue “organization schema missing on the homepage”

Small input → structured output → faster execution. Right?

And That’s a Wrap (Almost 😄)

Basically what I’m trying to say is that, in 2026 the role/skills for SEO project management have changed.

and this blog was my attempt to change the joke “AI won’t replace project managers, because there’s nothing to replace” to a no-joke.

If you pickup some hands-on SEO experience, and upskill yourself with necessary AI tools for project management, AI cannot replace you because you are down to the bones… you’re down to the real value that AI cannot replace!

That’s that for today folks and see you in the next newsletter!

Sources Worth Checking

SEO User Story Claude Skill

Support the Riddler!

Sign up for my newsletter if you’re not already.
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

Adapting to the Zero Click Era

Sara Taher — Tue, 23 Jun 2026 12:23:35 GMT

Hello Riddlers!

Alternatively you can become a paid subscriber

SEO has changed/is changing…. ya ya ya

You’ve probably heard it a thousand times by now: “SEO is changing”, “SEO is evolving”, and every other variation of the same message. And we’ve all been bombarded with GEO advice on how to optimize for the new world of LLMs.

Here’s the thing: LLMs still aren’t sending much traffic, so for now, we’re still focused on figuring out how to get the most out of Google.

But Google is now eating up most of the clicks… we’re in the zero click ERA 🤷‍♀️

“In the first four months of 2026, a whopping 68.01% of Google searches ended without a click,” according to a new study by Similarweb and SparkToro.

So where do the clicks go, what happens after the user searches?

There are 3 possible scenarios:

32% of the time people click to an organic result, to another Google property (Youtube, AI Mode, Maps, Images, etc…)
39% they do nothing after the search.
29% they do another search.

The study found that for every 1000 us Google searches, only 232 convert to clicks to a non-Google property.

That’s not good…

And so far every solution I see proposed to this dilemma misses the biggest opportunity out there…

We forgot that Google has different surfaces

I was listening to a webinar recently about “Google Discover” and it was during this webinar that I got my Aha moment.

Google Discover is a Google surface that shows users content based on their interests. Google also said they’re rolling the feature out on desktop last year—not sure where that went, but they said it’s coming.

Google discover is a huge traffic source, for a lot of website owners, it makes the biggest source of traffic, even more than traditional search. and that’s why we should consider optimizing for discover.

But here’s the thing…

the content that works in Google discover is slightly different than the content that works in traditional SERPs, although there’s an overlap of course.

So what type of content works well in Google Discover?

Content for Google Discover

Your best ideas are going to come from browsing your discover feed and seeing what made it there, however here are some pointers:

Sensational titles. But you need to avoid click bait, lean more on people’s natural curiosity vs been purely click-baity. An example is “New Zealand reaction to Mo Salah’s goal”.
Quoting people in titles. This is something that would not work in traditional SERPs, but for Google discover, having a title like “Mbappe on PSG’s season: “for me, a team that wins will always be an example of football”.
Questions in titles. Some have reported that this type of titles perform really well in discover. Worth testing out. An example is “why men keep dropping out of the workforce: it starts in childhood”
Timely fresh content. Yes you can still get visibility in discover with your evergreen content, but timely content has an edge. One thing you can do is to keep an eye on Google trends for keywords related to your niche, and once there’s something relevant to your audience, you can blog about it.
Images are very important for Google discover. Having the right featured image can improve your CTR there, this is something that is not relevant for traditional search results since you rarely see the featured image in search.
Localized content. Google discover surfaces content that is very relevant to the user and one of the factors that determine relevancy is location. If your customers are in a specific geographical area e.g Ontario, content like “Spring in Ontario. Uncover colors, rhythms and flavors that make this province a true cultural canvas” may work well.

Cheesy: understand your audience

Yes I know saying “understand your audience” is as lame as the Google advice “create helpful content” but it’s a step we rarely do in SEO.

Sometimes when I meet with business owners, I’m pleasantly surprised when they share things like:

“Our target audience are SEO agencies managing multiple clients who need to standardise workflows, speed up repetitive tasks, and maintain consistency without increasing headcount.”

Because then you can tailor interest-based content that would perform well in discover and on socials for your audience.

Topics you may write about for example:

Sensational: What actually breaks first when an SEO agency starts scaling
Quotes: “SEO manager at a 30-client agency: ‘Reporting is where we lose most of our time”
“Why do SEO agencies still do so much work manually in 2026?”
“Inside Canada’s SEO agencies: why reporting is still the biggest time drain”

You are not a ship anchored to the past

Our main focus has always been the 10 blue links, only today there’s rarely 10 blue links anymore… they’re usually less, with a SERPs designed to send less clicks to your website with Google’s AI experiences.

I’m not suggesting you abandon the 10 blue links, I just think you should widen your horizons a bit, and explore other surfaces available to you.

Creating content for Google Discover, can be the answer to your traffic challenges. And you probably have everything already in place:

Google technical foundations for discover highly are practically almost the same as the requirements for traditional serps.
User experience and bounce rate matters a lot for how your content would perform in discover, but these are things we should’ve always taken into account when creating content for SEO, right?
You probably have a social team in place, they can work with SEO to brainstorm content ideas that can work better for both social and discover!

The name of the game for SEO in 2026 is testing every thing, and this is one thing I think you should experiment with.

And That’s a Wrap (Almost 😄)

It’s time we tackle the zero-click reality with a different mindset and play the hand we’ve been dealt. The landscape has changed but there are still real opportunities for those who adapt to how distribution actually works today.

At the same time, we should keep an honest conversation going about the relationship between platforms and publishers.

Google continues to say “create helpful content,” and publishers have done exactly that, investing significant time, money, and effort. Yet the return in the form of clicks and owned audience is becoming harder to measure and less predictable.

As the web shifts toward more zero-click experiences, the question of fairness, attribution, and sustainability for publishers becomes harder to ignore.

We should continue pushing for a web where value flows both ways, not just one where content is consumed but not always returned to its source.

That’s that for today folks and see you in the next newsletter!

Sources Worth Checking

Google Discover Webinar with Barry Adams

Support the Riddler!

Sign up for my newsletter if you’re not already.
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

The Cause and Effect: How to Diagnose SEO Problems

Sara Taher — Tue, 16 Jun 2026 12:10:38 GMT

Hello Riddlers!

📌 📌 Set me as a preferred source on Google! Click here.

Warm Up

Around this same time last year I talked about diagnosing SEO traffic drops, I had a simple framework:

Since then, my SEO learning has continued — and today I want to share a technique that can diagnose any SEO issue.

It’s called the Ishikawa diagram (aka the Fishbone diagram): a visual root-cause analysis tool that connects cause and effect to uncover why an SEO problem is happening, and what you can actually test to fix it.

What I love about this technique is that it structures your thinking into a diagram you can share with your team and stakeholders.

So buckle up!

1# Start with the effect

Just draw a simple box (or whatever shape really) and write the problem or the symptom you’re seeing that you wish to address.

2# Buckets of causes

The next step would be to draw a diagonal line, and list all category of potential causes. We’re not listing exact causes yet, just categories of potential causes.

In SEO we have a handful of established categories, so our work is pretty clear:

Technical
Content
Authority
User experience

You may want to add more categories like market changes or Google updates and SERPs changes… totally up to you how you tackle this.

It’s fascinating how we can all work from similar-ish checklists yet arrive at completely different SEO results and performance in search.

I really think this is very helpful in breaking down a complex problem, and help to focus on it from various perspectives.

Reminds me of the 6 thinking hats technique, where you wear a different hat for a different mode of thinking all in an attempt to solve a problem.

This is pretty much similar, only here, instead of hats, you’re switching between a technical lens, a content lens, and so on.

3# Potential Causes

Now it’s time to fill in potential causes under each of the above buckets (and yes I changed the diagram color 😄)

A quick disclaimer: this is not meant to be an SEO checklist. It's simply an example of how one person might approach the problem. Your own experience, knowledge of the website, and understanding of the industry may lead you to a completely different diagram.

Here’s how I thought about this from each lens:

Technical lens: from my experience, sometimes things happen on the website itself that can cause a decline in traffic. Did your tech team remove the sitemap and not tell you, is there a UX change that included a lot of JS, is server returning 5XX errors or your internal links for one reason or the other have changed?
Content lens: if your content is low quality (AI and spammy for example) it will stop working at some point, that if it worked at all. So that maybe the issue, or is your content old and have not been updated in a long while? your competitors are working btw, they don’t stop, and so shouldn’t you, SEO is not a one and done thing so it maybe your content is up for a refresh. Finally, is your content a “commodity” type of content that is just lame, expected, and has no unique angle?
Authority lens: for so long in SEO pure brand mentions meant nothing, but that has changed, did your strategy for building authority adapt? Also let’s say you did a domain migration, and all your backlinks are pointing to your old domain (ya ya I know you created redirects and all), but wouldn’t it be nice to get a fresh pack of backlinks pointing to your new domain?
User Experience: a lot of SEOs would not bucket this alone, but I would 😄 mainly because it requires extensive type of work and analysis on it’s own. How’s the bounce rate on your page? while this specific metric is NOT a ranking factor, it can also indicate that there’s a problem with your content. How is your CTR? well I know that CTR is dead in search according to some ppl 😄 but it’s always worth a look to understand if there’s anything you can do to improve this (test different title formats perhaps?)

So basically keep asking why, and filling in the diagram until you’ve reached a useful level of detail. That is when the cause is specific enough to be able to test a change and measure it’s effect.

and you’re done! You now have a clear view of potential causes, that you can prioritize and start testing out.

And That’s a Wrap (Almost 😄)

It’s not uncommon for the most powerful ideas to be surprisingly simple.

There’s a quote often attributed to Albert Einstein: “If you can’t explain it simply, you don’t understand it well enough.” Whether or not he actually said it, the principle holds true.

What I appreciate most about the Ishikawa diagram is its simplicity. It helps organize messy problems into something visual, collaborative, and actionable.

And if you struggle to populate the diagram at all, that can be useful information too, it may simply highlight areas where you need to deepen your SEO knowledge and experience.

That’s that for today folks and see you in the next newsletter!

Sources Worth Checking

Support the Riddler!

Sign up for my newsletter if you’re not already.
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

How to Optimize Your Website for AI Agents

Sara Taher — Tue, 09 Jun 2026 11:54:17 GMT

Hello Riddlers!

📌 📌 Set me as a preferred source on Google! Click here.

Warm Up

So we’ve heard…

“Google search is AI search”, search is changing, and then cloudflare comes in and drops this data “Bot web traffic has overtaken human web traffic, data shows, for the first time” [1]

Google also added “agentic browsing audit” documentation to its lighthouse docs [2] that include reference to things like LLMs.txt after - also Google - repetitively telling us that LLMs.txt is not useful/important on various occasions 😂😂😂

It seems like we are trying to establish certainty in an environment where things are very dynamic and even the people building these systems don’t have all the answers yet.

Few topics capture this confusion better than LLMs.txt!

Over the course of several months, we heard:

Jun 18, 2025: “FWIW no AI system currently uses llms.txt”
Jul 22, 2025: noindex your LLMs.txt file
Jul 24, 2025: Google doesn’t support LLMs.txt and isn’t planning to
Aug 1, 2025: Why should AI systems trust a file that website owners can fill with whatever they want?
Dec 3, 2025: Google itself uploaded an LLMs.txt file for the Google Search Central portal (which is now gone btw 404-ing https://developers.google.com/search/docs/llms.txt but it still remains on many of Google’s other sites)
Jan 6, 2026: Google explains, they added LLMs.txt to their websites for other purposes
Jan 20, 2026: Google Search Team Does Not Endorse LLMs.txt Files

And on the other hand, we have reports that "Log file analysis shows that OpenAI crawls LLMs.txt file”

At this point, I think the entire industry deserves a group therapy session 😂

But all jokes aside, there are a few things we need to keep in mind:

When Google representatives speak publicly, they’re often speaking about specific products and teams. What applies to Google Search may not apply to other Google initiatives or to other companies entirely.
It was widely believed that Google doesn’t use metrics like CTR or bounce rate for rankings. But it turns out they do use other user behavior signals that are in the same general category🤷‍♀️. It just shows that there usually isn’t a simple yes or no answer, there are often a lot of details, and those details really matter.
We should stay open-minded about everything. A lot of things are still new to everyone. It helps to pause, think things through, test ideas, and then form your own conclusions. Also this isn’t something you do once and forget, it’s an ongoing process as more information keeps unfolding and technology continues to evolve..

In this new light💡here’s everything I know about getting your website ready for AI agents!

What agents do?

AI agents are software that performs specific tasks on your behalf autonomously. Unlike chatbots and LLMs, AI agents don’t just provide “answers”; they perform actual tasks, they can plan a trip for you, and make bookings for example.

“At its core, an agent is a system that receives input, interprets it, then plans and executes actions on behalf of the user” [3]

To do this, they need to browse the web, and be able to use your website.

In the AI-powered search era, we have three barriers we need to cross:

Discoverability: we need to be findable by different AI systems.
Citation and/or selection: being cited and selected as a source. And while AI agents may not provide outputs like AI mode and AI overviews for example, there has to be some type of mechanism where they narrow down the choices to start making decisions and taking actions.
Actions: can AI agents use your website? and that’s what we will focus on today!

Is your website screen reader friendly?

AI agents and screen readers use the accessibility tree as their primary roadmap to navigate websites and applications. This means that you can use chrome plugins like Text Mode to see how both screen readers and AI agents would see your website. Here’s a screenshot from my LinkedIn feed using this plugin:

What to tell your developers?

Use real HTML elements like
Use
Every input must have a that clearly explains what the field is for. For example, use “Email address” instead of relying only on placeholder text.
Add appropriate autocomplete values to forms, such as name, email, tel, street-address, and organization. This helps browsers and AI agents complete forms correctly.
Use headings in order (
→
→
) and avoid skipping levels (for example, jumping from
to
). This helps agents understand the structure of the page and its sections.
When building complex, custom, or dynamic user interfaces where native HTML elements do not provide the required functionality, use ARIA (Accessible Rich Internet Applications) attributes to fill those gaps.
Make sure important content is already present in the HTML when the page loads rather than being added later through JavaScript. Use Server-Side Rendering (SSR) so pages are fully built before they reach the browser. Avoid depending on JavaScript to display critical content, because AI agents may not run or fully process scripts. Anything important should not rely on user actions or client-side scripts to become available.
Use clear, descriptive link text that explains where a link goes. Avoid vague phrases such as “click here” or “read more.” Both users and AI agents rely on link text to understand the purpose of each link.

Google Lighthouse Agentic Browsing audit

If you download chrome canary (Chrome Canary is Google’s most experimental, cutting-edge version of the Chrome browser, designed primarily for developers, testers, and browser enthusiasts.), open your webpage, right click inspect, go to the lighthouse report, you’ll see that Google now has an Agentic browsing audit… ta-da!

You can use this audit to help you assess your webpage for agentic browsing (clear from the name daah 😂)

If you don’t want to do all of this, and want just to know what to tell your developer, here you go:

What to tell your developers?

CLS (Cumulative Layout Shift) just got another importance boost outside of the core web vitals score. Your CLS score is a measure of UI stability for interaction precision.
LLMs.txt, (please don’t laugh, mute giggles are ok though) but Google’s own documentation recommends “Create an llms.txt file and place it in the root directory of your website (for example, https://example.com/llms.txt). The file should follow the llms.txt specification and provide a concise Markdown summary of your site’s purpose and key links.” [4]
Use WebMCP [5]. WebMCP is a proposed protocol that lets websites explicitly define and surface their “tools” (like forms, actions, and workflows) to AI agents in a structured way, instead of relying on agents to infer behavior from HTML, JavaScript, or UI interactions. In practice, it lets a site declare “this is a search tool,” “this is a checkout action,” or “this is a booking flow,” along with the required inputs and expected outputs, so agents can interact with it reliably and consistently.

Markdowns

Another highly controversial topic is adding markdowns to your webpages. Markdown is a lightweight markup language used to format plain text. It is widely used online for platforms like Reddit for example to format text in posts, comments, and community sidebars.

Google has added markdown files, .md.txt files, to the Google Search help documents [6]. Here’s an example.

While John Muller explicitly said that “There were discussions around this on Linkedin a few weeks back. This is not being done for Search or generative AI responses in Search.” This does not mean that Markdowns are useless, they maybe used by AI agents.

One tip though, do not create a markdown version of each webpage, because this may cause duplication. At the moment I’m testing adding markdown in the html of the page. You can use markdown between html tags as long as you add to the tags the markdown attribute with value 1. markdown=”1”

Here’s how this looks like on my website right now:

And That’s a Wrap (Almost 😄)

While a lot of the above recommendations are classics and have existed long before AI agents, some are very new and come with a lot of skepticism.

Keep an open mind, test things out, and keep learning.

That’s that for today folks and see you next newsletter!

Support the Riddler!

Sign up for my newsletter if you’re not already.
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

Sources:

[1] Bot web traffic has overtaken human web traffic, data shows

[2] Lighthouse agentic browsing scoring

[3] Introduction to agents

[4] llms.txt

[5] WebMCP

[6] Google Adds Markdown Files To Help Docs But Not Used For Search

Google I/O: What's Exactly Happening in Search

Sara Taher — Tue, 02 Jun 2026 13:14:19 GMT

Hello Riddlers!

📌 📌 Your support keeps this newsletter FREE —> [☕Buy Me a Coffee☕]💡 Want more from me? [SEO Strategy Course] | [Build Your Own SEO Tools (with Python & Agentic AI)] | [Premium SEO Mastermind]

Warm Up

Since the Google I/O event a few weeks ago, my feed has been full of hot takes declaring “SEO is dead.”

Here’s one from TechCrunch:

It opens with:

“The era of the “ten blue links” is officially over.”

But if you read past that line (and most people don’t), the same article clarifies later:

“The links, to clarify, have not entirely disappeared; they are just no longer the priority for many types of searches.”

There’s plenty of speculation about what Google’s I/O announcements mean for search and with even more AI being woven in, the obvious question is: are we headed for another traffic drop?

AI search has become the synonymous to “less traffic to publishers”, so when Google’s VP of search, Liz Reid declares that ‘Google search is AI search, through and through,’ it’s not exactly great news 😆

So what are we going to do?

As of now, everything is still in the air, a lot of the AI features Google announced will be pushed to search later this summer, and as I will discuss below, they may not cause the apocalypse everyone is expecting. This much I know. Of course I can be wrong, but hear me out.

I also know that the web echosystem relies on publishers creating good content, if publishers cannot properly measure, let alone see the ROI of their work, the system will collapse, so whatever is happening in search, Google will have no option but to find ways to support publishers and compensate them.

Enter features like Preferred Sources, which just rolled out to all publishers after being limited to news outlets only. It's the shortcut no one expected! A way to get more visibility in search and across Google's AI surfaces.

If you're not familiar with it, it's a feature that lets people "follow" specific websites, so those sites appear more in their search results. [learn more here]

So what’s happening to search?

The Search Box

Google just launched their biggest upgrade to their search box ever. It anticipates your intent and helps you formulate your questions better that go beyond autocomplete.

What this means for SEO?

We will be seeing more and more long tail queries in GSC. Our traditional keyword research processes and optimizations will need to adapt accordingly.

It’s not going to be just simple keywords like “best SEO newsletters”, there will be more details attached like “best SEO newsletters for someone starting in SEO with no technical background” for example.

With more specific searches like this, your content will need to be more specific as well. But in a way, this has always been the case, over 70% of traffic on the web already comes from long tail keywords [source]. The queries are just going to get even more specific.

Additionally being more detailed and specific in your content has always been a thing, after all 94.74% of keywords have monthly search volumes of 10 or less [source]

Search Agents 🧐

Did you ever use Google alerts?

Well one of the features Google announced was search agents that keeps you up-to-date on whatever you need.

For example, if you’re apartment hunting, the new search agents feature can keep scanning the web for you and notifies you once there’s a new listing that fits your specific needs.

Ya… google alerts with more fine tuning features… one you might actually use 😆

What this means for SEO?

Agents will be searching on behalf of humans, LLMs do that at the moment in a way. Is your website agents ready?

Turns out adding an LLM.txt, markdown and schema may come in handy now right?

If your website has tools that agents maybe using, things like WebMCP protocol that Google quietly announced that didn't get nearly the attention it deserved maybe core to your web strategy. It now all makes perfect sense. It’s another piece of the agentic web puzzle.

The question is are we ready AI agents using your website on behalf of users?

Agentic coding in Search

You know when you want to teach a child something and you start by drawing it for them?

Search is about to do that.

Google announced that search results will be able to generate interactive, visual experiences on the fly. Not static diagrams, but actual tools you can play with.

For example,

ask how mixing red and yellow works, and instead of a text explanation of how they create the color orange, you get sliders to adjust the composition yourself and watch the color change in real time.

What this means for SEO?

I really don’t know. But I can make some guesses:

People will have higher expectations from educational content. A lame block of text may not cut it anymore. If you're teaching something, you'll need to go deeper, or offer something the demo can't.
A lot of learning will happen in search. I don’t think this is great news given we know the limitations of LLMs and how they hallucinate, a side of effect of their design.
Another drop in traffic to educational websites. Sorry, but not great news.

The Mini Apps & New Content Mindset

One thing the I/O blog mentioned that I find super interesting, is “But often, you aren’t asking one-off questions — you have an ongoing task you find yourself searching for over and over”

With the agentic capabilities coming to search, you will be able to create dashboards to track your progress on specific tasks. Like building a fitness tracker for your daily workout plans and meals.

“You can think of these like mini apps for your own specific tasks.”

But this feature will be only available at first to Google AI Pro and Ultra subscribers in the U.S.

What this means for SEO?

I know this may come off surprising, but this is the part I liked the most about the I/O announcements. It just gave us a major hint how we should be creating content.

Think of the task the user is trying to achieve that this one search is one part of, look for the bigger picture not just the exact searches and their intent.

Old thinking:

You sell evening dresses, you optimize your content as follows:

evening dresses product category page
individual evening dresses page
blogs about evening dresses (e.g what dresses are most slimming, and how much should you spend on an evening dress, etc..)

New thinking:

You sell evening dresses, you optimize your content as follows:

You ask “why is a customer looking to buy an evening dress”
- They have a date
- They are attending a wedding
- They just got invited to a black tie event and have no idea what to wear
From there, you build content around those bigger goals, the date, the wedding, the event, not just the dress. Some of the information will naturally overlap with what you’re already creating, but the structure shifts from “what are people searching for” to “what are people trying to accomplish.” Think of it as zooming out from the search to the task behind it.

Psst… his overlaps nicely with the commodity vs. non-commodity content framework from the Google search central live event. The content that survives is the content that serves a purpose no widget or AI snippet can replicate.

And That’s a Wrap (Almost 😄)

I’ll be honest, I’m excited about what’s coming, even if it’s a challenge. SEO has survived every major evolution so far, and I don’t think this one will be any different.

And did you notice how everything connects? Preferred Sources, search agents, WebMCP, agentic coding, mini apps, etc.. it’s all pointing in the same direction. The web is getting more agentic, and the SEOs who adapt their thinking early will be just fine.

That’s that for today folks and see you next newsletter 👋

Support the Riddler!

Sign up for my newsletter if you’re not already.
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

How to Use Reddit for SEO [Without Spamming]

Sara Taher — Tue, 26 May 2026 12:55:14 GMT

Hello Riddlers!

📌 Your support keeps this newsletter FREE —> [☕Buy Me a Coffee☕]💡 Want more from me? [SEO Strategy Course] | [Build Your Own SEO Tools (with Python & Agentic AI)] | [Premium SEO Mastermind]

Reddit is Organic

I was planning on sharing today how I got my Python course cited in ChatGPT without listicles or spamming, but I found a more interesting idea to write about!

Yesterday, I was running my monthly SEO town-hall-style meetup and Reddit came up, and I found myself discussing a cool idea, that I decided to share today instead!

Well, we all know how important Reddit is for visibility in LLMs. Reddit is a leading source across all AI search tools. It’s the top-cited domain on Perplexity, and among the top three on both SearchGPT and Google AI Mode [source].

And in case you didn’t know that, OpenAI officially uses Reddit as part of its training data [source].

Then, of course, we know that Google loves citing Reddit way more than we SEOs would like to see. It constantly raises eyebrows, really. Did you know that a Reddit thread can rank on Google within minutes 🤷? [source]

It’s the stuff fairy tales are made of for SEO people…

But the problem with Reddit is that it’s organic. If you try to hijack a thread to insert your brand, you will be easily spotted by Redditors and moderators. It’s not going to work. It can actually backfire for your brand.

The best engagements on Reddit about your brand are the ones that come naturally and organically, without you paying someone to promote your product or service around there. The strongest Reddit mentions are the ones you don’t manufacture.

That’s probably why Google and LLMs love Reddit…

But what can we as SEOs do then? How can we leverage that platform?

Let me tell you how!

Your New Customer Service Q&A Database

I wrote recently how Starbucks is now hiring more baristas as relying on automations backfired [source]. People want to engage with other people, it’s also more efficient sometimes (contrary to the popular belief that automation is always better)

There are moments when people would rather speak and engage with other people.

What if you use reddit as your customer Q&A forum? what if you direct users to your own brand subreddit where they can post questions about your brand and products, and get human-written informative answers.

What if your social media team uses reddit to engage with your customers on your brand subreddit.

I did this analysis over a year ago [source] and found many people are searching for brand subreddits.

Most of the time, these subreddits are not official. What if there’s an official one, where people can still post freely and ask questions, the only difference is that you are directing your customers to post their questions there.

You know how many brands have an FAQ page, that’s a great practice, but now your subreddit can supplement this page, and be a resource for every possible question your customers have.

If you’ve ever been on Facebook groups recently, you’ll see how many people are still engaging and asking questions there and some of those posts are also getting visibility in search, because that’s what people want.

And That’s a Wrap (Almost 😄)

SEO tactics are no longer limited to changes you make on the website in my opinion.

I personally audit Youtube, I audit socials, and I try to understand what factors and platforms impact a brand’s visibility in search and LLMS, then build my strategy accordingly.

And as simple as this reddit tactic is, I really think it would be useful for brands that have multiple lines of products or complex products that users can come back with questions about.

That’s that for today folks and see you next newsletter.

Support the Riddler!

Sign up for my newsletter if you’re not already.
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

How the Search Machine Messes with SEO Tests

Sara Taher — Tue, 19 May 2026 13:15:27 GMT

Hello Riddlers!

The Mindset

As a LinkedIn’er myself 😂, I see tons of SEO tests and experiments shared. Some are really good and some are… just broken, and I’m sure you’ve seen your fair share too.

The problem I realized exists is, we need a better mindset to create SEO experiments and to read and assess their results.

Unless we want to build our strategies on broken assumptions, or even worse, do the right SEO tactics, but assume they don’t work because… we’re approaching the tests with a mindset that’s lacking!

Let me explain what I mean… buckle up, this is super FUN!

Google Search Console is a broken tool

There… I said it… again…

The first time I said GSC was buggy, it raised a lot of eyebrows… to my surprise.

Well, a few quick examples to establish an agreement on that, include the URL Inspection Tool showing a page as “not indexed” while that same page is literally appearing in the Performance Report collecting impressions… or impressions being inflated for almost a year 😂😂😂

There’s this unwritten rule, that if your SEO strategy is working, the first early signs would be an increase in impressions. I still believe that’s true. But as of today… I don’t think this is enough.

You maybe thinking rankings is a better measurement…. well yes and no (I’ll explain in the next section), but my conclusion is:

if you rely solely on GSC to see if your test/experiment is working, you may not be able to spot that accurately. There’s data lag, there’s seasonality, there’s inflated metrics, and more…

The best approach is imo:

Combine data from multiple sources (e.g: GA4)
Check user behavior on the page because that’s all that matters (use MS clarity for example)
Have a baseline of your website performance, and understand potential impact of seasonality (decide on % lift or decline)

Google plays games

I mentioned above using rankings to measure the success of an SEO strategy, right?

Well, imagine this…

You try an SEO tactic, and the next day your ranking jumps from position 50 to position 3.

Then you repeat the same tactic 2–3 more times… and the same thing happens again.

Wouldn’t that reveal a LOT about how Google works?

And wouldn’t that open the door to even more search manipulation? (As if backlinks spam, listicles, and everything else weren’t enough already 😂)

Google has two patents (Ranking documents & Changing a rank of a document by applying a rank transition function) describing how it intentionally slow down ranking changes instead of updating rankings instantly 😂

Google is using social engineering to fight spam.

For example, if you build backlinks to your website, you may notice:

nothing happens at all
a decline in rankings
rankings fluctuations

for a period of time (a transition period), before you get rewarded, that is if Google sees you deserve to be rewarded.

This on it’s own is enough

to mess up with a lot of SEO tests.

You can be doing the right thing, but seeing no rewards at all. I experienced that in my own Python for SEO page. I did everything right imo, and it was not until recently that it finally made it to the top of the SERPs.

The core goal of Google is spam detection and dampening manipulative SEO, such as aggressive link schemes or constant micro‑tuning based on daily SERP feedback.

By introducing some type of a ranking transition period with seemingly random or counter‑intuitive ranking changes, Google makes it harder for spammers to map “I did X yesterday, rankings went up today, so X works.”

The patent discusses observing how site owners react to these fluctuations; if someone rapidly reverses or overreacts to every movement, that behavior can itself be a spam signal

For link building specifically, the patent says:

“it might take approximately 70 days for a change in a document’s link-based information to change the rank of the document to its steady state”

The problem when you have such a transition period, a lot of other changes have already occurred on your website, so it can become a challenge to isolate the exact tactic that brought the positive impact.

Also, anyone saying SEO does not need a minimum of three months to work, I hope they read this…. things can be fast, but also can be slow, by design.

Best approach for analysis imo

Well if data from GSC is… inflated sometimes, and rankings are tricking you, what can we do?

The best analysis imo would be existing SERPs and citations, and finding patterns. That is if you’re looking for a quick analysis. Otherwise, long term SEO studies expanding over at least 3 months (I’d say 6 months is even better since we saw how mount AI takes some time before it goes down) are your second best alternative.

And That’s a Wrap (Almost 😄)

My goals in this edition was to shed the light on why so many SEO tests are broken.

GSC inflated impressions and Google’s social engineering tricks are not the only reason though.

More often than not, the experiment itself has design flows. But that’s probably a topic for another day.

That’s that for today folks and see you next newsletter.

Support the Riddler!

Sign up for my newsletter if you’re not already.
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.

Ranking vs Grounding: They're Not the Same

Sara Taher — Tue, 12 May 2026 13:21:39 GMT

Hello Riddlers!

Truth Be Told

I’m about to once again step into the debate of GEO vs SEO 😄

Google has said on multiple occasions that they’re essentially the same thing. That there’s nothing particularly special or different you need to do for “GEO”.

But Google has also said a lot of things before…

…and ahem, we all know how some of those turned out 😄

So it was honestly refreshing to see Microsoft Bing publish a new blog discussing the difference between traditional search systems and grounding systems.

While much of the industry has been busy arguing whether GEO is “real” or not, Bing has quietly been one of the more transparent players during this AI-powered search wave.

They introduced AI Answers reporting inside Bing Webmaster Tools, publicly discussed the role of schema and structured data for LLM understanding, and now they’re sharing more details about how indexing systems and grounding systems actually work behind the scenes.

So buckle up, because I’m about to dissect what all of this actually means for SEO 🙏

Two Systems, Two Questions

I really think it’s important to start from the top and acknowledge that traditional search, and LLMs grounding are two separate systems.

As the new Microsoft Bing blog explains:

“Traditional search and grounding systems share the same foundation – crawling, understanding, and ranking the web – but they are optimized for fundamentally different outcomes. “

Here’s a simple analogy to explain the difference.

You walk into a restaurant and the waiter hands you the menu. That’s traditional search: a curated list of options they think may satisfy what you’re looking for.

But then you ask the waiter, “What’s the best dish here?”

Now the waiter shifts from simply presenting options to giving you a direct recommendation based on context, interpretation, and experience.

And that’s essentially the difference between traditional search and grounding.

Traditional search presents you with possible sources and options.

Grounding attempts to synthesize those sources into a direct answer.

It also fits well with the classic “librarian” analogy we’ve used for years in SEO.

A librarian can point you toward the books or sections where the information exists.

But if the librarian directly answers your question based on their own reading and interpretation of those books… that’s a very different role.

One system retrieves information sources and serves you the most relevant.

The other synthesizes and delivers answers.

Traditional search asks: which pages should a user visit? (the menu)

Grounding asks: what information can an AI system responsibly use to construct a response? (the dish)

How They Work Differently

The two systems clearly have two different goals/outcomes. And therefore it’s reasonable to assume they work differently, but the good news we don’t have to assume anymore!

Traditional search is about serving the user a list of options, they believe are the best to answer the user query. When you look at SERPs, you don’t expect every single URL to be great for you but, you expect the results to be “right enough” so you find what you need.

If you’re searching for a black shirt, you’ll get a lot of options in SERPs, not every option necessarily fits your taste, but overall the results are relevant to your search and you can eventually find what you need.

AI answers has an additional constraint, the goal is not to point the user to information sources, it’s goal is to use those information sources to answer the user query (well that explains why the clicks are less, the system by design is about providing the answer directly to the user, citations or not).

AI answers use the index. Let’s make that clear, however, it’s not looking for entire pages, it’s looking for “groundable information – discrete, supportable facts with clear provenance”

This means the grounding system needs to decide:

Can I answer this question/user query
Do I have enough information to do so while preserving the meaning, making sure users can verify that if needed, and detecting and representing conflicting views when relevant?

This is where the two systems clearly are optimizing for different things.

Zooming In

To be more clear, here are processes and things Grounding needs to account for, that traditional search does not:

AI systems often break content into smaller chunks (yes they chunk content) before retrieving it. The challenge is making sure the original meaning and context are still preserved after that process.
Source quality matters differently in AI answers. A grounding system cannot treat every indexed page as equally trustworthy if it’s going to generate a direct response from it.
Freshness becomes far more critical. In traditional search, outdated content may simply rank lower. In grounding systems, outdated information can directly become part of the answer itself.
Contradicting sources also become a bigger issue. A search engine can simply rank one page above another and let the user decide. But a grounding system has to recognize when sources disagree, otherwise the AI may confidently present the wrong information as fact.
Another important difference is that errors in grounding systems can snowball very quickly. In traditional search, if a result is irrelevant or inaccurate, users can usually recognize it, skip it, and adjust their search accordingly. But grounding systems don’t really have that safety net. If an AI retrieves slightly wrong, incomplete, or misleading information early in the process, that error can carry through the entire reasoning chain and influence the final answer without the user even realizing it. That means grounding systems need to optimize not just for retrieving information once, but for consistently retrieving accurate, reliable information across repeated reasoning steps and interactions.

So What?

At this point you’re looking for actionable steps, here’s some thoughts, learnings and future research ideas that I gathered at this stage:

Understanding and optimizing for chunking seems to be valuable based on the information in Bing’s blog. I highly recommend you read “Moving from a Google-shaped Web to an Agent-shaped Web: A Refutation of Misinformation about Chunking” to learn more (I will 😄).
We’ve seen over and over AI answers citing some URLs NOT in the SERPs. This requires an in-depth research to understand what did AI answers find in this source to cite, while this URL did not qualify for the top 10 results in SERPs (different systems, right?)
Freshness matters more than we think. That takes updating your content to a whole new level of importance (strategy update alert: if AI answers matter to you, prioritize updating content)

And That’s a Wrap (Almost 😄)

I think the confusion of the debate of SEO vs GEO all comes down to the fact that both search indexing and grounding indexing processes start at the same foundations.

That’s true, but that does not mean they’re identical systems. They have different goals and outcomes, so that cannot be.

As the blog perfectly puts it:

“Search indexing was built to help humans decide what to read.” while “Grounding indexing is being built to help AI systems decide what to say”

I wanna end today’s newsletter by saying that, every time there’s something new in SEO, there’s always those SEO naysayers ready to dismiss it immediately… that’s the actual root cause of the SEO vs GEO debate imo. Some people are just not open to new ideas and innovation🤷

You don’t have to follow this pattern, you can do your own thinking!

That’s that for today folks and see you next newsletter.

Bing’s post: Evolving role of the index: From ranking pages to supporting answers

Support the Riddler!

Sign up for my newsletter if you’re not already.
Share the newsletter and invite your friends to signup. Help me reach 2k signups by end of 2025 please 🙂
Provide feedback on how I can make this newsletter better!!!
Buy me coffee.
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.

Disclaimer: LLMs were used to assist in wording and phrasing this blog.