If you find value in my newsletter and want to support the work that goes into it consider —> [☕Buy Me a Coffee☕]💡 Want more from me? [SEO Strategy Course] | [Build Your Own SEO Tools (with Python & Agentic AI)] | [Premium SEO Mastermind]
Alternatively you can become a paid subscriber:
Intro
As usual, I spent sometime doing SEO with Claude and this time I decided to continue my work building my very own search engine!
I gotta tell you this is a very fun project, but it also makes you pause and think how do you want to do this or that, how something is traditionally done, and how you may have a different idea you want to try out instead.
All in all, building a search engine just makes your understanding of how search engines work. Let’s not forget that even LLMs use search engines to run live searches, so search engines are still very important.
This also brings us to the conversation of “SEO is dead”. It’s not. People will always want to search for things because naturally we don’t know everything, or where to find what we need. How people search, the tools we use and the search interaction, this is what is evolving… dead is not really the right word here.
Also why do we always have to kill something? I read a post the other day “AI is dead”… I was like “wow, already 😄😄😄”…
Well, without further ado, let’s dive into today’s progress!
Text Normalization
At this point we have crawled the URLs (see part 1), we have extracted the main content (see part 2) of the pages, so what’s next?
We have the text, but it is not ready to be used by the search engine for information retrieval and ranking purposes. Why is that? well because there’s just too many variations and noise in the text.
For example:
I have content that has Arabic letters
There’s a lot of emojis
Capitalized letters and lowercase letters
Punctuations
etc…
So text normalization removes unneeded variations and converts the text into a single, standardized format. This makes it easy to process later.
Examples of normalized text:
So in my code, I basically went through all 502 URLs in our crawled and removed all of the following:
Fancy quotes and dashes become plain ones, so don’t matches don’t.
Everything is lowercase, and accents are removed, so “cafe” and “café” are now the same word.
Non-English characters (Arabic, emoji, decorative symbols) are dropped.
Hidden characters and formatting marks (headings, bold, tables, leftover web code) are removed.
Code examples on technical pages are kept as text, since someone might search for them.
Results: about 794,000 words went in and 789,000 came out. The small drop is mostly formatting symbols.
Final output
This is the current structure of the project. We have stage 1 which is basically the crawler, then stage 2 the indexer stage where we first extracted the main content, then normalized it.
Normalization was the last step in the indexer stage and now we have a database table of normalized text ready for the next stage of doing actual searching, ranking and serving results to users!
Cool stuff, right?
Important Note: One thing worth mentioning is that normalization doesn’t mean throwing away the original content. The normalized version gives us a cleaner text to work with during retrieval and matching, but keeping the original extracted content is equally important as we’ll still need it later when displaying search results, generating snippets and debugging potential issues.
What’s coming next?
Here are what to expect in the coming stages of this project:
Tokenization: Breaking the normalized text into individual terms.
Inverted indexing: Building a mapping between terms and the documents containing them.
Retrieval: Finding documents relevant to a query.
Ranking: Ordering retrieved documents based on relevance.
I may add more steps, make minor adjustments here and there, but this is the overall gist of my plan for the next stages.
And That’s a Wrap (Almost 😄)
One more step towards building my search engine… I still have time to find a name for it… what should I call it? the SEO Riddler SE?
The purpose of this series is to build in public, encourage others to try the same, and inspire and motivate people that building a tool in 2026 doesn’t have to be complicated.
All you need to do is understand the concepts of what you’re trying to build and how processes work (e.g crawling, extraction, etc…) and then with some vibe coding practice, the sky is the limit imo!
That’s that for today folks and see you in the next newsletter!
Support the Riddler!
Sign up for my newsletter if you’re not already. (Pssst, you can also become a paid subscriber)
Share the newsletter and invite your friends to signup. Help me reach 2k signups on Substack by end of 2026 please 🙂
Provide feedback on how I can make this newsletter better!!!
If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.
Disclaimer: LLMs were used to assist in wording and phrasing this blog.