Kagi and some observations about website content

For some time now I've been using kagi as my main search engine, it's great and there will be a whole post about this in the future. For now we're here because I want to share an observation relating to a statement made by them.

First of all this is the statement in question made on their website:

Our unique algorithms down-rank pages with a lot of ads and trackers (which we have found correlate with a decrease in content quality)...

Let's break that down:

They down-rank pages where ads are a large part of the content.
Their theory is that this is a marker for the content quality of those pages.

While I intuitively tend to agree about the quality point (and hence find their results usually very high quality and pleasant) what I actually want to share is an observation on how far the balance between the actual content of a totally legitimate website and the ads can go.

As a small experimental side project I recently built a Laravel app that simply goes and has some Google Alerts and Exa searches which it monitors for me. The idea is to look at new pages coming up and map out how often certain problems in a given field are mentioned to get a feel for how prominent those are. To look at the pages I use Cloudflares browser rendering API and specifically the option to return markdown directly. The kind of websites that come up are typically industry news portals which have some mix of company provided press release style messages, actual editorial content from them and lots of advertising. So to be clear: we are talking about totally legitimate websites, not about something where content was fully generated via AI to direct people onto essentially an SEO content farm or anything like that.

How is the relation between content and ads then? Well: I ended up actually using an LLM to filter out all the ads in the markdown before doing any further operations on it, and typically I will have a text that is about 75% smaller that way. Anyone in the business of publishing news sites or, for that matter, websites just like those industry portals, may lament how AI kills their ad revenue but I tend to find this equally lamentable: a situation where I'm bombarded with 75% ad content and additional links, navigation etc in those places. Of course I will use all technical means at my disposal to circumvent that. For what it's worth: I have checked an ad-heavy website I have been involved with years ago and the outcome ends up being similar, so even without any specially bad intentions this is where we currently end up with when producing an ad financed website, even without any of the extreme things like big popups of ads or similar disruptive elements. Maybe we cannot do that much about the parts taken up by navigation and cross-linking on the websites, but for sure having ads be a smaller part is desirable.

To be clear: if you build a super quick and user-friendly paywall that allows me to pay for good editorial content I'm happy to do that - no need to bombard me with ads.