Dear Substack: Please Fix Your Search Engine

I thought it was discriminating against me, but actually it’s terrible at finding anything.

Looking for stuff using Substack’s search engine gets old fast

I posted my first Substack article on March 7, 2023. I was migrating away from Medium, on which I had posted many articles over the years, because Medium had started censoring my content without giving me specific reasons. Because I had several already-written articles to simply copy over, within one week I had thirteen articles posted on Substack.

I imagined that the best way for Substack readers to find me was via Substack’s own search engine, so I started searching for my own content using it. This was a curious experience. I’d enter search terms consisting of relatively rare words from my article titles, like “refutations” from my second title, and instead of finding my content, Substack’s engine seemed to desperately seek other titles with words that were lexically similar but not identical. For example, my “refutations” search would turn up titles containing words like reputation(s) and regulation(s), in a seemingly resolute attempt to avoid matching my search term precisely. Those few results containing the actual word (again, “refutations”) were articles from years ago, often where the word didn’t appear in the title or was perhaps incidental to the article’s topic.

I went back and forth with Substack’s support people, who were very gracious and who assured me that over time and with more subscriptions my articles would rise in the search results. But alas it is now mid-April, I have much more content and many more subscribers, yet the search results don’t seem to have changed. It finally dawned on me that the results aren’t going to change. The problem isn’t the newness of my material; the problem is that the Substack search engine delivers really poor results for everyone and everything.

The kicker came when I added an article containing the proper name “Soros” in its title. I gave the engine a few days to digest my new material, and then tried searching on “Soros”. The search engine ranked articles containing these terms:

·         somos

·         sorts

above articles whose titles or content contained “Soros”. Then came some exact matches, none of which was my article. Then came matches on:

·         more of the above (somos and sorts)

·         solos

·         sores

·         toros

It seems really odd that a search engine would assign a higher rank to many articles not matching this rather specific proper name than it would to my article containing the precise name.

My final dispatch to the Substack service folks on this subject was this:

There are billions of potential Substack readers in the world, and you'll need a lot of authors to reach any significant fraction of them. Your writers need all the help they can get. When an already-established celebrity becomes a Substack author, the quality of your search engine is probably immaterial; that person's followers will follow him/her around. Your big challenge with celebrities is to snag them.

But unknown authors like me are, well, unknown. Right now, Substack is losing money hosting my "free" content. I'd love to have paid subscribers, but it would be preposterous for me to require payment when I'm already having trouble getting readers to read for free. I need some sort of staircase where I can step up my readership gradually to the point where I could even contemplate producing paid content.

I spend a lot of time posting links to my articles as comments on others' social media posts (where the topic is appropriate). But my best source of new readers should probably be existing Substack readers. Substack readers, especially new ones, probably use Substack's own search engine when searching for content. My guess is that most such users have not yet realized that Substack's engine produces mostly nonsensical results, and that they'll never find most of what they're seeking by using it.

SO ... from my perspective, probably the most important thing that Substack could do to enhance its readership, including its paid readership, is to replace Substack's search engine with something that works.

So there you have it. What say, Substack? Can we have a new search engine? Thanks!

==== ADDENDUM OF 10/14/2024

So, in addition to the above (which has not yet been fixed), we Substack authors have a new search problem. Up until a few weeks ago, I could find my articles by title when using the Bing and DuckDuckGo search engines. For example, the following search strings (when submitted with the quotes):

    "Open Letter to Republicans and their campaigns"

    "No, we are not going to fix the public schools"

    "Climate Change is actually about going all-nuclear"

would each produce results in both Bing and DuckDuckGo with my article of that name near the top. This search-ability was essential when referring people to my articles on comment boards like Disqus, which prohibits the use of URLs in its comments; instead of posting a URL, I’d include the search string and tell readers which search engines to use (or which one to avoid).

In contrast to Bing and DuckDuckGo, Google would never find my articles by name. Until recently I presumed that Google was simply censoring all Substack content due to Google’s extreme wokeness. But then everything reversed. Google and the other two engines seem to have swapped places. Now, Google always finds my articles by name, and Bing and DuckDuckGo cannot find any of my articles.

So I started searching and found articles and threads like these:

• How I got my Substack to be Google searchable--and sadly, you probably cannot.
  

• Getting Your Substack Indexed on Google Search
  

• How do you get your substack site added to google search?
  

• Maximising Visibility: A Step-by-Step Guide to making your substack visible to Google search traffic

So apparently there is some technical fol-de-rol that has been making it impossible for Google to find my articles by name, even though the articles are sitting there with their names exposed for everyone, including Google, to find.

This recent change in search-engine behavior happened without my having done anything, so I must presume that someone at Substack changed things such that Google can suddenly find Substack articles.

Unfortunately, whatever Substack presumably did also made my Substack articles completely invisible to both Bing and DuckDuckGo, and somehow even made those engines’ vestigial memories of my articles disappear also. How is this even possible?

SIGH. This is getting really tiresome.

==== ADDENDUM OF 12/12/2024

Well the mystery deepens. I’ve done more research, and it now seems that my Substack account has specifically been blacklisted by the Bing search engine. To understand how I’ve arrived at this conclusion, I’ll need to bore you with some of my authoring history.

My History with Medium.com

1. I started writing articles on Medium.com back in 2016. My articles were on various non-political subjects, but as time went on they became increasingly political. You can see my history of Medium posts here: https://medium.com/@davidziffer
   

2. On March 3, 2023 I published an article called “You are Drowning in an Ocean of Lies”, simply containing a list of demonstrably false mainstream-media stories, with links to descriptions of them or to the stories themselves. Medium flagged that article with the following statement and removed the article from public view (which is why you cannot see it in my Medium publication history):

   

3. By the time I realized that my “Drowning” article had been censored, I had already written a couple more articles on Medium, neither of which had been censored. I appealed my “Drowning” article’s suspension and never heard back from Medium. At that point I decided to copy most of my Medium article series over to Substack and then publish my politics only on Substack.

4. Following my switch to Substack, I continued publishing on Medium, but I restricted myself to publishing only non-political articles.

My History with Substack

1. I started my Substack on March 7, 2023, by copying a couple of my Medium articles to Substack. Over the course of a few weeks I finished copying most of my Medium content over.

2. I then continued authoring my political articles only on Substack.

3. I quickly noticed that Substack’s own search engine was completely incapable of finding my articles by title, either with or without quotes. After a few days of struggling to find my own Substack content using Substack’s own search engine, I wrote the original portion of this article (see above).

4. I frequently comment on Disqus and other comment boards that do not allow links. To refer users to my articles, I generally embed the title of my article in the comment, surrounding the title with double-quotes, and instruct the reader to search for the article using the quotes, so that the reader will be searching for my specific article by its title. Usually my article titles are distinct enough that they’ll turn up as the very first search result. I provide some examples of such titles in my 10/14/24 addendum above.

5. Until somewhere in September 2024, I could always find most of my articles by title using either Bing or DuckDuckGo. (My understanding is that DuckDuckGo’s search engine is based on Bing, so henceforth I’ll just refer to the two as Bing.) During that time I was never able to find my articles by title on Google. I originally presumed that this was due to Google’s radically left-leaning tendency in its search results (which I discuss in part in my article Three Companies Now Run The World), but I later came to realize that Google wasn’t indexing any Substack content, and that a lot of Substack authors were complaining about it. SO .. when telling users how to search for my content, I’d tell them to avoid using Google.

6. In September 2024 Google started indexing Substack content, for reasons that I do not understand. I now get the same sorts of results when searching my articles by title with Google that I formerly got with Bing.

7. At that same point, my articles disappeared entirely from Bing searches. This radical change in both engines seemed too close in time to be a coincidence.

So to summarize:

• I have a bunch of old articles on Medium up through March 2023 that also exist (mostly verbatim) as Substack articles.

• From 2023 until September 2024 I was always able to find my articles by title using Bing, and I was never able to find my articles by title using Google.

• But in September 2024 Bing and Google switched places. Now I can almost always find my articles by title using Google, and I’m completely unable to find them using Bing.

My theory about this, given the timing, was that Substack had made some technical change that enabled Google at the expense of Bing, such that Bing was now incapable of finding Substack content (just as Google formerly was). But then I realized that Bing does find other Substack articles by title; it seems to be just my Substack articles that it cannot find. My research shows:

• If I search for one of my articles that appears under both Medium and Substack (example: "Seventeen Mail-In-Voting Security Vulnerabilities"):

  • Bing will find the Medium article but not the Substack article.

  • Google will find the Substack article (and others’ cross-posts of the article if they exist) but generally not the Medium article.

• If I search for one of my articles that appears only on Substack (example: "Democrats repackage debauchery and death as joy"):

  • Bing finds nothing.

  • Google finds it, and if the article has been referenced by others, it also finds the references.

• Here’s the kicker: If I search for one of my articles that has been copied verbatim into another Substack author’s article (example: "Minnesota Elections are a Farce"):

  • Bing will find the other author’s version of it, but not my original version.

  • Google finds both our versions and puts my version first.

Because of this final item above, it would seem that Bing has a vendetta against me specifically. The conclusion seems inescapable: Bing will find another Substack author’s reprint of my article (with the same title and content), but it will not find my original version of the article.

The question now arises: why me? I’m a tiny fish with only 555 subscribers at this writing. It seems both preposterous and presumptuous to imagine that the great and mighty Bing would be taking the trouble to specifically censor little me. But on the other hand it might not have taken much to get me censored; all it takes, probably, is one annoyed reader who has some clout to somehow submit a complaint. I’ve sent emails to all my Republican state legislators on three occasions, each with a link to one of my articles. Maybe some powerful people, presumably on the other side of the aisle, have got enough influence to get me canceled.

Or it could be that Bing has decided to blacklist the articles of Substack authors who have small numbers of subscribers. Who knows?

All of which makes me contemplate continuing as an author. On my current trajectory and without censorship, it’d probably take me another five years of plugging away weekly to establish any significant readership. And even then, my fate would be in the hands of a single corporation (Substack) that might change owners and/or policies on a whim, and beyond that I’d be at the mercy of this social-media and search-engine empire whose workings are probably beyond anyone’s comprehension.

And even beyond that, there is the utterly unpredictable course of American politics. Trump’s coming second term could be a rebirth for free speech in the USA, but what then? His successor could be another Biden; such an administration could be the ultimate death of our Republic, and the first casualty of that death would be my freedom to speak my mind on any platform.

Yup. I’m starting to think this all just isn’t worth it.

==== ADDENDUM OF 12/28/2024

For everyone’s entertainment, I opened a complaint on the Microsoft Community web site. Enjoy reading as I try to convince a Microsoft employee that there is some sort of problem:

Bing blocking my Substack articles from search results since September 2024