Video web crawlers and the Google Monopoly over them.

Happy 2021 - Web crawling only gives you YouTube first, last, and always.

Back in 2016, we made an article about asking YouTube "Where is the fair use?" which was a slogan that the Nostalgia Critic was behind as parodies and criticism of film and literature is a form of free speech. Despite all of the Drama-Lama action that his agent "Channel Awesome" happened which is beyond the scope of this article. The statement was a valid question to ask YouTube. To the point where we even got hit with a copyright strike over the song "William Tell" which is a classical music song that falls under the category of public domain.

Read on if you want to know more.

"Note: If you get a black screen but the audio is playing. Or you got a potato PC and it's studdering. We have fallback Links below."

Video tutorial fallback mirrors:

In case you have no-script enabled or for some reason cannot see the title video on this website. We have provided direct links for these videos. For more information about the standards we use on this site click here if you would like to know more.

  
AV1       - Link
WebM VP9  - Link
OGV       - Link
MP4/.h264 - Link

We will try to keep this article as up-to-date as best as we can. However, since we are talking about search engines which is an ever-evolving technology you have to take the time this article was published which is January 01, 2021, at face value. If anything this posting should serve as a historical point within the internet as to how the normalcy of the net has become.

Humble beginnings.

We were happy that Google was indexing our website and categorizing the hard work that we put into our videos and respectfully placed not only our site into the videos tab but also indexed those videos as we were also using later encoding technologies such as WebM. Although MP4 video encoding has gone the way of the dodo on this site Google continues to index our videos and respectfully place them into the "Videos" category of their search engine. It is our guess that their web crawlers go through the content of each of our blog articles and upon finding a link to a self-hosted video another half of Google's crawler goes to work indexing said video.

This meant that if someone typed in this very website "s-config.com". Our videos show up before YouTube. As it should because the domain and text coming from that domain should be a priority over a third-party site like Vimeo, YouTube, Etc. This was good news, because if more people self-host their videos then in turn those people become less dependent on YouTube's tyranny.

The other sites.

What about sites beyond the empire of Google? Sure, they do not have as much pull as Google does here in America. One can even make the argument that it's virtually impossible to compete against Google. That Google in itself is a Monopoly. But really how far does that go? If other websites are using their own engines doing the same thing Google is doing then surely they may even provide a better result structure than Google itself, right?

Well! Let us find out:

Duration and environment.

Just for reference party people. My site has been online for 8 years with maybe an hour of downtime per entire year. It's safe to say that thanks to AnubianHost my uptime has been about %99.8 and the %00.2 percent is mostly us being dumb-asses modifying code on a live production server instead of having a secondary backup WordPress site within our VPS. Since my site has been up 8 years with only 2 theme changes there really should be no excuse for any of the crawlers to do a good job at indexing this site. Hell, even the internet archive has a few of our pages.

The relevance of the "Video" tab.

I had some friends that legit said this to us.

I don't even use the video tab on a search engine. I just go to YouTube. - Anonymous

Here in lies the problem. Society is trained so well to go to one source for all of its information that they dare not try other methods. It's a bit of a chicken-vs-egg mentality. How is a search engine supposed to get better at finding things if no one uses the search engine? Especially if the search engine decides that if your site isn't Disney clean then you minus will be blacklisted like a *****o service.

But looking at the egg. Could it be the quality of those methods which is why no one uses them? There could be a logic behind this statement. Let us find out.

The baseline comparison - Google at #1

Typing my domain name into Google does not change a whole lot. Because we know that google tracks the living hell out of you. We even used our Tor client and performed the same results in other countries just to ensure there was no favoritism at hand because we go to our own site so much. The articles shift a little under our site but the structure generally remains the same.

Clicking on the tab for videos yields that Google is still doing a great job routing people to my site. That although the content shifted away from Ouya and more into gaming controllers it's still pointing to videos that exist on my site.

Google has been picking up more and more of our technical articles than art which just means we need to sit down and do more drawing to balance things out. Despite all of this Google stays focused on my site for a good page and a half before it starts looking at other web pages, social networks, and thumbnails. Which is very good and what we would expect if someone typed in our domain name.

Bing comparison test - #2

Bing pushed hard into the market in the mid-2000s powered by Microsoft as being a competitive search engine to Google. Bing claims to use its own methodologies of indexing and even has its own webmaster utility on ranking better with Bing. Looking through our own statistical analysis Bing only directs people to this site about %4 percent of the time in contrast to Google's direction of traffic at %64.

Bing search results start off similar to Google with a random selection of articles. Strangely the second result is yet another hit to my site referencing one of our LSD-driven short stories. and then it proceeds to reference other sites that we published works on and eventually forums and wikis there-after. Perhaps Bing likes stories more so than Google?

 

The video search bar on Bing has to be the most disappointing out of all of the engines out there. Although if you search the forums of Bing; They claim that they index DailyMotion, Vimeo, YouTube, and even indexes videos on ABC, NBC, TMZ, and The associated press.

Since we never made it to big-time news outlets (Nor would we want to) It gives this illusion that S-Config.com does not even exist. If you start clicking through some of these videos it's YouTubers referencing to my site over one topic or another which is totally cool. However, Bing never indexes videos that exist on this very server. We have sent an inquiry about this on the Microsoft WebMaster Bing Forums however we are still awaiting some form of response to this.

This to us is super odd. Because Microsoft wants a piece of the market and wants to provide people with accurate and relevant searches for what they are looking for. But completely ignore the small-time webmaster despite offering tools for people like us to use. On one hand, you have Microsoft asking people to use their search engine. Only to turn around and divert all of the traffic to YouTube either out of incompetence, laziness or perhaps they were bought out by Google in this department. No one but the developers of Bing will truly know this answer.

Perhaps we talked some crap about how they performed planned obsolescence on Xbox 360 controllers. We doubt this as we're rather sure Microsoft has some rather thick skin when it comes to criticism.

 

Bings imaging results end up becoming increasingly more chaotic than Google. The third picture taking you off to a YouTube video named "Config" with someone from Asia appearing to be setting up their key-map controls for Monster Hunter. In essence, you start to see Bing's engine fall apart by not honoring the quotations as Google would. However, on the more positive side, Bing tends to grab a lot more colorful articles we've done throughout our time instead of just screenshots. Which is nice.

SafeSearch off.

Apparently, Bing's infinite wisdom has decided that my site is unsafe for *****ren. Talking about Raspberry Pi's, Xbox controllers, and android installation guides is the same as looking at hardcore *****. Brilliant.

It should also be noted that a lot of the videos bing is picking up a reference to older videos when we used to host MP4 files. Indicating that Microsoft is still holding onto dying codec standards of the .h264 as the only means of letting users within their search engine know that you host videos. We've asked the Bing Webmaster forums several times about how Bing processes videos and in general, we were met with absolute silence. Perhaps they're too busy. Or, perhaps they simply do not care.

DuckDuckGo comparison test - #3

DuckDuckGo's big deal is they are effectively Google but without the song from "The Police - Every breath you take" playing in the background while you type away. They do not spy on you. Because of this DuckDuckGo will not generate a 'bubble' of preferred searches based on your interests. Only show you the raw results.

DuckDuckGo's search results mirror more of Bing than Google in terms of structure. Again, focusing on articles we've made before diving into forums and wiki's.

 

DuckDuckGo's video search results are more disappointing than Bing in regards that for a company that is super-concerned about the privacy of its users, it recommends going to a Sister company of Google that tracks everything you do. You could argue that depending on what you are searching for a YouTube link not being a priority is impossible. As we've stated with the Bing results. My videos are locally hosted. There's absolutely no reason for other search engines to index/verify the videos that we post out there.

One last thing for those interested. we did turn SafeSearch off on DuckDuckGo as they tend to bury it into the settings panel of their search engine. Even after disabling SafeSearch, we did not show up still. Indicating that either DuckDuckGo is also looking for MP4 files but as we serve those anymore this could result in a total absence of results going to our domain.

Also, note that it found out twitch profile. That profile has absolutely nothing recorded in it. Yet, it's showing in the videos tab which is inherently wrong.

Again, the video results from DuckDuckGo have that eerie Microsoft Bing vibe to them. This of course makes us wonder if DuckDuckGo is relying way too much on the same search engine mechanism that Bing is using as well,

Additional DuckDuckGo hilarity

It appears that AFTER you click on a YouTube video link, and then tab back the DuckDuckGo reminds you in the most passive-aggressive way possible that it sends you to a website by clicking on the text. YouTube does not admire your privacy as much as DuckDuckGo does. How about this DuckDuckGo. You link videos on my site which has a far better track record of privacy than YouTube and you don't have to show warnings like this. No? Oh, okay.

If you click on the images within DuckDuckGo only then does it show you the warning about Privacy in Advance.

Dogpile comparison test - #4

Whoah, we haven't heard from this engine since the mid-2000s. However,  they are still in the game and they too have a images and video tab for us to check out. Perhaps the underdog of search engines will surprise us here.

So it's not as structured mark-up happy as the other three search engines. But for the most part, it stays on target with my website first, then other places that we've published second. The site also wants to suggest meta-data that someone might be looking for the command line "sconfig.exe" from Microsoft instead of me. We kind of expected that too.

Mmmmmm, my favorite way of viewing images is if we're on a tablet despite the fact we are on a desktop. Kind of a weird vibe when it comes to displaying the images in a mosaic fashion. But regardless it's similar to Google with filling up the first page with images from my domain before moving on to other sites.

Dogpile needs to some tender loving care in the CSS department and build themselves a responsive website that can handle dynamic resolutions. Despite this, the search results are identical to DuckDuckGo with misfiring the same Twitch profile as well as repeating the same videos just in thumbnail/description mode instead of just picture mode.

After turning SafeSearch off and content moderation off on DogPile, it crashed, reloading again. Once again nothing about my site. Disappointing.

Yahoo comparison test - #5

Yahoo has had a decline in its search engine almost to the point where it's laughable to even use them as a search engine. There was a point during y2k that Yahoo felt almost unstoppable. We were even using them to plan to go on vacations. With endless security breaches and it's a miracle the company is still alive. Yet, it's still around because Yahoo wants to become a media/news site and less of a search engine. Since if you go into the search engine it has the same tabs as everyone else we're giving them a try next here.

Hmmmm. The results certainly scream Bing on this one.

The image results feels like this is what DogPile would look like if they actually fixed their site.

If you replace the Yahoo logo with Bing No one would know the difference.

Again, with SafeSearch off. My older content that used to my MP4 appears on the listing.

Bonus Round! Presearch!

There's a lot to unpack about PreSearch. Click here if you want to see that article.

Bonus Round! Brave Search Engine (Beta)- 04/18/2022

So brave decided to jump into the search engine game by scraping other engines without verification for their end results.The filter system is known as a !bang. Generally, it's not fair to line up a beta against everything else. But let's see how it fairs and hopefully, they will fix it in the future.

I should note there is one thing that seriously cripples our ability to measure this search engine which is unlike all of the other search engines Brave does not deal with absolute search data when you type something in quotes. In other words.

"s-config.com"

would mean the same as:

s-config.com

which means anything with the word "s" or "config" or "com" will pop up in a situation like this.

Okay, no real structure markup. and you can already see the search engine failing by bringing in random websites like the ARK PC Game.

Okay, Brave seems to do a rather good job staying on target with content coming from our website which is a good thing.

Regardless if safe-search was on or off Brave failed us in the video department horribly by simply breaking down the exact words we're looking for into key phrases like "Config" which of course not only takes people away from my site but takes them over to completely irrelevant YouTube videos.

At around position 39 is when we finally start to see the videos that we published. At least that's something in the respect that even with safe-search set to moderate we still show up. However, Brave effectively buries our videos despite the name we're searching for being within the domain of where the videos are linked.

So whether brave is understanding this or not. They're just falling in line with everyone else helping YouTube become the dominant video-sharing service in America.

Self-Assessment time.

Perhaps we're doing something wrong? Perhaps we're unanimously pissing off every search engine in existence? Well, we actually did find some things wrong with our site.

One of those things is the usage of sitemap-video.xml . Apparently, every plugin known to WordPress Claims to reference your Videos and catalog them into a sitemap-vimeo.xml for not only Google to catalog but for Bing and every other search engine. Sound amazing? Too good to be true? Well, it is!

Every single sitemap plugin we tried completely skipped over our self-hosted videos and only published a sitemap of YouTube links that we used purely for reference in some of my blogs. Great! So WordPress is flooded with plugins that shill for Google, YouTube, and Vimeo, Fantastic.

To make matters worse. Because of my gallery system that we use for video presentation. The videos are linked to the gallery and not to the blog itself. So even if these plugins were working according to plan there's no way in hell they would be able to associate what video goes to what blog entry.

After a bit of reading, we wrote our own sitemap-video.xml . It was a kind of a pain at first. But realistically since we publish a handful of videos it will be easy to maintain in the future.

I pointed out all of the webmasters to take a look at it. They acknowledge that it's a valid XML with 30 entries. But.. Alas, no change really.

Asking for help from the search engines themselves.

I first asked the Bing Webmaster forum for support. The only response we received was from a person who was wondering what video hosting service I was using. Vimeo, Youtube. At which we responded with self-hosted which brought forth silence thereafter.

The next I asked was DuckDuckGo which ironically uses Reddit which violates the privacy of its users as much as FaceBook. Learning from my mistakes on Bing we explained the situation in detail. and received nothing in return.

Finally, we asked about the forums on Brave. Silence.

Apparently, the concept of self-hosting videos is alien and/or foreign to most understandably. But also bitterly sad at the same time.

Where do we go from here?!?

Now, a blog is supposed to help its readers. After all, if you've read this far you almost deserve a possible solution. Lights at the end of the tunnel as it were. To people like us, this is a lot of doom and gloom. But the reality comes down to how you feel politically about YouTube and the internet ecosystem that surrounds it.

For example, if you are willing to be a bit of a conformist the best solution is to use them both! Have a website that you promote at the end of each of your videos with a blog entry going to your YouTube! However! Also, encode your own videos and attach them to the blog as a backup like how we're doing it!

Yes, you're going to be exploited by YouTube like a person who just entered Hollywood. Yes, you might even get some of your videos demonetized or even pulled off of their network. But understand that if you have your own website with backups and you keep telling everyone about your personal site at the end of your videos and even on your profiles. They will come to see what was so terrible about your video that it was banned from YouTube.

Also, if you preserve your MP4s and spin the legal roulette wheel with MPEG-LA's vague laws of usage. You might even have better luck having your encoded videos searchable on networks beyond Google.

Small-web complex?

We acknowledge that we are a small website. Thus, it would be very easy for the big search engines to overlook us hosting our own videos. Or, perhaps they simply do not want us small-timers 'poisoning the well' as it were.

When you start going from search engine to search engine. It becomes evident. Unless you're a major media outlet or Google, or you've been around since the dawn of time like Vimeo. you're not getting any traction to your videos. Period. Independent hosting like what we are doing can't even turn to projects like PeerTube because as discovered with Mastodon a decentralized network does not play nice with us a centralized network. In order to be acknowledged on PeerTube, we must join PeerTube and participate in their network. If it behaved more like Tor/Onion networks it would be acceptable.

Unfortunately, that is not the case. As that too is subject to community-based censorship. Federated level fragmentation is discussed in this blog entry. So ultimately you will never see the whole picture with a service such as PeerTube. Perhaps that is also by design.

You find out that competing with YouTube is a lot like your local store competing against Wal-Mart. You can't. They have more money than you, more clout than you, and will erase you out of existence in every way. Now, if blogging was like our livelihood we would be devastated at this point.

YouTube knows this and does not care about your intellectual property.

YouTube is very much aware that there is nowhere for a video blogger to go. This is why they decided to steal revenue from hard-working video producers. The very thing the music industry accused us of doing before we left YouTube. Total and complete hypocrites by nature so long as they get their ad revenue.

YouTube is the criminal stealing from artists around the world!

Individuals have no powerful law firm and no agents that represent them. Of course, We know this is what parliament/congress/senators are supposed to be there for. But with millions of dollars from Google funneled into their campaigns, it's easy for them to look the other way. That and most are so old they don't even know what a computer is!

A giant middle finger to your copyright claim and if you don't like it you CAN take your ball and well... Be erased from existence.

We have no rights, no permission over our own works when you submit anything to YouTube. If you go anywhere else on the net. Even the startup video hosting services. Prepare to be ignored by every major search engine and web crawler in existence because for all we know every one of them Google's makes sure that their service and ONLY their service appears on these engines. Face the void and be forgotten. That, my friends, is where the real monopoly is. You have to go out of your way to use another video service and even when you do you end up with minimal content.

Final thoughts.

We could keep going on with other sites such as SwissCows, or StartPage, or half of what the sociopathic bloggers talk about when it comes to the 'alternative' Google search engines. When the truth is. There is no 'alternative.' Just other sites putting a coat of paint on the same product. Google is the Monopoly for America and most of the world. The first page of almost any topic you type in will be filled with google-only content. They will steal other people's data such as answers, recipes, and formulas, and paste it onto the search engine so you never have to leave Google. Unfortunately, the American government does not seem all that keen on changing the direction of how information and innovation flow. They're happy with stagnation. They're happy with everything existing as it is.

The fact is, we'll face the void alright because throughout the course of my existence, I have deprived YouTube of about 20 dollars. Now, the conversion of investment in the site versus a loss of revenue is terrible. But it's the little victories that we like.

Anyhow, for those who want to give it a go at self-hosting video blogging. Have the endurance of a God and balls that clink when you walk. Because it's going to be a long haul. Until next time.

Server protect you.

+++ END OF LINE