Do I need to block the Yandex Bot?

408    Asked by AndreaBailey in SQL Server , Asked on Jan 4, 2022

Should I block Yandex Boy or take any other action?

A blog that I was going through explained that Yandex Bot – Yandex bot is Yandex's search engine's crawler. Yandex is a Russian Internet company which operates the largest search engine in Russia with about 60% market share in that country.

Answered by Andrea Bailey

First, if the bot is a legitimate search engine bot (and nothing else), they won't hack you. If not, blocking a User agent won't help, they'll just use another one.


I would like to point out that Yandex Bot as well as other search engine bots in general might not intentionally want to access your backend. Remember bots are crawling the sites by following the links, so imagine if the bad guys would put some of your backend's urls in some other website's pages, and the search engine simply indexed those pages and now is trying to follow the links from there. So, it will look like the search engine is trying to access your backend - but it is just crawling the net: it does not know that it is your backend. Similar thing might happen by accident. Lets say a non-tech savvy user posted a url in some forum, that is only accessible when you are logged in - by crawling the search engine will try to follow those links and you will end up seeing logs as I assume you did.

UPDATE: I think you might want to set in your robots.txt rule to disallow yandex to access specific urls. Btw, you better define specific rule with its name, I am not sure, but it might happen, that yandexbot can ignore User-agent: *, so you can do smth like this(according to your backend urls) User-agent: Yandex Disallow: /admin/* So, in this way you will disallow it to try to access backend urls - matching that pattern, but at the same time it(yandexbot) will be free to crawl other pages of your website.



Your Answer

Interviews

Parent Categories