.Google.com's John Mueller answered an inquiry about why Google.com marks webpages that are actually forbidden coming from creeping by robots.txt as well as why the it's safe to dismiss the similar Search Console reports about those creeps.Robot Web Traffic To Concern Specification URLs.The person inquiring the inquiry documented that crawlers were developing web links to non-existent query specification Links (? q= xyz) to web pages with noindex meta tags that are likewise obstructed in robots.txt. What urged the question is actually that Google is actually creeping the web links to those web pages, obtaining blocked through robots.txt (without watching a noindex robotics meta tag) after that getting turned up in Google.com Look Console as "Indexed, though blocked out by robots.txt.".The individual asked the observing concern:." Yet right here's the significant question: why would Google mark pages when they can not also see the material? What is actually the benefit in that?".Google.com's John Mueller verified that if they can't crawl the page they can't view the noindex meta tag. He also helps make an exciting acknowledgment of the internet site: hunt operator, suggesting to dismiss the end results since the "average" users will not view those end results.He wrote:." Yes, you are actually correct: if our experts can't crawl the webpage, we can not find the noindex. That mentioned, if our team can not creep the pages, then there is actually certainly not a whole lot for our company to index. Thus while you may view a number of those webpages along with a targeted web site:- concern, the average user won't find them, so I definitely would not bother it. Noindex is actually additionally alright (without robots.txt disallow), it merely indicates the Links are going to end up being actually crept (as well as end up in the Browse Console document for crawled/not recorded-- neither of these statuses induce problems to the remainder of the web site). The essential part is actually that you do not make them crawlable + indexable.".Takeaways:.1. Mueller's solution affirms the restrictions being used the Site: search accelerated search operator for diagnostic factors. Among those causes is actually because it is actually certainly not connected to the regular search index, it's a different factor completely.Google's John Mueller commented on the website hunt operator in 2021:." The quick answer is actually that a web site: question is not implied to be complete, nor used for diagnostics reasons.A website concern is a particular type of search that confines the end results to a certain website. It is actually generally simply the word website, a bowel, and after that the web site's domain name.This inquiry confines the results to a details web site. It is actually certainly not indicated to be a comprehensive compilation of all the webpages from that internet site.".2. Noindex tag without using a robots.txt is actually great for these kinds of conditions where a robot is actually connecting to non-existent web pages that are receiving uncovered by Googlebot.3. URLs with the noindex tag will certainly produce a "crawled/not catalogued" entry in Browse Console which those will not have a damaging effect on the rest of the website.Check out the inquiry and respond to on LinkedIn:.Why would Google index pages when they can not also view the web content?Included Photo by Shutterstock/Krakenimages. com.