.Google.com's John Mueller addressed a question concerning why Google.com indexes pages that are forbidden coming from creeping through robots.txt as well as why the it is actually risk-free to dismiss the similar Explore Console files concerning those crawls.Bot Traffic To Query Specification URLs.The person inquiring the question recorded that bots were actually producing hyperlinks to non-existent concern specification Links (? q= xyz) to webpages with noindex meta tags that are likewise shut out in robots.txt. What caused the inquiry is actually that Google is actually crawling the hyperlinks to those webpages, obtaining obstructed through robots.txt (without noticing a noindex robotics meta tag) at that point acquiring turned up in Google.com Browse Console as "Indexed, though blocked out by robots.txt.".The individual asked the adhering to inquiry:." However right here's the huge concern: why will Google index webpages when they can not even find the web content? What's the advantage in that?".Google's John Mueller confirmed that if they can not crawl the webpage they can't observe the noindex meta tag. He likewise produces an interesting acknowledgment of the web site: search driver, urging to neglect the end results given that the "normal" users will not view those results.He composed:." Yes, you are actually proper: if our company can not creep the page, our team can not find the noindex. That claimed, if our company can't crawl the webpages, at that point there is actually not a lot for our team to index. Thus while you might see several of those webpages with a targeted internet site:- question, the ordinary consumer won't observe all of them, so I wouldn't fuss over it. Noindex is actually also alright (without robots.txt disallow), it only means the URLs will definitely find yourself being crept (and end up in the Explore Console file for crawled/not catalogued-- neither of these conditions cause concerns to the remainder of the website). The vital part is that you do not produce all of them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the limitations in operation the Site: hunt advanced search operator for analysis reasons. Among those factors is due to the fact that it is actually not attached to the normal search mark, it is actually a different factor altogether.Google.com's John Mueller discussed the internet site hunt driver in 2021:." The short answer is actually that an internet site: inquiry is actually not indicated to become full, neither utilized for diagnostics purposes.An internet site query is actually a particular type of search that limits the outcomes to a specific website. It's generally simply words site, a digestive tract, and then the website's domain.This inquiry limits the outcomes to a details site. It's not indicated to be an extensive assortment of all the webpages from that site.".2. Noindex tag without using a robots.txt is actually fine for these sort of situations where a crawler is linking to non-existent webpages that are acquiring found through Googlebot.3. Links along with the noindex tag will definitely generate a "crawled/not catalogued" entry in Search Console and also those will not possess a negative effect on the remainder of the internet site.Check out the concern and respond to on LinkedIn:.Why would Google.com index pages when they can not also see the web content?Featured Photo through Shutterstock/Krakenimages. com.