Ask Jim: Blocking Google’s search bot without blocking Adwords
Hi Jim,
If you built a webpage and blocked it from Google bot via robots.txt file and that same landing page was used for an Adwords campaign, does Google still read that page regardless of the disallow instruction in robot.txt?
Is there a separate bot for Adwords and for standard search (SEO)?
I assume Google still has to read the landing page to ascertain keyword quality scores and ensure it meets their guidelines, but just wondered if you knew the definitive answer/process?
Cheers, 
Steve.
Hi Steve,
If you’ve blocked your site from bots in the robots.txt file, Google’s bots will not be able to read your page for either search or Adwords, which will make your Adwords quality score suffer.
While Google does have separate bots for search and Adwords that can be blocked individually, blocked bots still have the ability to read your URL, which can result in your page still getting indexed by Google.
The best solution for your site is to allow all bots and simply add the noindex meta tag onto the pages you want to block from search. That way, the Adwords bot will still be able to determine your landing page quality, but Google’s search bot will determine that the page is not to be indexed into search results.
To block robots from indexing your page, put the below meta tag into the section of your page.
<meta name="robots" content="noindex">
Hope that helps!
-Jim
Got a question for Jim?
Send your question to [email protected] or get in touch on Facebook or Twitter.
Jim’s been here for a while, you know who he is.
