Using Site Diagnostics

In order to track down all the data needed for ad placement — as well as tracking
down all the data that goes into the reports I talk about in this chapter —
AdSense needs to be able to crawl your Web site. The crawler — not the same
crawler that Google uses to include your site in search results, by the way —
visits your site once each week to check for content, design, and usage information.
I cover how this works in a lot of detail in Chapters 2–4, so if you need a
refresher, you can flip there. I’ll wait.
Done? Good. The AdSense crawler visits your site once each week. During
that visit, the crawler looks at all the pages on your Web site to ensure that
your ads are properly targeted. If the crawler encounters a problem, such as
a page that’s redirecting unexpectedly (and without the proper redirect information
in place), or if the crawler is denied access to the site, it flags that
issue in the Site Diagnostic area of your AdSense account.
To get to the Site Diagnostic area, do the following:
1. Sign in to your AdSense account.
2. From the Reports tab, choose the Site Diagnostics link.
When that page opens, a list of problems that were encountered with
your site is displayed.
If the crawler didn’t encounter any problems, the list is mostly blank,
displaying only the You Currently Have No Blocked URLs message in
light gray.
If you do have issues — if the crawler can’t get to your site for some
reason — the reason is displayed in the table provided along with the
date of the last crawl attempt on your Web site.
It’s possible (though highly unlikely) that when you updated your site, Google
was trying to crawl your site at the exact same time. Sometimes, if you’re
uploading files, the crawler can’t access the site at the same time. If that’s the
case, the date of the last crawl reflects that.
Far more likely, however, is that a flaw in the site design or some other problem
is keeping the crawler off your site. If that’s the case, the error displayed
helps you to figure out what needs to be done to allow the crawler to have
access.
In a lot of cases, the problem is simply that you have a robots.txt file that
denies access to certain portions of your page. (Remember this little bugger?
I tell you all about him in Chapter 3.)
Sometimes, you deny a crawler access to a page on your Web site for good
reason. For example, if you have several pages that are essentially the same,
you might not want a crawler to look at all the pages. The redundancy could
cause your search result rankings to drop. Usually, if you’ve designed your
site well and with the Google Webmaster Guidelines in mind, you have no
reason to keep the crawler off any pages of your site.
Sometimes, though, the errors that the crawler encounters aren’t directly
related to that robots.txt file. If that’s the case, you see the exact error
listed on this diagnostics form. You can then repair the error. After you do,
however, it might take up to a week for the crawler to hit your site again.
You may have to be patient to see the results of your repair.
A good example here is when a crawler can’t navigate your site because
of broken links. If the crawler can’t follow the site navigational links, it
will leave your site and come back at another time. If you fix the broken
links, the next time the crawler comes through you shouldn’t have any
problems.