We did it. We solved one of the unsolved big SEO (Search Engine Optimization) mysteries of the modern time. It took quite some time, dragged us down in the deep pits of the TCP/IP and HTTP 1.1 specification, but finally we emerged victorious.
What’s the mystery:
Sometimes in Google Webmaster Tools -> Sitemaps you see error messages like: “We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.”
Then you start investigating: yes, the file is there, yes, it is accessible for googlebot, yes, it has content, yes, it’s like specified on sitemaps.org.
Then you dig deeper: the error logs, the access logs, all logs, and then you realize: that f*cking GET request does not exist in the timespan Google reported (to be sure you look at a longer timespan, still nothing there).
No GET request!
You look at the network, you look at the DNS, could it be that the requests went astray. You dig and dig, even bother to write a ticket to your server housing company. Still, nothing comes up, no leads.
Then you dig deeper: TCP/IP HTTP 1.1
- You realize that googlebot makes multiple (we counted up to 11) GET requests in one single TCP/IP connection. (which is OK according to the HTTTP 1.1 spec).
- You realize (with the help of stackoverflow) that these multiple GET requests in the some TCP/IP connection are processed in sequence (one after the other).
- You realize that if one these GET requests has a major time lag (is much slower than the other GET requests) Google cuts the TCP/IP connection.
- Because all the GET requests in the connection were processed in sequence, all the GET requests after the cut are lost. You don’t see them in the error/access logs as they were never processed, even though they were sent.
- You see an error in Google Webmaster Tools, without a trace in your logfiles.
SEO Mystery solved.
If you don’t understand a single word i just wrote, please remember, we are geeks.
15 Comments
Nice discovering friends. I have to test that immediately. How did you find out about that – is there a log or tool that enables you to see which http requests share the same tcp connection?
Nice discovering franz. Thanks for sharing
You just saved SEO managers a lot of time!
first: thx
second: dedicated servers in a high quality server housing environment, the “slower than the rest” requests where something out of the ordinary, an issue within the “garbage collection” logic.
Franz,
Great investigative work to nut that one out.
In your particular instance, was the ‘slow’ request anything out of the ordinary or just your server deciding to serve that particular resource slower than others? On the server front, is your site hosted on a good quality hosting or is it on cheaper style hosting?
Al.
That is some next level analysis!
very interesting. I have to test that immediately.
This is great news! I also have problem like this. Will try what you did and maybe get mine to start working. Thanks.
SEOP.com
I’d say serve the googlebot HTTP/1.0. That way it won’t use keep-alive, or just configure apache to turn off keepalive for googlebot.
no offense or anything, but I wouldn’t call it “solved”, feels more like you identified it, which is already a big step, I suppose…
Interesting–good find.
Yeah nice discovery. Now what?
@til: As Franz said, we captured a pcap file with tcpdump:
tcpdump -w foo.pcap net 66.249.0.0/16 # ‘tcp port 80 and (((ip[2:2] – ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)’
The download the aptly named foo.pcap and analyze with Wireshark.
we used an installation of http://www.wireshark.org/ on the server.
>make sure that all requests are getting served fast enough?
that’s the funny thing, all request must get served – pretty much – equally fast. the googlebot decision if a connection gets cut is measured relatively. if you have a “fast, fast, fast, very slow, fast, fast” package, googlebot will be likely to cut the connection at “very slow”. if you have “slow, slow, slow, slow, slow, very slow, slow, slow” package, googlebot does not seem to cut the connection. (but will spend much less ressources on the site in general).
so yes, the best way is to server all request fast-er, but you could also deliver all request equally slow (not recommended).
Wow, interesting. How did you find out about that – is there a log or tool that enables you to see which http requests share the same tcp connection?
And what can one do against it – make sure that all requests are getting served fast enough?