I complained to Google (with an online spam form, not a DMCA letter or any formal avenue) and they have been removed from the index. I also emailed their hosting company and the site is now down.

They were definitely doing it deliberately. Their URL resembled that of a legitimate site, and their front page redirected to proxy that site.

Thanks for sharing this info. WS

There's no difference between a spoofed header from curl and any other spoofed header.

Even if it were possible to fake both ends of the forward and reverse lookup, it's hardly necessary, if your only goal is to construct a proxy that will fetch a page and return it to a search engine's bot.

All you have to do is look like a normal user, strip out robot meta tags and X-Robots headers, and pass the result back.

If this problem reappears (Google seems to have solved it for now), and a bunch of idiots start doing that, we have a number of additional countermeasures which would thwart the attempt.

I will not discuss the nature of those countermeasures unless it becomes necessary to engage the issue again.

This would at least protect you from the home made proxies.

But I did a search for checking if a request is curl_init and found nothing.

As far as someone pretending to be an IP that they are not is very easy to do! Piggyback on another IP! Have that IP make the request. You can also use IP spoofing to make the handshake.

Read this. http://www.computing.net/answers/programming/php-remoteaddr-integrity/9140.html

Tell us if you find a way to check for curl_init() request.

What you describe is a pretty standard step in preventing content theft – this particular issue is a bit more subtle.

That's a nice way to identify bad bots if you disallow /trap in robots.txt – since the bad guys rarely bother to read that file, and if they do it's usually the first place they look.

But it wouldn't do much good when you're trying to identify good bots coming in by proxy, unless you plan to allow them to spider past a hidden link.

It also would not prevent them from grabbing other content (like your home page) at a proxy URL while you're waiting for them to request the trap page.

Then this script takes note of the IP address and adds it to your .htaccess script blocking the IP after validating if it is a bot you want to deal with (using ARIN lookups). You can have other scripts that expire the blocks or count the number of return offenses.

Drak

Very good and logical question. Not sure about this. I would imagine not, but who knows! Is the response based on the TCP/IP handshake or on browser sessions?

Need to consult Dark Vader on this and will get back to you as more information gets illuminated from the Dark Side.

May the force be with us to defeat Evil Google!