Bot Filtering: AdPlugg Takes on Bot Traffic


What do the Terminator, the Star Wars films, the Matrix, and AdPlugg have in common? They all feature epic battles against robots :-).

AdPlugg’s battle is against bot traffic penetrating statistical data. This bot traffic can skew results, show false impressions and even worse, false clicks. This is a major concern and one that we at AdPlugg take very seriously.

So what is a “bot”? Bots are automated programs that browse the web. They might be browsing for all sorts of different reasons but the most common is for creating search indices. Bots that crawl the web are known as crawl bots or “spiders”. Of the crawl bots, the most well known is Google’s Googlebot. Crawl bots read a page and then follow all of the links on the page. The bot then does this again on the next set of pages, creating a spiderweb of linked pages that make up the known web.

This can be problematic for online advertising as online ads aren’t the same as other links. Online ads are designed to be viewed and followed by humans only. This is for two reasons. The most important being that advertisers buy ad space to get impressions and clicks from humans, not bots. Also important is that paying another site to link to yours is a “black hat” practice that can get you penalized or even banned from search engine results.

To combat the issue, ad links are required to include a ‘rel=”nofollow”‘ attribute. This attribute tells bots, “hey this is an ad, don’t follow it.”. Bots that follow the REP (Robots Exclusion Protocol), won’t follow the link. This attribute is included automatically on all ad links that AdPlugg serves.

However, ‘rel=”nofollow”‘ doesn’t stop all bots. Even amongst the largest search engines, there is inconsistent support for the standard. Wikipedia’s Nofollow article reports that while Google and Ask.com fully respect the attribute, Yahoo and Bing both follow the link anyway (but exclude the link from their search rankings).

Other offenders are much less well intentioned. Email harvesters, spambots, malware and bots that scan for security vulnerabilities are unlikely to respect the nofollow attribute.

This leaves it up to the ad servers and ad trackers to identify and filter bot traffic. It’s tricky however, because the links can’t appear broken. If a bot, such as Bingbot, tries to follow the link (even though nofollow is set), the link should work. Otherwise, it’s possible that Bingbot will demote the page (from the Bing search engine rankings) for having broken links.

AdPlugg handles bot traffic by allowing it to pass through but filtering it out of the statistics. This keeps all links working but stops bot traffic from being included in the ad’s impressions and click statistics.

The trick is identifying the bots. AdPlugg now scans for and filters out over 400 known bots. We have systems in place to update our bot list as we become aware of new bots. Similar to the plight of CAPTCHA technology, identifying bot traffic for the purpose of ad stats is a difficult battle. Bots that don’t want to be identified are remarkably good at it.

AdPlugg’s battle against the bots is an ongoing effort and our systems are continually being worked on and improved. We often get asked, “what differences make AdPlugg better than other options such as stand-alone plugins?” Well, bot filtering is a big one.

Ad Reports (BETA) Released!

ad-reportHow much is an ad report worth? Well, pretty much what you paid to run the ad. Reporting lets the advertiser know first and foremost that the ad ran at all. Additionally, the report can show the advertiser useful statistics on how the ad performed each day both in terms of impressions and clicks. AdPlugg has just released the BETA version of a new reporting feature that makes it easy to send ad reports to your advertisers as PDFs (Portable Document Format).

AdPlugg has always tracked impressions and clicks for every ad that it serves. This along with a smattering of other data including the who, what, where, and when of the impression or click event is recorded. We’ve also long had a charting feature where you can graphically view the performance of your ads over time. Now we’ve added a new PDF ad reporting feature that allows you to create a PDF ad report for each of your ads. The report shows a listing of each day that the ad ran along with the number of impressions and clicks that the ad received that day. The image above shows a sample and should give you an idea of what the ad reports look like.

The new ad reports are generated as PDF files which makes them ideal for emailing to your advertisers. This is a good practice that helps you to retain your advertisers and keep them advertising with you. While they can see how much traffic is coming from your site via tools such as Google Analytics, they can’t see how many impressions their ads are getting unless you provide them with reports.

Right now sending the reports to the advertisers is a manual process but we plan to have them send automatically (at the publisher’s option) in an upcoming release.

If you would like to gain access to the new ad reports feature, please contact us.