Dear Support Team,
We are facing an issue with suspicious traffic to the website which
seems to be originating from various msn/bing bots trying to index
various parts/subpages of the website.
I've updated the robots.txt configuration of the Public Pages of the
site, to the following rules:
**
User-Agent: *
Disallow:
User-agent: bingbot
Disallow:
/
User-agent: msnbot
Disallow: /
Sitemap: [$PROTOCOL$]://[$HOST$]:[$PORT$]/sitemap.xml
**
This would disallow accessing the site pages from Agent that contain
'bingbot' or 'msnbot' in the String(s).
Since this doesn't seem to have stopped the bots from crawling the
website, do I need to add anything else to these rules or somehow
add/re-apply anything else?
Kind Regards,
Antonis
