You are viewing a single comment's thread from:

RE: Google's Bot Imposes a Large Impact on the Performance of SteemYY.com

in #witness-category • 2 years ago

I had the same issue!!! google bot and also Bing bot impacted my site.
I changed the robots.txt and limited the pages they can crawl and it seems to be working... not 100% because some bots ignore robots.txt

2 years ago in #witness-category by
$0.00
    Reply 3
    Sort:  
  • Trending
    • [-]
     2 years ago 

    For me, I don't want to ban entirely the Bot, is there a way to rate limit bots in robots.txt?

    $0.00
      Reply
      [-]
       2 years ago 

      I am not a robots.txt expert... but I think so...
      at least you tell them which bots are allowed what pages and others are delayed crawling... or something like that

      $0.00
        Reply