I use the WordPress Redirection plugin (https://wordpress.com/plugins/redirection) to manage redirects on our production site.

One of the more annoying things I deal with is random crappy bots that just crawl the site occasionally. Because they crawl so rarely (and inexpertly) they often look for ancient pages, or use ridiculous URL parameters (if anyone knows why they do this can you let me know in the comments?)

Rather than treat these as valid 404s, I think it’s best to ignore them completely. However, matching a user agent string in Redirection is not straight forward. To this end I wrote a Regex to match three of the most annoying culprits: MojeekBot, SeekportBot & Barkrowler.

Maybe it’ll help you too.

Leave a Reply

Your email address will not be published. Required fields are marked *

To respond on your own website, enter the URL of your response which should contain a link to this post's permalink URL. Your response will then appear (possibly after moderation) on this page. Want to update or remove your response? Update or delete your post and re-enter your post's URL again. (Find out more about Webmentions.)