I use the WordPress Redirection plugin (https://wordpress.com/plugins/redirection) to manage redirects on our production site.

One of the more annoying things I deal with is random crappy bots that just crawl the site occasionally. Because they crawl so rarely (and inexpertly) they often look for ancient pages, or use ridiculous URL parameters (if anyone knows why they do this can you let me know in the comments?)

Rather than treat these as valid 404s, I think it’s best to ignore them completely. However, matching a user agent string in Redirection is not straight forward. To this end I wrote a Regex to match three of the most annoying culprits: MojeekBot, SeekportBot & Barkrowler.

Maybe it’ll help you too.

Leave a Reply

Your email address will not be published. Required fields are marked *

Your post must include a link to this post.

How does this work?
Write a response on your own website and include a link to this post. Then submit your response’s URL here. Your response may appear here after moderation. To update or remove it, edit or delete your original post and submit the same URL again. Learn more about Webmentions.