What is SEMrush Bot? Should You Block it? And How (Tutorial)?
Disclosure (full version- https://nikolaroza.com/affiliate-disclosure/): Some of the links you’ll encounter are affiliate links. If you click and buy something, I may get a commission. Thank you!
Jump here: what is SEMrush bot| should you block the SEMrush block| how to block the SEMrush bot| SEMrush bot FAQ| conclusion| post scriptum|
In this article, I am going to talk about the SEMrush bot, whether you need to block it or not, and how to block it from your online property.
Let’s get started!
What is SEMrush Bot?
Definition: SemrushBot is a search bot software that Semrush sends to the web to discover and collect new and updated web content. Data gathered by SemrushBot is served to SEMrush paid and free users alike. SEMrush has one of the fastest bots on the web, which makes the data SEMrush shows up to date and as accurate as possible.
How Does the SEMrush Bot Work?
SEMrush bot uses links to find new websites, analyze them, and collect data from each site. This is similar to what other web crawler bots do (Ahrefs-bot, Google-bot and Serpstatstat-bot)
Collected data is used for :
- Brand monitoring;
- Competitor research;
- Link building;
- Keyword research;
- SEO content writing;
- Backlink research;
- SEO auditing, On-page;
- etc.
For further information, take a look at what SEMrush have to say about their bot and how it crawls the web.
“SemrushBot’s crawl process starts with a list of webpage URLs. When SemrushBot visits these URLs, it saves hyperlinks from the page for further crawling. This list, also known as the “crawl frontier”, is repeatedly visited according to a set of Semrush policies to effectively map a site for updates: content changes, new pages, and dead links.”
How To Identify SEMrush Bot?
To identify SEMrush bot first you need to get the SEMrush spider IP address from your website’s server logs.
Next, paste the IP address into this free reverse DNS lookup tool.
The host domain name should be from Semrush.com and if it is not, then you might have a fake bot.
If so, you need to block that foreign bot from your site quickly before it does damage to your website (incessant and aggressive crawling can bring your whole website down, especially if you’re on some cheap shared host).
Should You Block the SEMrush Bot?
It’s very easy to block the SEMrush bot.
But before blocking, you should know when to block SEMrush bot, and when to leave it alone.
Read on below to find out.
You Shouldn’t Block SEMrush Bot If You Use SEMrush Tools
Marketers and SEOs use SEMrush data to optimize their sites for maximum Google and search engine traffic.
With an offer of 50+ marketing tools, you can use SEMrush to get insights on pretty much any area of SEO and it’s an invaluable tool for marketers and bloggers.
If you’re currently using SEMrush then don’t block their bot as you won’t be able to get valuable data about your website.
And if you intend to use SEMrush then also don’t block their bot as again, you will lose access to valuable data from your website.
And here are the reasons why you should block the SEMrush bot.
#1- You’re Seeing An Unexpected Increase In Bandwidth
Have a slow website?
Nobody wants that!
If the SEMrush bot starts visiting your website too often it will consume a lot of bandwidth. And because of that, your website will start to load slowly for other visitors.
That’s a valid reason to block the SEMrush bot.
#2- Block SEMrush to Prevent Them From Indexing Your Links
As SEMrush bot is crawling the web, it comes up on pages that host your links and counts them in their backlink index.
This means those links are now visible to all premium SEMrush users, including your direct competitors.
If you want to prevent it, then block the SEMrush bot.
#3- You’re Not Using SEMrush at all
This goes without saying, though here I am saying it anyway:)
If you’re not using SEMrush at all, and you know you won’t be using it, then block it from your website for good. At least you’ll end up with a faster site.
Here's Why You Should Block the SEMrush Bot #seo Click To TweetHow to Block the SEMrush Bot?
There are two simple ways how to block the SEMrush bot. Read them both and pick the option better for you.
Method 1: Block SEMrush bot by updating robots.txt
Note: your website’s robots.txt file serves up instructions to all bots that want to come and crawl your site. You can set up generic rules that every bot should follow, or you can set up specific rules for one particular type of bot.
In this case, we want to block the SEMrush bot while not disturbing other good bots.
This is the easiest and quickest way to block the SEMrush bot by updating your robots.txt. You just need to copy this code to your robots.txt file:
User-agent: SemrushBot
Disallow: /
Note: you can also use robots.txt to add further instructions to SEMrush bot, without blocking it completely.
To tell SEMrush to go easy on your site simply add:
User-agent: SemrushBot
Crawl-delay: 60
To Block SEMrush’s backlink audit tool, but allow other tools simply add:
User-agent: SemrushBot-BA
Disallow: /
Block SemrushBot from crawling your website for On-Page SEO Checker tool:
User-agent: SemrushBot-SI
Disallow: /
Block SemrushBot for SWA tool URL checking:
User-agent: SemrushBot-SWA
Disallow: /
Block SemrushBot from crawling your website for Post Tracking and Content Analyzer apps/tools:
User-agent: SemrushBot-CT
Disallow: /
Block SplitSignalBot from crawling your website for the SplitSignal app:
User-agent: SplitSignalBot
Disallow: /
Block SemrushBot-COUB from crawling your website for the Content Outline Builder app:
User-agent: SemrushBot-COUB
Disallow: /
Method 2: Block SEMrush bot Using The .htaccess File
The .htaccess file is a security guard who’s watching over your website making sure no intruder gets through.
And .htaccess file can see who is the bot trying to crawl your site and what they are trying to do on your website.
You can copy this code into your website .htaccess file:
RewriteCond %{HTTP_USER_AGENT} \
SemrushBot |\
SEMrush Bot FAQ- Your Questions Answered!
My mini SEMrush bot FAQ section below has the answers you seek about SEMrush bot, how to block it, and whether you even should.
#1- Can I Block SEMrush Bot via IP Address?
You cannot block SEMrush bot via IP address.
According to SEMrush:
“Please do not try to block SEMrushBot via IP in .htaccess as we do not use any consecutive IP blocks”.
#2- What is SEMrush Bot IP Address?
SEMrush bot IP addresses are:
- 85.208.98.50;
- 18.197.42.174;
- 35.177.199.105;
- 13.53.131.112
SEMrush bot is using a standard 80 HTTP port to connect.
#3- Is SEMrush Crawler a Good Bot? Or is it Bad?
SEMrush crawler is a good bot that will obey the instructions you put in your robots.txt file.
add part how it sometimes takes time to react.
#4- Can I Block SEMrush Bot via .htaccess?
You can block the SEMrush bot via .htaccess.
Please check my guide here.
#5- What Should I Do if I Notice SEMrush Bot Attack?
If you notice that SEMrush bot is crawling and hitting your site way too much, the first thing you need to do is add instructions to your robots.txt file telling the SEMrush bot to slow down and back off a bit.
Then wait 2 weeks because that’s how long it takes the SEMrush bot to notice and acknowledge the robots.txt instructions.
If this doesn’t work then temporarily block the bot, either via robots.txt file or via .htaccess.
#6- Does SEMrush Have a Crawler?
SEMrush does have a crawler. “SEMrush crawler” is just another name for “SEMrush bot”.
SEMrush uses its bot to crawl sites and deliver their proprietary data to paying customers and free SEMrush users alike.
#7- Can SEMrush Try to Crawl a Page Which No Longer Exists?
Once SEMrush finds a 404 page, it knows not to crawl it further and waste resources on it. However, occasionally the bot will come back to recrawl the page, especially if there are internal or external links pointing to it.
#8- Where Can I Ask for Help Regarding SEMrush Bot?
You can ask for help with SEMrush bot problems via this email bot@semrush.com.
#9- Do Some Web Hosting Providers Block The SEMrush Bot?
Most quality hosts will not block good bots from crawling your site. However, occasionally it happens.
For example, according to people from this Reddit thread, Bluehost, HostGator and other web hosting services owned by EIG are actively blocking SEMrush’s and most other SEO tool bots.
They say it’s because the SEMrush bot attacks the server and wastes bandwidth like there’s no tomorrow.
#10- What Types of Bots use the Major SEMrush Alternatives?
There are plenty of SEMrush alternatives and competitors on the market. You can read my guide to the best SEMrush alternatives to learn more about it.
But in general, SEO and SEM tools either use their proprietary bots or source their data from elsewhere.
- SE Ranking– SERanking uses the SE Ranking bot (SEMrush vs SE Ranking guide)
- Ahrefs– Ahrefs uses the Ahrefs bot (SEMrush vs Ahrefs guide)
- Yoast– Yoast doesn’t have their own bot (SEMrush vs Yoast SEO guide)
- SpyFu– they operate their Spyfu bot (SEMrush vs Spyfu guide)
- Serpstat– Serpstat operate their Serpstat bot (SEMrush vs Serpstat guide)
- Moz– Moz uses Roger bot (SEMrush vs Moz guide)
- Screaming Frog– they use the Screaming Frog bot (SEMrush vs Screaming Frog guide)
- Majestic SEO– MAjestic SEO uses the MJ12Bot (SEMrush vs Majestic SEO guide)
- Answer the Public;
- Long Tail Pro– Long Tail Pro sources their data from other providers (SEMrush vs Long Tail Pro guide)
- BuzzSumo– Buzzsumo uses other providers data. (SEMrush vs BuzzSumo guide)
- UberSuggest– Ubersuggest uses other providers data (SEMrush vs UberSuggest guide)
- Mangools– Mangools source their data from other providers (SEMrush vs Mangools guide);
- Google Analytics– GA works via Javascript code and not via crawling bots (SEMrush vs Google Analytics guide)
- Google Search Console– Google Search Console uses Google-bot to crawl the web (SEMrush vs Google Search Console guide)
SEMrush Bot Guide for 2023 (Conclusion)
In this article, you learned what SEMrush bot is, how to block it, and whether you should block it from your website.
Let me know what you think in the comment section below.
Post Scriptum
Hey, you seem to be interested in SEMrush.
If you haven’t gotten this all-in-one SEO app before, then what are you waiting for?
Please read one of my SEMrush guides below to learn more about it.
Nikola Roza
Nikola Roza is an affiliate marketer and blogger behind Nikola Roza- SEO for the Poor and Determined. He writes for bloggers who don't have huge marketing budgets and who still want to carve out a niche online and a better life for themselves. Learn about Nikola here, or read his blog posts and guides here.