Website status checker
Check if crawlers can reach a website: robots.txt on every version of the domain, the homepage, which bots are allowed, noindex, speed and the sitemap. Every result below is a real answer from the site — nothing is guessed.
What our bot does
Users of LINKBOOSTER can let IndexPulseBot crawl their own website: it reads robots.txt and every sitemap (post, page and category sitemaps too), visits the pages, follows internal links to find posts that are not in a sitemap, and sends the URLs for indexing checks. Each crawl collects at most as many URLs as the user's plan allows. Every step is shown live in the user's account.
How it behaves
- Identifies itself as
Mozilla/5.0 (compatible; IndexPulseBot/1.0; +https://seobacklinkoworld.online/bot). - Respects
robots.txt(user-agent tokenIndexPulseBot, falling back to*). - Waits at least 1 second(s) between requests to the same host.
- Downloads at most 2 MB per page, never executes scripts and never submits forms.
- Skips admin, cart, login and feed URLs, images and files.
- Never impersonates Googlebot or any other crawler.
Blocking the bot
User-agent: IndexPulseBot
Disallow: /
Changes to robots.txt are picked up within one hour. If you believe the bot is misbehaving, contact us with the affected host and timestamps.
Crawl & index your site
Free accounts can crawl too — the bot collects as many URLs as your plan allows and sends them for indexing. You watch it work live.
Create a free account