Preventing bizlistBot from Crawling Your Website

At bizlist.ai, we value and prioritize every company's autonomy and privacy wishes. If you'd rather keep our LeaddataBot from crawling your website's data, it's simple to instruct our crawler to steer clear of your domain. This can be done by making a modification to your website's robots.txt file.

Here’s your guide on how to keep bizlistBot at bay:

What is robots.txt?

The robots.txt file is a standard protocol used by websites to communicate with web crawlers and other web robots about which pages on their site should not be processed or scanned. By specifying user-agents and directives, you can guide which sections of your site (or the entirety of it) you'd like to be off-limits to particular bots.

Steps to Block LeaddataBot:

  1. Locate/Create your robots.txt file: Typically, this file is found in the root directory of your website. If one doesn't exist, it's a straightforward process to create a plain text file named robots.txt.
  2. Add directives for LeaddataBot: To specifically block our crawler, append the following lines to your robots.txt file:

User-agent: bizlistBot Disallow: /

These lines tell LeaddataBot to refrain from crawling any section of your website.

  1. For Specific Directories Only: If you wish to block just certain areas of your site, adjust the Disallow directive. For instance, to prevent LeaddataBot from accessing a directory named "private", use:

User-agent: bizlistBot Disallow: /private/

  1. CMS Users: If you're using a Content Management System (CMS), there's a possibility you can control your robots.txt from within the CMS settings. For WordPress users, if you have the Yoast SEO plugin, it provides an easy interface to edit your robots.txt. Here's a step-by-step guide from Yoast on how to edit the robots.txt through Yoast SEO.
  2. Save and Upload: After making the desired changes, save your robots.txt file and make sure it's uploaded to your website's root directory.
  3. Verify: You can confirm your robots.txt is correctly implemented by visiting http://www.yourdomain.com/robots.txt. Replace "yourdomain" with your actual domain name.

While bizlistBot will honor the rules in your file, it's worth noting that not all crawlers are as courteous. Ensure you regularly review and update your robots.txt file to address your site’s evolving requirements.

Should you require further guidance or have questions, please reach out to our support team at [support@bizlist.ai]. We are committed to respecting your choices and ensuring a smooth experience.

cloud-sync lock earth