Crawler Policy

1. Introduction

TheyfyBot is the official web crawler used by theyfy.com to read your website content and power your AI assistant with accurate answers. If you're using Theyfy Lite or Theyfy Live, by entering your domain your bot will automatically visit your site and learn from your public pages.

2. Why Does It Matter?

For your AI assistant to answer questions about your website (such as services, prices, and contact information), TheyfyBot needs permission to read your pages. If you don't have a robots.txt file, the bot will freely crawl your site.

3. Allowing TheyfyBot in robots.txt

If your website has a robots.txt file, please include the following rule to allow crawling:

# Example robots.txt entry to allow TheyfyBot
User-agent: TheyfyBot
Allow: /

This ensures your website permits TheyfyBot to crawl and learn from it.

4. Blocking TheyfyBot

If your robots.txt blocks all bots (e.g. Disallow: /), the assistant will not be able to fetch your content and may not provide accurate answers to your visitors.

5. Alternative Options

Allowing TheyfyBot in your robots.txt is only required if you want your assistant to crawl your website directly. If you prefer, you can instead upload documents (PDFs, DOCX, TXT, etc.) into the Knowledge Base tool for training.