Robots.txt is a text file at your website root that tells search engine crawlers which pages to access and which to skip. It follows the Robots Exclusion Protocol.

No, but recommended. Without it, crawlers will attempt to index everything. Having one gives you control over what appears in search results.

No! Disallow is a polite request, not security. Well-behaved crawlers honor it, but malicious bots ignore it. Never rely on it for security.

It blocks the entire site from that crawler. An empty Disallow (or none) means allow everything. Be careful with this!

Usually yes, to keep admin URLs out of search results. But remember: this doesn't secure your admin – use proper authentication.

It requests crawlers wait X seconds between requests. Googlebot ignores this (use Search Console instead), but Bingbot and others respect it.

* matches any sequence of characters. $ means end of URL. Example: '/*.pdf$' disallows all PDF files regardless of folder.

In your website's root directory, accessible at yourdomain.com/robots.txt. It must be at the root – subdirectory robots.txt files are ignored.

Use Google Search Console's robots.txt Tester. It shows if specific URLs are blocked and highlights syntax errors.

Yes! Adding 'Sitemap: URL' helps crawlers discover all your pages. Use the full URL including https://.