Free Tool
Robots.txt Generator
Generate a properly formatted robots.txt file for your website. Control what search engines can and cannot crawl.
Settings
Your robots.txt
User-agent: * Allow: / Disallow: /admin/ Disallow: /wp-admin/ Disallow: /api/
What Is a Robots.txt File?
A robots.txt file tells search engine crawlers which pages on your site they can and cannot access. It lives at the root of your domain (e.g. example.com/robots.txt) and is one of the first files Google checks when crawling your site. Without one, Google may waste crawl budget on pages that do not matter — or worse, index admin pages and API endpoints.
How to Add Robots.txt to Your Website
Copy the generated file above and save it as robots.txt in your website root directory. For WordPress, upload it via FTP to your public_html folder. For Next.js or Vercel sites, add it to your public/ folder. Always include your sitemap URL — it helps Google discover all your pages faster.