# Don't allow web crawlers to index Craft User-agent: * Disallow: /craft/ Disallow: /clearcache.php Disallow: /machform/ Disallow: /demo/ Disallow: /assets/PDFs/