User-agent: * # this directs to all spiders who access this file Disallow: /dashboard/ Disallow: /accounts/ Disallow: /codes/ Disallow: /pdf/ Disallow: /reports/ Disallow: /images/ Disallow: /admin/