User-agent: Scrapy Allow: / User-agent: * Crawl-delay: 2 Sitemap: https://www.tne.co.uk/sitemap_index.xml Sitemap: https://www.tne.co.uk/post-sitemap.xml Sitemap: https://www.tne.co.uk/page-sitemap.xml