Add custom robots.txt on CDN SUN

Create robots-cdn in CDN Sun (https://cdnsun.com/)

1. Create robots-cdn.txt file on the root directory

Add the following code

User-agent: *
Allow: /wp-content/*
Disallow: /

2. On your site .htaccess add the following

RewriteEngine On
SetEnvIf X-Resource "#########" IS_CDNSUN_REQUEST="yes"
RewriteCond %{ENV:IS_CDNSUN_REQUEST} "yes"
RewriteRule "robots.txt" "robots-cdn.txt" [L]
### = resource can be found at CDN Sun dashboard

3. on CDN sun service setting, disable block crawlers
https://cdnsun.com/cdn/settings
Block Crawlers: disable

References
https://cdnsun.com/knowledgebase/cdn-static/setting-a-block-crawlers-avoid-duplicate-content-seo

Leave a Reply

Your email address will not be published. Required fields are marked *