From what I’ve read and seen, CloudFront does not consistently identify itself in requests. But you can get around this problem by overriding robots.txt at the CloudFront distribution.
1) Create a new S3 bucket that only contains one file: robots.txt. That will be the robots.txt for your CloudFront domain.
2) Go to your distribution settings in the AWS Console and click Create Origin. Add the bucket.
3) Go to Behaviors and click Create Behavior: Path Pattern: robots.txt Origin: (your new bucket).
4) Set the robots.txt behavior at a higher precedence (lower number).
5) Go to invalidations and invalidate /robots.txt.
Now domainname.cloudfront.net/robots.txt will be served from the bucket and everything else will be served from your domain. You can choose to allow/disallow crawling at either level independently.
Another domain/subdomain will also work in place of a bucket, but why go to the trouble.
In this article, we are how to upload CSV file from a system and import in the database using Laravel. There are countless bundles accessible in Laravel to import dominant documents in the data set
Read More
Everyone enjoys travel and travel becomes a hobby to the peoples over all the world. Travelers are increased repeatedly. While the initial thoughts of the journey arise in the mind, Peoples will sourc
Read More
The success of your business might be determined by the platform you choose in the constantly changing world of e-commerce. A popular option for companies of all sizes is Magento, a powerful open-sour
Read More