Google webmaster temporarily unreachable

From what I’ve read and seen, CloudFront does not consistently identify itself in requests. But you can get around this problem by overriding robots.txt at the CloudFront distribution.

1) Create a new S3 bucket that only contains one file: robots.txt. That will be the robots.txt for your CloudFront domain.

2) Go to your distribution settings in the AWS Console and click Create Origin. Add the bucket.

3) Go to Behaviors and click Create Behavior: Path Pattern: robots.txt Origin: (your new bucket).

4) Set the robots.txt behavior at a higher precedence (lower number).

5) Go to invalidations and invalidate /robots.txt.

Now domainname.cloudfront.net/robots.txt will be served from the bucket and everything else will be served from your domain. You can choose to allow/disallow crawling at either level independently.

Another domain/subdomain will also work in place of a bucket, but why go to the trouble.

Tags

  • Google webmaster fetch as google temporarily unreachable

  • Google Webmaster tool fetch error

Related Posts

Building a Craigslist-Inspired Website on WordPress Without Coding

Building a Craigslist-Inspired Website on WordPress Without Coding

In the huge world of online marketplaces, developing a dynamic platform that draws inspiration from Craigslist's success can have revolutionary effects. Without having to dive into complex coding,

Read More
How to Create PDF file from webpage Using PHP and mpdf

How to Create PDF file from webpage Using PHP and mpdf

In today’s digital world, generating dynamic PDF documents is a common need—whether for invoices, reports, eBooks, or download-ready web content. One of the easiest ways to convert a webpa

Read More
C

Convert base64 Data to image file and write To folder in PHP

In this post, I will be able to allow you to skills to handle the pictures encoded with Base64 and convert the pictures to a folder. While working with API for app, you'll notice that they'

Read More