Fetch as Google gets Temporarily unreachable error from Algolia calls

I am debugging why googlebot is unable to render our pages using Fetch as Google. The root cause appears to be that it is not getting a response from the Algolia requests which populate key data.

One possibility is that it does not work because algolia.net does not have a robots.txt file. This is based on one of the answers here: google search console - "Temporarily unreachable" when using fetch as Googlebot from Webmasters Tools - Webmasters Stack Exchange

Googlebot is now capable of making POST requests, which it could not before, so I don’t believe that is the cause.

31 PM

Hmm, let’s investigate, I’ve definitely got pages accessible with Googlebot before, for example the Yarn site worked in fetch by Google before.

Now when I try the UI looks different, I don’t get the side-by-side view as I did before…

1 Like

Thanks for looking into this! Here’s a cURL for a request that it gets no response from:
curl 'https://a7zwrybddg-dsn.algolia.net/1/indexes/products_name_asc/query?x-algolia-agent=Algolia%20for%20vanilla%20JavaScript%203.19.2&x-algolia-application-id=A7ZWRYBDDG&x-algolia-api-key=ZWVjNzUzNTY3NGI1NWUwNDM4MDY2NGRmM2M2NTIwOGU3ZGUzODIwZjViYzZjYTU0NDMyYTljZDE2NTI2OTVkOGZhY2V0RmlsdGVycz10ZW5hbnQlM0E1ODUzM2UwZDNiNzdjODAwMTcyY2ExNGUlMkMlMjBzdGF0ZSUzQUFjdGl2ZQ%3D%3D' -H 'Origin: https://nils-plus-test.volusionplatformdev.com' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36' -H 'content-type: application/x-www-form-urlencoded' -H 'accept: application/json' -H 'Referer: https://nils-plus-test.volusionplatformdev.com/c/stickers' -H 'Connection: keep-alive' --data '{"params":"query=&facetFilters=%5B%22state%3AActive%22%2C%5B%22categoryIds%3A59397c58ebd2970b2c55024c%22%5D%5D&hitsPerPage=16&page=0"}' --compressed

gets result for me:

$ curl 'https://a7zwrybddg-dsn.algolia.net/1/indexes/products_name_asc/query?x-algolia-agent=Algolia%20for%20vanilla%20JavaScript%203.19.2&x-algolia-application-id=A7ZWRYBDDG&x-algolia-api-key=ZWVjNzUzNTY3NGI1NWUwNDM4MDY2NGRmM2M2NTIwOGU3ZGUzODIwZjViYzZjYTU0NDMyYTljZDE2NTI2OTVkOGZhY2V0RmlsdGVycz10ZW5hbnQlM0E1ODUzM2UwZDNiNzdjODAwMTcyY2ExNGUlMkMlMjBzdGF0ZSUzQUFjdGl2ZQ%3D%3D' -H 'Origin: https://nils-plus-test.volusionplatformdev.com' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36' -H 'content-type: application/x-www-form-urlencoded' -H 'accept: application/json' -H 'Referer: https://nils-plus-test.volusionplatformdev.com/c/stickers' -H 'Connection: keep-alive' --data '{"params":"query=&facetFilters=%5B%22state%3AActive%22%2C%5B%22categoryIds%3A59397c58ebd2970b2c55024c%22%5D%5D&hitsPerPage=16&page=0"}' --compressed
{"hits":[{"id":"59b6eebbebd29709c8d9bb43","name":"Badge - Digital Download","price":5.5,"msrp":null,"pricing":{"listPrice":3,"salePrice":null},"descriptions":{"short":"","long":"<p>Our Travel & Bowlus Road Chief trailer blog, Instagram Feed, and soon to be YouTube Channel</p>","extended":""},"weight":null,"createdOn":"2017-09-11T20:14:51.72Z","updatedOn":"2017-10-19T01:29:39.3933039Z","state":"Active","categoryIds":["59397c58ebd2970b2c55024c"],"seo":{"title":"The Distance Calling Badge - Digital Download","metaDescription":"Our Travel & Bowlus Road Chief trailer blog, Instagram Feed, and soon to be YouTube Channel","friendlyName":"badge"},"images":[{"imageLink":{"id":"59dbc63aeb049800101b5517","createdOn":"2017-10-09T18:56:16.381Z","updatedOn":"2017-10-09T18:56:23.686Z","fullUri":"https://res.cloudinary.com/materialsandbox/image/upload/v1507575376/58533e0d3b77c800172ca14e/duwud0sp27vhfgfcebzs.png","uriBase":"https://res.cloudinary.com/materialsandbox/image/upload/","imagePath":"v1507575376/58533e0d3b77c800172ca14e/duwud0sp27vhfgfcebzs.png","revision":2,"tags":["product"],"tenant":"58533e0d3b77c800172ca14e"},"title":"","description":"","altText":""}],"revision":13,"sku":"badge","requiresShipping":false,"isTaxable":true,"tenant":"58533e0d3b77c800172ca14e","variantOptions":[],"productVariants":[{"id":"59b6eebbebd29709c8d9bb42","sku":"badge","variants":[],"isShopperVisible":true,"price":5.5,"priceOverride":null,"isInventoryTracked":false,"quantity":0,"imageLinkIds":[],"skuImageLinkIds":[]}],"objectID":"59b6eebbebd29709c8d9bb43","_highlightResult":{"name":{"value":"Badge - Digital Download","matchLevel":"none","matchedWords":[]},"descriptions":{"short":{"value":"","matchLevel":"none","matchedWords":[]},"long":{"value":"<p>Our Travel & Bowlus Road Chief trailer blog, Instagram Feed, and soon to be YouTube Channel</p>","matchLevel":"none","matchedWords":[]},"extended":{"value":"","matchLevel":"none","matchedWords":[]}},"createdOn":{"value":"2017-09-11T20:14:51.72Z","matchLevel":"none","matchedWords":[]},"updatedOn":{"value":"2017-10-19T01:29:39.3933039Z","matchLevel":"none","matchedWords":[]},"state":{"value":"Active","matchLevel":"none","matchedWords":[]},"categoryIds":[{"value":"59397c58ebd2970b2c55024c","matchLevel":"none","matchedWords":[]}],"seo":{"title":{"value":"The Distance Calling Badge - Digital Download","matchLevel":"none","matchedWords":[]},"metaDescription":{"value":"Our Travel & Bowlus Road Chief trailer blog, Instagram Feed, and soon to be YouTube Channel","matchLevel":"none","matchedWords":[]},"friendlyName":{"value":"badge","matchLevel":"none","matchedWords":[]}},"tenant":{"value":"58533e0d3b77c800172ca14e","matchLevel":"none","matchedWords":[]},"productVariants":[{"id":{"value":"59b6eebbebd29709c8d9bb42","matchLevel":"none","matchedWords":[]},"sku":{"value":"badge","matchLevel":"none","matchedWords":[]}}]}},{"id":"59397ce7ebd2970b2c550250","name":"Blue Sun Sticker","price":1.23,"msrp":null,"descriptions":{"short":"","long":"<p>This is a description.\n</p>","extended":""},"weight":0.01,"createdOn":"2017-06-08T16:35:51.779Z","updatedOn":"2017-08-15T17:40:24.0867695Z","state":"Active","categoryIds":["59397c58ebd2970b2c55024c"],"seo":{"title":"Blue Sun Sticker","metaDescription":"SEO Meta Description","friendlyName":"blue-sun-sticker"},"images":[{"imageLink":{"id":"59397c16ebd2970b2c55024b","createdOn":"2017-06-08T16:32:22.782Z","updatedOn":"2017-06-08T16:32:22.782Z","fullUri":"https://res.cloudinary.com/materialsandbox/image/upload/v1496939557/58533e0d3b77c800172ca14e/t8yrnbz1ak04pvw0m8tu.jpg","uriBase":"https://res.cloudinary.com/materialsandbox/image/upload/","imagePath":"v1496939557/58533e0d3b77c800172ca14e/t8yrnbz1ak04pvw0m8tu.jpg","revision":1,"tags":[],"tenant":"58533e0d3b77c800172ca14e"},"title":"","description":"","altText":""},{"imageLink":{"id":"59397c6febd2970b2c55024d","createdOn":"2017-06-08T16:33:51.625Z","updatedOn":"2017-06-08T16:33:51.625Z","fullUri":"https://res.cloudinary.com/materialsandbox/image/upload/v1496939646/58533e0d3b77c800172ca14e/zqdxns3myz1hptcu4obk.jpg","uriBase":"https://res.cloudinary.com/materialsandbox/image/upload/","imagePath":"v1496939646/58533e0d3b77c800172ca14e/zqdxns3myz1hptcu4obk.jpg","revision":1,"tags":[],"tenant":"58533e0d3b77c800172ca14e"},"title":"","description":"","altText":""}],"revision":35,"sku":"blue-sun-sticker","tenant":"58533e0d3b77c800172ca14e","variantOptions":[{"id":null,"name":"Color","options":["blue","orange"],"priceDifferences":[0,3],"order":1,"imageLinkIds":[[],[]]},{"id":null,"name":"Size","options":["2\"","4\""],"priceDifferences":[0,0],"order":2,"imageLinkIds":[[],[]]}],"productVariants":[{"id":"598b770debd2970684ff59cb","sku":"blue-sun-sticker-blue-2-","variants":["blue","2\""],"isShopperVisible":true,"price":1.23,"priceOverride":null,"isInventoryTracked":false,"quantity":0,"imageLinkIds":[],"skuImageLinkIds":["59397c16ebd2970b2c55024b"]},{"id":"598b770debd2970684ff59cc","sku":"blue-sun-sticker-blue-4-","variants":["blue","4\""],"isShopperVisible":true,"price":1.23,"priceOverride":null,"isInventoryTracked":false,"quantity":0,"imageLinkIds":[],"skuImageLinkIds":["59397c16ebd2970b2c55024b"]},{"id":"598b770debd2970684ff59cd","sku":"blue-sun-sticker-orange-2-","variants":["orange","2\""],"isShopperVisible":true,"price":4.23,"priceOverride":null,"isInventoryTracked":false,"quantity":0,"imageLinkIds":[],"skuImageLinkIds":["59397c6febd2970b2c55024d"]},{"id":"598b770debd2970684ff59ce","sku":"blue-sun-sticker-orange-4-","variants":["orange","4\""],"isShopperVisible":true,"price":4.23,"priceOverride":null,"isInventoryTracked":false,"quantity":0,"imageLinkIds":[],"skuImageLinkIds":["59397c6febd2970b2c55024d"]}],"objectID":"59397ce7ebd2970b2c550250","_highlightResult":{"name":{"value":"Blue Sun Sticker","matchLevel":"none","matchedWords":[]},"descriptions":{"short":{"value":"","matchLevel":"none","matchedWords":[]},"long":{"value":"<p>This is a description.\n</p>","matchLevel":"none","matchedWords":[]},"extended":{"value":"","matchLevel":"none","matchedWords":[]}},"createdOn":{"value":"2017-06-08T16:35:51.779Z","matchLevel":"none","matchedWords":[]},"updatedOn":{"value":"2017-08-15T17:40:24.0867695Z","matchLevel":"none","matchedWords":[]},"state":{"value":"Active","matchLevel":"none","matchedWords":[]},"categoryIds":[{"value":"59397c58ebd2970b2c55024c","matchLevel":"none","matchedWords":[]}],"seo":{"title":{"value":"Blue Sun Sticker","matchLevel":"none","matchedWords":[]},"metaDescription":{"value":"SEO Meta Description","matchLevel":"none","matchedWords":[]},"friendlyName":{"value":"blue-sun-sticker","matchLevel":"none","matchedWords":[]}},"tenant":{"value":"58533e0d3b77c800172ca14e","matchLevel":"none","matchedWords":[]},"productVariants":[{"id":{"value":"598b770debd2970684ff59cb","matchLevel":"none","matchedWords":[]},"sku":{"value":"blue-sun-sticker-blue-2-","matchLevel":"none","matchedWords":[]},"variants":[{"value":"blue","matchLevel":"none","matchedWords":[]},{"value":"2\"","matchLevel":"none","matchedWords":[]}],"skuImageLinkIds":[{"value":"59397c16ebd2970b2c55024b","matchLevel":"none","matchedWords":[]}]},{"id":{"value":"598b770debd2970684ff59cc","matchLevel":"none","matchedWords":[]},"sku":{"value":"blue-sun-sticker-blue-4-","matchLevel":"none","matchedWords":[]},"variants":[{"value":"blue","matchLevel":"none","matchedWords":[]},{"value":"4\"","matchLevel":"none","matchedWords":[]}],"skuImageLinkIds":[{"value":"59397c16ebd2970b2c55024b","matchLevel":"none","matchedWords":[]}]},{"id":{"value":"598b770debd2970684ff59cd","matchLevel":"none","matchedWords":[]},"sku":{"value":"blue-sun-sticker-orange-2-","matchLevel":"none","matchedWords":[]},"variants":[{"value":"orange","matchLevel":"none","matchedWords":[]},{"value":"2\"","matchLevel":"none","matchedWords":[]}],"skuImageLinkIds":[{"value":"59397c6febd2970b2c55024d","matchLevel":"none","matchedWords":[]}]},{"id":{"value":"598b770debd2970684ff59ce","matchLevel":"none","matchedWords":[]},"sku":{"value":"blue-sun-sticker-orange-4-","matchLevel":"none","matchedWords":[]},"variants":[{"value":"orange","matchLevel":"none","matchedWords":[]},{"value":"4\"","matchLevel":"none","matchedWords":[]}],"skuImageLinkIds":[{"value":"59397c6febd2970b2c55024d","matchLevel":"none","matchedWords":[]}]}]}}],"nbHits":2,"page":0,"nbPages":1,"hitsPerPage":16,"processingTimeMS":1,"exhaustiveNbHits":true,"query":"","params":"query=&facetFilters=%5B%22state%3AActive%22%2C%5B%22categoryIds%3A59397c58ebd2970b2c55024c%22%5D%5D&hitsPerPage=16&page=0&facetFilters=tenant%3A58533e0d3b77c800172ca14e%2C%20state%3AActive"}

Yes it does, and it works on our site to. The issue is it doesn’t work for googlebot. The simplest explanation I’ve seen, but one that I can’t experiment with on my end, is that it’s because the domain doesn’t have a robots.txt file:

I don’t think that’s the issue, and we should explicitly not have a robots.txt, since people have sensitive data on our servers. It’s not an easy problem to solve.