Fetch as Google gets Temporarily unreachable error from Algolia calls

I am debugging why googlebot is unable to render our pages using Fetch as Google. The root cause appears to be that it is not getting a response from the Algolia requests which populate key data.

One possibility is that it does not work because algolia.net does not have a robots.txt file. This is based on one of the answers here: https://webmasters.stackexchange.com/a/79318

Googlebot is now capable of making POST requests, which it could not before, so I don’t believe that is the cause.

Hmm, let’s investigate, I’ve definitely got pages accessible with Googlebot before, for example the Yarn site worked in fetch by Google before.

Now when I try the UI looks different, I don’t get the side-by-side view as I did before…

1 Like

Thanks for looking into this! Here’s a cURL for a request that it gets no response from:
curl 'https://a7zwrybddg-dsn.algolia.net/1/indexes/products_name_asc/query?x-algolia-agent=Algolia%20for%20vanilla%20JavaScript%203.19.2&x-algolia-application-id=A7ZWRYBDDG&x-algolia-api-key=ZWVjNzUzNTY3NGI1NWUwNDM4MDY2NGRmM2M2NTIwOGU3ZGUzODIwZjViYzZjYTU0NDMyYTljZDE2NTI2OTVkOGZhY2V0RmlsdGVycz10ZW5hbnQlM0E1ODUzM2UwZDNiNzdjODAwMTcyY2ExNGUlMkMlMjBzdGF0ZSUzQUFjdGl2ZQ%3D%3D' -H 'Origin: https://nils-plus-test.volusionplatformdev.com' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36' -H 'content-type: application/x-www-form-urlencoded' -H 'accept: application/json' -H 'Referer: https://nils-plus-test.volusionplatformdev.com/c/stickers' -H 'Connection: keep-alive' --data '{"params":"query=&facetFilters=%5B%22state%3AActive%22%2C%5B%22categoryIds%3A59397c58ebd2970b2c55024c%22%5D%5D&hitsPerPage=16&page=0"}' --compressed

gets result for me:

$ curl 'https://a7zwrybddg-dsn.algolia.net/1/indexes/products_name_asc/query?x-algolia-agent=Algolia%20for%20vanilla%20JavaScript%203.19.2&x-algolia-application-id=A7ZWRYBDDG&x-algolia-api-key=ZWVjNzUzNTY3NGI1NWUwNDM4MDY2NGRmM2M2NTIwOGU3ZGUzODIwZjViYzZjYTU0NDMyYTljZDE2NTI2OTVkOGZhY2V0RmlsdGVycz10ZW5hbnQlM0E1ODUzM2UwZDNiNzdjODAwMTcyY2ExNGUlMkMlMjBzdGF0ZSUzQUFjdGl2ZQ%3D%3D' -H 'Origin: https://nils-plus-test.volusionplatformdev.com' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36' -H 'content-type: application/x-www-form-urlencoded' -H 'accept: application/json' -H 'Referer: https://nils-plus-test.volusionplatformdev.com/c/stickers' -H 'Connection: keep-alive' --data '{"params":"query=&facetFilters=%5B%22state%3AActive%22%2C%5B%22categoryIds%3A59397c58ebd2970b2c55024c%22%5D%5D&hitsPerPage=16&page=0"}' --compressed
{"hits":[{"id":"59b6eebbebd29709c8d9bb43","name":"Badge - Digital Download","price":5.5,"msrp":null,"pricing":{"listPrice":3,"salePrice":null},"descriptions":{"short":"","long":"<p>Our Travel & Bowlus Road Chief trailer blog, Instagram Feed, and soon to be YouTube Channel</p>","extended":""},"weight":null,"createdOn":"2017-09-11T20:14:51.72Z","updatedOn":"2017-10-19T01:29:39.3933039Z","state":"Active","categoryIds":["59397c58ebd2970b2c55024c"],"seo":{"title":"The Distance Calling Badge - Digital Download","metaDescription":"Our Travel & Bowlus Road Chief trailer blog, Instagram Feed, and soon to be YouTube Channel","friendlyName":"badge"},"images":[{"imageLink":{"id":"59dbc63aeb049800101b5517","createdOn":"2017-10-09T18:56:16.381Z","updatedOn":"2017-10-09T18:56:23.686Z","fullUri":"https://res.cloudinary.com/materialsandbox/image/upload/v1507575376/58533e0d3b77c800172ca14e/duwud0sp27vhfgfcebzs.png","uriBase":"https://res.cloudinary.com/materialsandbox/image/upload/","imagePath":"v1507575376/58533e0d3b77c800172ca14e/duwud0sp27vhfgfcebzs.png","revision":2,"tags":["product"],"tenant":"58533e0d3b77c800172ca14e"},"title":"","description":"","altText":""}],"revision":13,"sku":"badge","requiresShipping":false,"isTaxable":true,"tenant":"58533e0d3b77c800172ca14e","variantOptions":[],"productVariants":[{"id":"59b6eebbebd29709c8d9bb42","sku":"badge","variants":[],"isShopperVisible":true,"price":5.5,"priceOverride":null,"isInventoryTracked":false,"quantity":0,"imageLinkIds":[],"skuImageLinkIds":[]}],"objectID":"59b6eebbebd29709c8d9bb43","_highlightResult":{"name":{"value":"Badge - Digital Download","matchLevel":"none","matchedWords":[]},"descriptions":{"short":{"value":"","matchLevel":"none","matchedWords":[]},"long":{"value":"<p>Our Travel & Bowlus Road Chief trailer blog, Instagram Feed, and soon to be YouTube Channel</p>","matchLevel":"none","matchedWords":[]},"extended":{"value":"","matchLevel":"none","matchedWords":[]}},"createdOn":{"value":"2017-09-11T20:14:51.72Z","matchLevel":"none","matchedWords":[]},"updatedOn":{"value":"2017-10-19T01:29:39.3933039Z","matchLevel":"none","matchedWords":[]},"state":{"value":"Active","matchLevel":"none","matchedWords":[]},"categoryIds":[{"value":"59397c58ebd2970b2c55024c","matchLevel":"none","matchedWords":[]}],"seo":{"title":{"value":"The Distance Calling Badge - Digital Download","matchLevel":"none","matchedWords":[]},"metaDescription":{"value":"Our Travel & Bowlus Road Chief trailer blog, Instagram Feed, and soon to be YouTube Channel","matchLevel":"none","matchedWords":[]},"friendlyName":{"value":"badge","matchLevel":"none","matchedWords":[]}},"tenant":{"value":"58533e0d3b77c800172ca14e","matchLevel":"none","matchedWords":[]},"productVariants":[{"id":{"value":"59b6eebbebd29709c8d9bb42","matchLevel":"none","matchedWords":[]},"sku":{"value":"badge","matchLevel":"none","matchedWords":[]}}]}},{"id":"59397ce7ebd2970b2c550250","name":"Blue Sun Sticker","price":1.23,"msrp":null,"descriptions":{"short":"","long":"<p>This is a description.\n</p>","extended":""},"weight":0.01,"createdOn":"2017-06-08T16:35:51.779Z","updatedOn":"2017-08-15T17:40:24.0867695Z","state":"Active","categoryIds":["59397c58ebd2970b2c55024c"],"seo":{"title":"Blue Sun Sticker","metaDescription":"SEO Meta Description","friendlyName":"blue-sun-sticker"},"images":[{"imageLink":{"id":"59397c16ebd2970b2c55024b","createdOn":"2017-06-08T16:32:22.782Z","updatedOn":"2017-06-08T16:32:22.782Z","fullUri":"https://res.cloudinary.com/materialsandbox/image/upload/v1496939557/58533e0d3b77c800172ca14e/t8yrnbz1ak04pvw0m8tu.jpg","uriBase":"https://res.cloudinary.com/materialsandbox/image/upload/","imagePath":"v1496939557/58533e0d3b77c800172ca14e/t8yrnbz1ak04pvw0m8tu.jpg","revision":1,"tags":[],"tenant":"58533e0d3b77c800172ca14e"},"title":"","description":"","altText":""},{"imageLink":{"id":"59397c6febd2970b2c55024d","createdOn":"2017-06-08T16:33:51.625Z","updatedOn":"2017-06-08T16:33:51.625Z","fullUri":"https://res.cloudinary.com/materialsandbox/image/upload/v1496939646/58533e0d3b77c800172ca14e/zqdxns3myz1hptcu4obk.jpg","uriBase":"https://res.cloudinary.com/materialsandbox/image/upload/","imagePath":"v1496939646/58533e0d3b77c800172ca14e/zqdxns3myz1hptcu4obk.jpg","revision":1,"tags":[],"tenant":"58533e0d3b77c800172ca14e"},"title":"","description":"","altText":""}],"revision":35,"sku":"blue-sun-sticker","tenant":"58533e0d3b77c800172ca14e","variantOptions":[{"id":null,"name":"Color","options":["blue","orange"],"priceDifferences":[0,3],"order":1,"imageLinkIds":[[],[]]},{"id":null,"name":"Size","options":["2\"","4\""],"priceDifferences":[0,0],"order":2,"imageLinkIds":[[],[]]}],"productVariants":[{"id":"598b770debd2970684ff59cb","sku":"blue-sun-sticker-blue-2-","variants":["blue","2\""],"isShopperVisible":true,"price":1.23,"priceOverride":null,"isInventoryTracked":false,"quantity":0,"imageLinkIds":[],"skuImageLinkIds":["59397c16ebd2970b2c55024b"]},{"id":"598b770debd2970684ff59cc","sku":"blue-sun-sticker-blue-4-","variants":["blue","4\""],"isShopperVisible":true,"price":1.23,"priceOverride":null,"isInventoryTracked":false,"quantity":0,"imageLinkIds":[],"skuImageLinkIds":["59397c16ebd2970b2c55024b"]},{"id":"598b770debd2970684ff59cd","sku":"blue-sun-sticker-orange-2-","variants":["orange","2\""],"isShopperVisible":true,"price":4.23,"priceOverride":null,"isInventoryTracked":false,"quantity":0,"imageLinkIds":[],"skuImageLinkIds":["59397c6febd2970b2c55024d"]},{"id":"598b770debd2970684ff59ce","sku":"blue-sun-sticker-orange-4-","variants":["orange","4\""],"isShopperVisible":true,"price":4.23,"priceOverride":null,"isInventoryTracked":false,"quantity":0,"imageLinkIds":[],"skuImageLinkIds":["59397c6febd2970b2c55024d"]}],"objectID":"59397ce7ebd2970b2c550250","_highlightResult":{"name":{"value":"Blue Sun Sticker","matchLevel":"none","matchedWords":[]},"descriptions":{"short":{"value":"","matchLevel":"none","matchedWords":[]},"long":{"value":"<p>This is a description.\n</p>","matchLevel":"none","matchedWords":[]},"extended":{"value":"","matchLevel":"none","matchedWords":[]}},"createdOn":{"value":"2017-06-08T16:35:51.779Z","matchLevel":"none","matchedWords":[]},"updatedOn":{"value":"2017-08-15T17:40:24.0867695Z","matchLevel":"none","matchedWords":[]},"state":{"value":"Active","matchLevel":"none","matchedWords":[]},"categoryIds":[{"value":"59397c58ebd2970b2c55024c","matchLevel":"none","matchedWords":[]}],"seo":{"title":{"value":"Blue Sun Sticker","matchLevel":"none","matchedWords":[]},"metaDescription":{"value":"SEO Meta Description","matchLevel":"none","matchedWords":[]},"friendlyName":{"value":"blue-sun-sticker","matchLevel":"none","matchedWords":[]}},"tenant":{"value":"58533e0d3b77c800172ca14e","matchLevel":"none","matchedWords":[]},"productVariants":[{"id":{"value":"598b770debd2970684ff59cb","matchLevel":"none","matchedWords":[]},"sku":{"value":"blue-sun-sticker-blue-2-","matchLevel":"none","matchedWords":[]},"variants":[{"value":"blue","matchLevel":"none","matchedWords":[]},{"value":"2\"","matchLevel":"none","matchedWords":[]}],"skuImageLinkIds":[{"value":"59397c16ebd2970b2c55024b","matchLevel":"none","matchedWords":[]}]},{"id":{"value":"598b770debd2970684ff59cc","matchLevel":"none","matchedWords":[]},"sku":{"value":"blue-sun-sticker-blue-4-","matchLevel":"none","matchedWords":[]},"variants":[{"value":"blue","matchLevel":"none","matchedWords":[]},{"value":"4\"","matchLevel":"none","matchedWords":[]}],"skuImageLinkIds":[{"value":"59397c16ebd2970b2c55024b","matchLevel":"none","matchedWords":[]}]},{"id":{"value":"598b770debd2970684ff59cd","matchLevel":"none","matchedWords":[]},"sku":{"value":"blue-sun-sticker-orange-2-","matchLevel":"none","matchedWords":[]},"variants":[{"value":"orange","matchLevel":"none","matchedWords":[]},{"value":"2\"","matchLevel":"none","matchedWords":[]}],"skuImageLinkIds":[{"value":"59397c6febd2970b2c55024d","matchLevel":"none","matchedWords":[]}]},{"id":{"value":"598b770debd2970684ff59ce","matchLevel":"none","matchedWords":[]},"sku":{"value":"blue-sun-sticker-orange-4-","matchLevel":"none","matchedWords":[]},"variants":[{"value":"orange","matchLevel":"none","matchedWords":[]},{"value":"4\"","matchLevel":"none","matchedWords":[]}],"skuImageLinkIds":[{"value":"59397c6febd2970b2c55024d","matchLevel":"none","matchedWords":[]}]}]}}],"nbHits":2,"page":0,"nbPages":1,"hitsPerPage":16,"processingTimeMS":1,"exhaustiveNbHits":true,"query":"","params":"query=&facetFilters=%5B%22state%3AActive%22%2C%5B%22categoryIds%3A59397c58ebd2970b2c55024c%22%5D%5D&hitsPerPage=16&page=0&facetFilters=tenant%3A58533e0d3b77c800172ca14e%2C%20state%3AActive"}

Yes it does, and it works on our site to. The issue is it doesn’t work for googlebot. The simplest explanation I’ve seen, but one that I can’t experiment with on my end, is that it’s because the domain doesn’t have a robots.txt file:

I don’t think that’s the issue, and we should explicitly not have a robots.txt, since people have sensitive data on our servers. It’s not an easy problem to solve.