Netlify Crawler not crawling SPA React app

Netlify Integration
,
1

I am not having success getting the Algolia for Netlify crawler to crawl my site. I have docs-style React Single-Page-Application (SPA) built with react-router that contains a bunch of pages with educational contact. (https://grammar.lingdocs.com - source: GitHub - lingdocs/pashto-grammar: LingDocs Pashto Grammar Reference).

Google has no trouble crawling and indexing all the content of this site, but the Algolia crawler seems not to see any of the content. At first it just crawled the index.html and didn’t find any links or content to index. Then I added a sitemap.xml and it crawled all the pages but still found nothing. :confused: Is there a way to configure the crawler to be able to read the info rendered on this React site?

2

Hi, your website requires JavaScript, if you are not able to expose a light version for robots, the plugin has an option to render it using JavaScript: Plugin configuration | Algolia

3

Thanks, I just added these lines to my netlify.toml and that solved it.

[[plugins]]
package = "@algolia/netlify-plugin-crawler"
  [plugins.inputs]
  renderJavaScript = true
1 Like