Is there a way to set Crawler: ExclusionPatterns on the Netlify plugin

Netlify Integration
1

I would like to exclude certain pages from the Algolia netlify crawler, but don’t want to exclude other robots from crawling them.

I would like to be able to set Crawler: ExclusionPatterns in the Algolia crawler API, but it doesn’t seem to be an available configuration in the Netlify plugin.

Is there a way to set this parameter, or could we request to have that as a configuration feature in the Netlify plugin?

2

Hi, the Netlify plugin indeed doesn’t expose such configuration. To exclude some pages only for the Algolia Crawler, you can leverage a robots.txt file with some specific rules for our user agent.