Can't get netlify plugin working: Forbidden (403) all the time

Hi, I tried a few things.

  1. the initial setup from algolia via the UI didn’t put environment variables in for me in Netlify, so I had to put them myself. I suspect this might be because I’m on a free plan at netlify at the moment (just trying to test if this is a direction I want to go) and the free plan only allows certain types of environment variables - i.e. might be a breaking change on their side.
  2. I’ve tried to use the api key generated by the plugin setup (I found it on my api keys page) and put that in the ALGOLIA_API_KEY field.
  3. I tried both with & without the ALGOLIA_BASE_URL specified (btw, the plugin / help suggests it should be input as “https://crawler.algolia.com/” (with the trailing slash) but that seems to not be what the plugin is expecting as that will fail with an extra “/” in the URL it uses to contact the service.
  4. I tried turning on NETLIFY_BUILD_DEBUG but that didn’t really show any extra information.

Here is the output from my plugin run:

11:02:30 AM: ────────────────────────────────────────────────────────────────
11:02:30 AM:   6. @algolia/netlify-plugin-crawler (onSuccess event)          
11:02:30 AM: ────────────────────────────────────────────────────────────────
11:02:30 AM: ​
11:02:30 AM: Step starting.
11:02:30 AM: Step started.
11:02:30 AM: Plugin logic started.
11:02:30 AM: Algolia Netlify plugin started
11:02:30 AM: Sending request to crawl https://crawler.algolia.com/api/1/netlify/crawl
11:02:30 AM: {"branch":"main","siteName":"rainbow-arithmetic-34c709","deployPrimeUrl":"https://6376850d7b713403acce7e60--rainbow-arithmetic-34c709.netlify.app","version":"1.0.0"}
11:02:30 AM: Response {
11:02:30 AM:   size: 0,
11:02:30 AM:   timeout: 0,
11:02:30 AM:   [Symbol(Body internals)]: {
11:02:30 AM:     body: PassThrough {
11:02:30 AM:       _readableState: [ReadableState],
11:02:30 AM:       _events: [Object: null prototype],
11:02:30 AM:       _eventsCount: 5,
11:02:30 AM:       _maxListeners: undefined,
11:02:30 AM:       _writableState: [WritableState],
11:02:30 AM:       allowHalfOpen: true,
11:02:30 AM:       [Symbol(kCapture)]: false,
11:02:30 AM:       [Symbol(kCallback)]: null
11:02:30 AM:     },
11:02:30 AM:     disturbed: false,
11:02:30 AM:     error: null
11:02:30 AM:   },
11:02:30 AM:   [Symbol(Response internals)]: {
11:02:30 AM:     url: 'https://crawler.algolia.com/api/1/netlify/crawl',
11:02:30 AM:     status: 403,
11:02:30 AM:     statusText: 'Forbidden',
11:02:30 AM:     headers: Headers { [Symbol(map)]: [Object: null prototype] },
11:02:30 AM:     counter: 0
11:02:30 AM:   }
11:02:30 AM: }
11:02:30 AM: Could not reach algolia Error: Forbidden {}
11:02:30 AM:     at onSuccess (/opt/build/repo/.netlify/plugins/node_modules/@algolia/netlify-plugin-crawler/dist/index.js:107:19)
11:02:30 AM:     at processTicksAndRejections (node:internal/process/task_queues:96:5)
11:02:30 AM:     at async Object.run (file:///opt/buildhome/node-deps/node_modules/@netlify/build/lib/plugins/child/run.js:14:5)
11:02:30 AM:     at async handleEvent (file:///opt/buildhome/node-deps/node_modules/@netlify/build/lib/plugins/child/main.js:28:42)
11:02:30 AM:     at async process.<anonymous> (file:///opt/buildhome/node-deps/node_modules/@netlify/build/lib/plugins/ipc.js:91:24)
11:02:30 AM: Step erroring.
11:02:30 AM: Error: Could not reach Algolia's Crawler, got: Forbidden {}
    at onSuccess (/opt/build/repo/.netlify/plugins/node_modules/@algolia/netlify-plugin-crawler/dist/index.js:112:21)
11:02:30 AM:     at processTicksAndRejections (node:internal/process/task_queues:96:5)
11:02:30 AM:     at async Object.run (file:///opt/buildhome/node-deps/node_modules/@netlify/build/lib/plugins/child/run.js:14:5)
11:02:30 AM:     at async handleEvent (file:///opt/buildhome/node-deps/node_modules/@netlify/build/lib/plugins/child/main.js:28:42)
11:02:30 AM:     at async process.<anonymous> (file:///opt/buildhome/node-deps/node_modules/@netlify/build/lib/plugins/ipc.js:91:24)
11:02:30 AM: Step completed.
11:02:30 AM: ​
11:02:30 AM: ────────────────────────────────────────────────────────────────
11:02:30 AM:   Plugin "@algolia/netlify-plugin-crawler" failed               
11:02:30 AM: ────────────────────────────────────────────────────────────────
11:02:30 AM: ​
11:02:30 AM:   Error message
11:02:30 AM:   Error: Could not reach Algolia's Crawler, got: Forbidden {}
​
11:02:30 AM:   Plugin details
11:02:30 AM:   Package:        @algolia/netlify-plugin-crawler
11:02:30 AM:   Version:        1.0.0
11:02:30 AM:   Repository:     git+https://github.com/algolia/algoliasearch-netlify.git
11:02:30 AM:   npm link:       https://www.npmjs.com/package/@algolia/netlify-plugin-crawler
11:02:30 AM:   Report issues:  https://github.com/algolia/algoliasearch-netlify/issues
11:02:30 AM: ​
11:02:30 AM:   Error location
11:02:30 AM:   In "onSuccess" event in "@algolia/netlify-plugin-crawler" from Netlify app
11:02:30 AM:       at onSuccess (/opt/build/repo/.netlify/plugins/node_modules/@algolia/netlify-plugin-crawler/dist/index.js:112:21)
11:02:30 AM:       at processTicksAndRejections (node:internal/process/task_queues:96:5)
11:02:30 AM:       at async Object.run (file:///opt/buildhome/node-deps/node_modules/@netlify/build/lib/plugins/child/run.js:14:5)
11:02:30 AM:       at async handleEvent (file:///opt/buildhome/node-deps/node_modules/@netlify/build/lib/plugins/child/main.js:28:42)
11:02:30 AM:       at async process.<anonymous> (file:///opt/buildhome/node-deps/node_modules/@netlify/build/lib/plugins/ipc.js:91:24)
11:02:30 AM: ​
11:02:30 AM:   Resolved config
11:02:30 AM:   build:
11:02:30 AM:     command: astro build
11:02:30 AM:     commandOrigin: config
11:02:30 AM:     environment:
11:02:30 AM:       - ALGOLIA_API_KEY
11:02:30 AM:       - ALGOLIA_BASE_URL
11:02:30 AM:       - NETLIFY_BUILD_DEBUG
11:02:30 AM:     publish: /opt/build/repo/dist
11:02:30 AM:     publishOrigin: config
11:02:30 AM:   functionsDirectory: /opt/build/repo/netlify/functions
11:02:30 AM:   plugins:
11:02:30 AM:     - inputs: {}
11:02:30 AM:       origin: ui
11:02:30 AM:       package: '@algolia/netlify-plugin-crawler'
11:02:30 AM:     - inputs: {}
11:02:30 AM:       origin: config
11:02:30 AM:       package: netlify-plugin-cypress