Can partial crawl in docsearch-scraper?

Hi,

Can I crawl only ‘the new’ on my website using docsearch-scraper?

I add some new contents on my website, but I don’t want to crawl the whole website.

Thanks!

:wave: @webosose

Yes you can but you will only have these new pages indexed into your index. You can use the stop_urls settings to avoid crawling old pages.

Cheers

1 Like