Can partial crawl in docsearch-scraper?


Can I crawl only ‘the new’ on my website using docsearch-scraper?

I add some new contents on my website, but I don’t want to crawl the whole website.


:wave: @webosose

Yes you can but you will only have these new pages indexed into your index. You can use the stop_urls settings to avoid crawling old pages.


1 Like