No hits when filters are toggled

So I have a filters configuration using <Configure filters={filters()} /> where i conditionally set them based on selections. When i click x filter, remove x filter, and click x filter again, no hits are returned.

Only the first click on x filter the hits are returned perfectly. Perhaps the page is not reset when scrolling down? (I use infinite scroll) Not sure.