Magento 2 Indexing Queue keeps growing

cron
queue
magento-222

#1

I have 6 stores in a Magento 2.2.2 environment. Our queue/cron settings are set at 250 each cron interval which is set at the default 5 minutes. However, the queue keeps growing and never catches up. Is this working right or do I need to filter what it indexes or not somehow?


#2

I’m seeing this on 2.2.6, queue cron running every minute, 10 items per run, and it has been growing. Earlier I cleared the queue and reset all algolia indices and it started pushing products, but the queue size is increasing at a phenomenal rate. There are 45k entries, with 50 products in each. We only have about 5k products across 5 stores.


#3

Since posting this topic, I have been in contact with Algolia support. Items are added to the queue when a product is added, deleted or edited. 4 times per day, we run a .csv import that update inventory values. So, in effect we are editing 29K skus multiple times each day.

Their response:

This way, your whole catalog will be re-indexed 4 times a day but keep in mind what the “on save” and “on delete” events won’t be triggered anymore if you edit a product in the back-office, so you’ll have to make sure your catalog stays up-to-date in case you edit it manually (by running a full reindex manually for example).

We are testing that solution now.


#4

I realised the reindex of all products was running as part of my cron every minute, hence we were trying to reindex everything continually.

I’ve changed to updater on save as we will only be pushing products a max of once per day which should be safe.

Thanks for the advice - we will follow that if needed later.