Reindexing Large Shopify Catalog

We’re working with a catalog of over 50,000 products. Each of these products has a metafield with comma-separated collection handles (e.g. ‘shirts,sunglasses,accessories’) that the product belongs to, and this metafield is configured as a facet in Algolia.

We’re having issues reindexing this catalog. The “Indexing” tab in the Algolia Shopify app only recognizes about 1,300 products (of over 50,000), only recognizes 1 collection (of over 200), and clicking the “Reindex” buttons doesn’t seem to do anything.

The POST request to /algolia_configs/trigger_indexing returns 200 when clicking “Reindex”, but nothing is ever queued, and no more than ~1300 products and 1 collection are ever recognized.

This is consistent with the very limited results we see on the search page of our site.

Can you help give us some insight into the potential causes of this? Thanks!

Also, we just found that when we trigger reindexes and wait for a while, the following appears in the indexing queue:

delete_collection 1,030
index_all 11
index_collection 54,209
index_blogs 1
index_product 100,320
index_variants 1

This only appears after waiting for a minute, and then once we refresh the page it disappears. This seems like odd behavior. Should we consider reinstalling or something?

The 11 index_all definitely seems weird. Could you provide us with your store name or your app id?

I’ve DMed you the Application ID.

We tried upgrading our plan to the “Essential” tier, it looks like it may have been disabled earlier due to that (though our Shopify app was displaying a more general non-specific error). Even after this upgrade though, the number of indexed products is not increasing (still only about 1.3k of 56k).

Help is greatly appreciated – we’re really hoping to have this functioning ahead of early next week. If the issue is with plan limitations, we’re happy to upgrade further.

Hi Jerska,

Do we have an update on this yet? We’re really hoping to have this resolved ahead of next week. I still can’t seem to get it working – product count is still ~1.3K over 24 hours later.

Any advice is greatly appreciated. Thanks.

The issue was that there was a problem on Shopify’s API side when we tried to contact them.
Network errors are to expect in such systems, and we do have a retry strategy in those cases.
However, this seems to have lasted longer than our 5 tried spanning over an hour.
Whenever those 5 tries in a row fail, we stop indexing for your store until a manual action is made to restore it.

The POST request to /algolia_configs/trigger_indexing returns 200 when clicking “Reindex”, but nothing is ever queued, and no more than ~1300 products and 1 collection are ever recognized.

Actually it did its job : enqueuing an index_all job. But since the queue was paused, it solely enqueued them and never processed them.
At the top of the app, you should have had a red warning. Indeed, the error was probably not specific, since 500’s are not an error we provide an human-readable description to, so we simply display An unknown error has happened, please contact us.

Since your last messages, it seems like you’ve clicked on Resume indexing, since you don’t have any job enqueued anymore.
I hope this explanation will make sense. :slight_smile: