Not reindexing products

Hello,

we are still running on an algolia trial with Magento 1.9.x where we’d like replace our internal search.
I manually indexed the data on the cli … and while all the other indexes worked fine, the one for products doesn’t bring anything to the indexes.
It just loads for like 20 seconds and then tells that its done, not errors, not killed or alike.
I also tried the queue, changing a product which was somehow started to process but still didn’t end up in the index.

We have about 30k products which are enabled, visible in catalog&search, available for Sale and at least one item in stock. We are using dropship module though … does that make any difference?
Any ideas what could go wrong there? What I could look into? The log is not giving any error or additional information.

Thanks,

Thomas

1 Like

Hello @thomas.seifert,

for the store with around 30k products, it might be complicated to index all products at once. Because of that I’d recommend you to enable indexing queue as it indexes products in smaller batches and it’s not that demanding on hardware and PHP resources.

The way how the queue works is that when you reindex products indexer, it actually doesn’t reindex products to Algolia immediately, but it creates a bunch of reindexing jobs and those jobs are stored in algoliasearch_queue table in your Magento database.

In order to process those jobs, you need to set up a cron job which will run every few minutes (let’s say 5) and each time it runs it processes a small portion of those created jobs. Once all the jobs are processed, your data will be reindexed.

Here you can find our indexing documentation and you can find there more information on how indexing and the indexing queue works in Magento.

For now as you’ve said you had enabled indexing queue I’d recommend you to go to you Magento database and truncate the content of algoliasearch_queue table. Then you can navigate to Algolia configuration in Magento and enable the indexing queue there.
Then you can reindex Products and go back to your database and check if there are any rows in the algoliasearch_queue table.
If there you find some jobs there, feel free to set up a queue runner cron job and let it process those jobs. Once the jobs are processed, you’ll see products indexed.

Let me know if it works for you.

1 Like

Hello @jan.petr

Thanks but it seems to add only 236 products to the queue, each time its run :-/.
Thats by far not all … but also the queue runner doesn’t reduce them or push them to algolia itself.
Will try to upgrade to 1.8.1 now to see if it helps.

Same thing with 1.8.1, no improvement :frowning:

In the log I see stuff like that … and the log ends afterwards.

2017-02-22T11:28:34+00:00 DEBUG (7):
2017-02-22T11:28:34+00:00 DEBUG (7):
2017-02-22T11:28:34+00:00 DEBUG (7): >>>>> BEGIN rebuildStoreProductIndexPage 3 (XXX.com) page 11 pageSize 1000
2017-02-22T11:28:34+00:00 DEBUG (7):
2017-02-22T11:28:34+00:00 DEBUG (7):
2017-02-22T11:28:34+00:00 DEBUG (7): >>>>> BEGIN START EMULATION
2017-02-22T11:28:34+00:00 DEBUG (7): <<<<< END START EMULATION (0.012844085693359sec)
2017-02-22T11:28:34+00:00 DEBUG (7):
2017-02-22T11:28:34+00:00 DEBUG (7):
2017-02-22T11:28:34+00:00 DEBUG (7): >>>>> BEGIN LOADING 3 (XXX.com) collection page 11, pageSize 1000
2017-02-22T11:28:41+00:00 DEBUG (7): Loaded 1000 products
2017-02-22T11:28:41+00:00 DEBUG (7): <<<<< END LOADING 3 (XXX.com) collection page 11, pageSize 1000 (6.5070929527283sec)
2017-02-22T11:28:41+00:00 DEBUG (7):
2017-02-22T11:28:41+00:00 DEBUG (7):
2017-02-22T11:28:41+00:00 DEBUG (7): >>>>> BEGIN CREATE RECORDS 3 (XXX.com)
2017-02-22T11:28:41+00:00 DEBUG (7): 1000 product records to create
2017-02-22T11:28:41+00:00 DEBUG (7):
2017-02-22T11:28:41+00:00 DEBUG (7):
2017-02-22T11:28:41+00:00 DEBUG (7): >>>>> BEGIN CREATE RECORD 85633 3 (XXX.com)
2017-02-22T11:28:41+00:00 DEBUG (7): Product type (simple, mapped to: simple)

Hello, it’s weird.
But it looks like you change the default settings of the queue. Can you set the maximum number of elements to 100 and maximum number of jobs to process to 10 and then try it all again?

@jan.petr
did that, no real change …

2017-02-22T14:07:25+00:00 DEBUG (7): >>>>> BEGIN PRODUCTS FULL REINDEX
2017-02-22T14:07:49+00:00 DEBUG (7): <<<<< END PRODUCTS FULL REINDEX (23.596950054169sec)
2017-02-22T14:08:08+00:00 DEBUG (7):
2017-02-22T14:08:08+00:00 DEBUG (7):
2017-02-22T14:08:08+00:00 DEBUG (7): >>>>> BEGIN rebuildStoreProductIndexPage 7 (xxx.ca) page 1 pageSize 100
2017-02-22T14:08:08+00:00 DEBUG (7):
2017-02-22T14:08:08+00:00 DEBUG (7):
2017-02-22T14:08:08+00:00 DEBUG (7): >>>>> BEGIN START EMULATION
2017-02-22T14:08:08+00:00 DEBUG (7): <<<<< END START EMULATION (0.0053019523620605sec)
2017-02-22T14:08:08+00:00 DEBUG (7):
2017-02-22T14:08:08+00:00 DEBUG (7):
2017-02-22T14:08:08+00:00 DEBUG (7): >>>>> BEGIN LOADING 7 (xxx.ca) collection page 1, pageSize 100
2017-02-22T14:08:11+00:00 DEBUG (7): Loaded 100 products
2017-02-22T14:08:11+00:00 DEBUG (7): <<<<< END LOADING 7 (xxx.ca) collection page 1, pageSize 100 (2.7761931419373sec)
2017-02-22T14:08:11+00:00 DEBUG (7):
2017-02-22T14:08:11+00:00 DEBUG (7):
2017-02-22T14:08:11+00:00 DEBUG (7): >>>>> BEGIN CREATE RECORDS 7 (xxx.ca)
2017-02-22T14:08:11+00:00 DEBUG (7): 100 product records to create
2017-02-22T14:08:11+00:00 DEBUG (7):
2017-02-22T14:08:11+00:00 DEBUG (7):
2017-02-22T14:08:11+00:00 DEBUG (7): >>>>> BEGIN CREATE RECORD 3256 7 (xxx.ca)
2017-02-22T14:08:11+00:00 DEBUG (7): Product type (simple, mapped to: simple)

Is the entire queue processed?

no, it simply stops there.
no entries from the queue are being removed.

Can you check your Magento’s error logs if there are any logged exceptions or errors?

thanks @jan.petr but there is also nothing shown in those logs. Thats what is killing me somehow here.
The last entry in those logs is from a couple days ago.

Hey Thomas,

it’s really weird. Would you mind sending us an e-mail to support+magento@algolia.com?
I will get back to you from there.

Thanks!