Indexing of big records in rails

Hi, I’m trying to solve the following problem:

I have a record of about 5 KB, to which I want to add a property that could weight 30+ kb. Since this goes over the 20kb limit I found the Distinct to Index Large Records article which says you should split it into multiple records.

The thing is that the current rails gem does not support the record splitting since it treats each Database record as a single record in algolia, I read issue #241 on that repo and saw that there is no current way to do that on it.

So my question is, what is the recommended approach here? I’d have to rewrite a lot of my algolia code if I change every part of the rails app to work with the pure ruby gem.

Is there any other library in another language that you guys offer that actually does record splitting correctly?

Thanks!

Hi Alejandro!

A property that could weight 30+ KB? That seems like a lot!

In order for us to help you find relevant solutions, can you tell us more about your use case and what kind of records you intend to store in your search index?

Sure, so what I currently store in algolia is a model which has some info about it and a tags attribute. It already works good on production and search works great with the tags and the name it already has.

However we also have an additional set of tags we want that object to have so we can use Algolia to search it by them, this tags are about 10 bytes each, but we currently have some objects with over 1,500 tags, so this would be about 15 KB. The number of tags could grow all the way to 3,000 or more. The tags could also be created, updated or deleted at any time (so the tags need to be updated)

The tags are stored as an array.

The solution I had come up with (without rails) is to implement a basic overflow algorithm, I limited each record to N tags that would make the record < 20 KB. Once the record surpassed the N tags I cloned it, changed the objectID and added another record with the remaining tags and so on. This solution works good since I handle deletions by just removing the element from the array and each time a new tag is added I added it in whichever record had less than N tags. I already coded a ruby script for this

The things is that the rails gem does not seem to have any way to implement something like this, and if I remove it I lose the ability to use the callbacks that it already provides for create, update and delete

I’m also trying to do this with one search since everything is already implemented in the frontend and I don’t want that interface to change.

1 Like

Thanks for your reply, Alejandro!

Your use case makes sense and your implementations seems good! We follow a similar approach in some of our products (e.g. docsearch, I believe).

If you would like to integrate this possibility into our official gem for rails, I believe that the best way is to propose a Pull Request to the corresponding repo. The PR could contain that splitting feature, or a partial refactor that would allow this logic thru a plug-in. As I am not a contributor of that codebase, nor a Rails developer, I don’t think I can help you further on this topic, unfortunately.