Django ORM query optimisations

Env versions:

  • Django version: Django==2.2.8
  • Algolia Django integration version: algoliasearch-django==1.7.1
  • Algolia Client Version: algoliasearch==1.20.0
  • Language Version: english

Problem

The rebuild_index script is creating many queries when building the index.

The following stats were observed when profiling queries:

When testing locally, an Index with local fields on the same model (No FK or M2M enabled) in fields.

1 query for 35 records (as expected since local fields)

When FK fields were enabled (no M2M on model):

Using the get_queryset method, selected_related and iterator was added. to speed things up.

376 queries for 35 records (using select_related)
481 queries without select_related

I expect the select_related to save a lot more queries. But it’s not?

The library asks us to put all fields on model methods - but is there something happening different where the Django ORM cannot optimize these queries with the way the library internals work?

When indexing large sets of records, the process can easily run out of memory.

Hey, thanks for reaching out! If that works for you, we’ll reply directly on the GitHub issue that you opened :upside_down_face: