Should i change my backend database to adapt to Algolia or not?

I’m a backend developer trying to use Algolia for the client’s needs and I’m having a bit of trouble adapting to this indexing logic.

Database (Mongo) for the backend app has entities like Venue (a place and information about the location) and Market (a more abstract entity with information about schedule, and vendors present there).

which are interconnected with a foreign key. For example Market documents have field venueId which states what is the venue this market is currently at.

Currently, I think my indexing of these documents (sending these data to Algolia) is going to happen upon their creation and editing, meaning Algolia will have a truthful ‘copy’ of these two collections.


With this setup, I will have two indexes (market_index and venue_index) and my instant search (on frontend) can successfully (at least I think) lookup on:

A) markets named N (this is because Market has title field) :heavy_check_mark:
B) venues named N (this is because Venue has title field)
C) venues near location L (this is because Venue has _geoloc field)

but my question is: can it also lookup on:
D) markets near location L

With the above setup, i don’t think so? So as i thought about this, two options emerged:

  1. Change the database models to have all necessary information for the algolia search engine, meaning add _geoloc field to the Market field. A duplicate of information - yes, for the sake of Market index and searching for “markets near location L”
  2. Change the database models to not use foreign keys but nested documents, meaning convert Market’s venueId (type ObjectId) into venue (type Object) and thus allow (i guess?) “markets near location L” because _geoloc is also nested inside Market object.
  3. Change what is indexed, meaning that i don’t send copy of the database-like structure, but rather a structure from either point 1) and 2) just so that algolia can do these searches, and that i keep data as it is right now in the database
  4. Some other approach?

Points 1) and 2) are troublesome to me since they require me to drastically change how the database is interconnected just to adapt to some service like Algolia. Yes - the app is primarily for searching places - but should third party service really require database adapted just to it?

Point 3) is troublesome because I just don’t have the knowledge/information whether something like this should be done, and in what shape or form? Without examples i couldn’t figure out if my indexes should be some kind of aggregation based on some parameter, or should they be like collections in Mongo?

If someone could clarify this I am happy to give more concrete examples on the actual models, the app is not just Market and Venues but also Organizers and Brands/Vendors, so these questions are just problem-describing, not all picture just for sake of giving you enough information for me to form a question.

1 Like

Hi Nikola! These are great questions :slightly_smiling_face:

Algolia is an external service. You may have other services that you want to connect to your application, such as monitoring or analytics, and these services may change down the road. For those reasons, you should definitely not adapt your data model to Algolia (or any other service), but instead write code that allows you to transform your data into a format that Algolia understands (and same for any other service). As you correctly raised, “should third party service really require database adapted just to it?”. The answer is no, and we definitely don’t recommend doing it.

What you need to change is what is indexed to Algolia. We don’t recommend sending an exact copy of your data to Algolia, because it’s often not the right format. Yes, Algolia accepts JSON records which look a lot like Mongo collections, but it has certain needs (e.g., the special objectID or _geoloc attributes) that you may not have in the same format in your data store.

One of the important things about Algolia is that it’s schemaless and doesn’t support joins with foreign keys as you usually do. An Algolia record is a result: it should be self-contained, and hold all the necessary information to be found, filtered, and displayed.

We have a full guide on this topic in our documentation: https://www.algolia.com/doc/guides/sending-and-managing-data/prepare-your-data/how-to/handling-data-relationships/

So, in your specific use case, you have Venues and Markets, but the geo information of each Market is in their associated Venues. This means you indeed need to aggregate your data before sending it to Algolia, so that each record that represents a Market has the associated geo information.

In the end, you want your records to look like this contrived example:

[
  {
    "name": "Market A",
    "venue": "Venue 1",
    "_geoloc": { "lat": 40.712776, "lon": -74.005974 },
    "type": "market"
  },
  {
    "name": "Venue 1",
    "venues": ["Market A"],
    "_geoloc": { "lat": 40.712776, "lon": -74.005974 },
    "type": "venue"
  },
]

Here, we have Venue 1 which has many markets, and Market A which has one venue. We have duplicate geo information so both records are independant. We’ve also added information, such as the type attribute, which isn’t in the initial collection but that we added for convenience so we can perform front-end logic in our search experience based on record type.

So, to summarize, we recommend you craft your data model in the way that best fits your application’s business logic and domain, not adapt it to Algolia, and write an Algolia-specific service that queries data from Mongo, aggregates and transforms it for Algolia, and indexes it with one of our API clients.

Best,

1 Like

Great, that’s actually the approach I have started implementing. (adding _geoloc to each entity but only ‘towards algolia’). Thanks for the response, glad I’m on the right track. :smile:

1 Like