Algolia as a datastore

Has anyone used Algolia as their primary datastore much like a database? Where all search queries are indexes and updates take place much like a read/write query within an application.

I see potential in using Algolia as not just a search function but to retrieve data for a more conventional application, content, feeds etc.

Thoughts?

3 Likes

Hi @jjaybrown98

Algolia wasn’t designed to be used as a database. It means you will miss some features compared to more traditional databases. For example, you will have no way to determine relations between records of different indices.

Also records needs to be indexed by Algolia before being returned by queries, so you’ll have to make sure your app deals correctly with eventually consistent data.

That being said, if you are fine working around the downsides you could definitely use Algolia as a database.

I’d gladly discuss your implementation further if you decide to build a proof of concept :wink:

Hi @rayrutjes, sorry let me clarify, I intend to use it as the primary source of data, not strictly as a database, it was more for context, that most applications use a database as their primary source of data.

Way I see it, we only need to retrieve read-only data, insertions happen elsewhere, potentially using a conventional db then updating the index. The app we’re working on, will only use the updated index.

1 Like

Great question @jjaybrown98! It’s one we get often and I’ll go into a little extra detail to explain the way we think about it. There are two parts to the “Algolia as a datastore” conversation.

Writes

We always recommend writing to a conventional datastore first, then pushing the data to Algolia. There are several reasons for this.

  1. Most conventional databases are immediately consistent (you can read immediately after you write and get the new data). This is essential for CRUD-based applications where saved form data is presented back to the user right away. Algolia is eventually consistent. It can take a second or two after indexing before new records appear in results. That’s not a good fit for CRUD apps.

  2. The best JSON format for search might be different than the best general purpose format. The indexing process often involves a transformation where attributes of the original JSON are added to, deleted or modified. For example, your general purpose data might have “phone_number” and “email” attributes, but for performance and security reasons you are better to exclude then from your search JSON if they can’t be searched on. If you only have an index but not a database behind it, you are forced to keep everything in the index, which may end up becoming unwieldy with too many attributes.

  3. You might need multiple indices, each with a different JSON structure, to handle all of your use cases. Safely updating multiple indices simultaneously is hard to reason about, especially if no single index can be considered a full, single source of truth. If an update succeeds against one index but fails against another, you may end up in an inconsistent state. You cannot make the whole transaction atomic (where all-or-nothing succeeds or fails) and you will need to write complicated retry logic to try to avoid inconsistencies.

These reasons are why we always recommend that you use a conventional database as the single source of truth for your application and push data from that database to Algolia. It might be a little more up-front setup but it saves headaches and provides flexibility down the road.

Reads

If most of the data your application needs can be fetched via an Algolia search, without jumping through a lot of hoops, than using Algolia as the “primary” source for reads can work really well. Not all searches require that a user explicitly types in a search query. Many other search refinements exist including facets, filters, and different orderings that can be prefilled by your application. So you can display “search results” to a user without them knowing it’s a search.

This is very common actually. The pattern works great for creating a browsing experience alongside a search, and many good applications combine both for the best overall experience. A big advantage of using Algolia to power browsing features is speed. Algolia’s infrastructure is highly optimized for low latency and will nearly always return results faster than an application server with a default or even optimized configuration, especially as the number of searchable records increases (and when taking into account global latency or using DSN). Using Algolia means that your user will see the page populated and ready to interact with in less time, even as your data set grows.

One thing we don’t recommend is using Algolia to do reads that must be immediately consistent, like when the data is being used to populate a form that the user will modify and save. If the data that is used to perform an update is out of date, the newest data will be overwritten. Remember that indexing operations can take a few seconds to complete, meaning that there is a window of time where your database and Algolia will serve different data for the same record.

So to sum it up - for writing data from your application, always write to a conventional database first and then push the data to Algolia. This gives you the most reliability and flexibility. Don’t write directly to Algolia without a database or you will be subject to inconsistencies and race conditions that will require you to maintain complex indexing code. For reading data from your application, use Algolia to power both search and browsing experiences. Algolia is heavily optimized for speed and will provide your users with the fastest loading times even as your application scales.

9 Likes

I built https://appapp.io with exactly this architecture.

There is a relational DB that is the master record, but all user access is exclusively with the Algolia index.

The architecture works well for this app, but I’d be hesitant to use it on any app that would require ‘proper relational’ functionality.

2 Likes

That’s super cool Dan! I hadn’t seen AppApp before, thanks for sharing it as an example. The second-level menu for filtering inside of the Games category is :ok_hand:.

Feel free to drop a link over in Projects if you want more people to check it out :slight_smile:

1 Like