Concern with Firebase + algolia indexing

Currently I am building a React Native Application with a Firebase backend. I needed to do some more elaborate querying and decided to turn to Algolia to support it. So far everythign has worked great! I love all the functionality that Algolia has to offer but am concerned about the number of indexing queries that are potentially being run when I setup a ref to my firebase database.

Previously our course data (the table we needed Algolia for) has ~17,000 nodes on it but was static data. Our requirements have changed and it is becoming evident that this data will no longer be static. Certain fields will be updating constantly (pretty much every hour of so).

To do this I found / wrote a nifty update function:

updateCoursesAlgolia() {
  let algoliaClient = algoliasearch("XYZ", "ZYX);
  let courseRef = firebase.database().ref("courses");
  let courseIndex = algoliaClient.initIndex("courses"); // this will actually just grab the index

  courseRef.on("child_added", addOrUpdateIndexRecord);
  courseRef.on("child_changed", addOrUpdateIndexRecord);
  courseRef.on("child_removed", deleteIndexRecord);

  function addOrUpdateIndexRecord(dataSnapshot) {
    // Get Firebase object
    let firebaseObject = dataSnapshot.val();
    // Specify Algolia's objectID using the Firebase object key
    firebaseObject.objectID = dataSnapshot.key;
    // Add or update object
    courseIndex.saveObject(firebaseObject, function(err, content) {
      if (err) {
        throw err;
      }
      console.log("Firebase object indexed in Algolia", firebaseObject.objectID);
    });
  }

  function deleteIndexRecord(dataSnapshot) {
    // Get Algolia's objectID from the Firebase object key
    var objectID = dataSnapshot.key;
    // Remove the object from Algolia
    courseIndex.deleteObject(objectID, function(err, content) {
      if (err) {
        throw err;
      }
      console.log("Firebase object deleted from Algolia", objectID);
    });
  }

Let me start by saying, this is working fantastically BUT I notice that every time I load my app I see 17000 console logs in my console addOrUpdateIndexRecord function. This means that it is changing every single record on my Aloglia index. I was expecting to see console logs for only those that had changed, not EVERY one. Is something flawed in my setup? I am not too concerned about this from a performance perspective BUT I am concerned about this from a cost perspective. Will I be charged for all the updating of my instances? Also, is there a way to only update these when they change using a firebase ref?

Thanks everyone!

Hey there :wave:

From the Algolia side you’re doing it right: by calling courseIndex.saveObject({...}) you’re executing an operation which will be counted in your plan quota but the actual object upsert will be handled by Algolia by using their ObjectID. As long as the ObjectID doesn’t change every time the callbacks are invoked you should be fine :ok_hand:

I think that the culprit here is one of the Firebase callbacks, child_added:

https://firebase.google.com/docs/database/admin/retrieve-data#child-added

child_added is triggered once for each existing child and then again every time a new child is added to the specified path.

That would explain the 17.000 console logs each time the app is loaded.

Let me know if this helps you out!

1 Like