'Record at the position 0 is too big size=24307 bytes. Contact us if you need an extended quota'

I tried to run a search on a something inside Firebase and my records weren’t getting updated. For example if a book’s name was “Harry Patter” and then I corrected it and updated the name to Harry Potter my Algolia console was still showing Harry Patter. I ran $ heroku logs and this is the message that came back:

‘Record at the position 0 is too big size=24307 bytes. Contact us if you need an extended quota’

I did some research on your forums and another user had asked a similar question.

I read @haroen answer and he said to use distinct

It got me to thinking though. When I looked at my Algolia Indices console I saw the searchable attributes and then I noticed everything else under that attribute was also inside the console even though I never specified that.

For example a book has a title, author, publisher, and ISBN number. Say my app lets users upload their favorite books and leave a review about it. I only want users to be able to search by title, author, isbn, and publisher. Searching those attributes works fine but when I look inside the Algolia Indices console I also see all the other information such as the the usersReviews node, each userId who left a review, the review itself, the date, and the postId from that userID. When I looked at the heroku logs all this information was visible and it seems like the review that the user left is what’s causing the file size issue because the whole thing printed out.

Here’s a short description of the process:

  1. SearchBar says: Search by title, author, isbn, and publisher.

  2. A user types in Harry Potter and Harry Potter and the Sorcerer’s Stone appears.

  3. The user chooses Harry Potter and then a new scene gets pushed on and that scene shows a tableView with the the reviewer’s name, date, reviewTitle. The user’s chooses a cell and then a scene with the review gets pushed on.

None of the reviewers info is searchable via Algolia and I pull all of that information independently inside each view controller (scene). There is no reason for it to appear inside the Algolia Indices console.

Inside the Algolia Indices console:

author: "JK Rowling"
isbn: "12345"
publisher: "Bloomsbury Publishing"
title: "Harry Potter and the Sorcerer's Stone"
userReviews: > {zz00zz11: {qwe789rty123}} // why is this getting indexed?

My two questions areif I only set the searchable attributes as title, author, isbn, and publisher, why is everything else (userReviews node) getting indexed?

Could this be the reason the Indices didn’t update to show Harry Potter but was still showing Harry Patter even though firebase was updated to show Harry Potter?

Btw I’m on the $35 plan. I’m not sure if that makes a difference or not.

Here’s the Firebase layout:

root
  |
  @---books
        |
        @----12345 // book's isbn number
               |
               |-----"title": "Harry Potter and the Sorcerer's Stone"
               |
               |-----"author": "JK Rowling"
               |
               |-----"isbn": "12345"
               |
               |-----"publisher": "Bloomsbury Publishing"
               |
               @----userReviews
                        |
                        @-----z00zz11 // this is a user's id
                        |       |
                        |       @-----qwe789rty123 // postId
                        |                  |
                        |                  |-----"review": "this is a very long 7 paragraph review...."
                        |                  |-----"date": "07292018"
                        |                  |-----"reviewTitle": "I love this book!"
                        |                               
                        @-----a22aa44 // this is different user's id
                                |
                                @-----y177xx2v2 // postId
                                           |
                                           |-----"review": "this is a 5 paragraph review...."
                                           |-----"date": "07292018"
                                           |-----"reviewTitle": "I hate this book!"

Inside my iOS app I set the search attributes as

buildingIndex = apiClient.index(withName: "books")
query.hitsPerPage = 15
query.attributesToRetrieve = ["title", "author", "isbn", "publisher"]

and inside the Algoia console I set the Searchable Attributes to match:

22 AM

Hello!

The reason is because “searchable attributes” does not mean that the data is not in your index. One of the reasons why you can specify “searchable attributes” is to avoid downloading a lot of data when all you need is a few attributes. That will make the query faster and will require less network bandwidth.

So in order to avoid putting all attributes in your index, all you need to do is to remove them from your object before indexing (when calling AddObjects or UpdateObjects for example).

Another more advanced way is to use a generated secured api key that can only modify certain attributes. You would exclude usersReviews and other “unwanted” fields.

Yep that is the reason. In order to go above the limit of 20KB for essential plans, you have to be in a business or enterprise account.

We really recommend that you find a way to split up the records if they’re big with the distinct attribute which you mentioned. This will be very beneficial for the performance and speed of your queries.

Hey @guy.daher, thanks for the much needed help! I was going crazy all week because i though the databaseAuthVariableOverride was causing me problems again. Its such a relief it’s not that again.

I’m not a native node.js developer so I’d rather go the easiest way possible. Instead of splitting the records using distinct it seems that removing them from addObjects or UpdateObjects is the best bet.

Here’s the thing, there are also other fields that are getting indexed that I want to exclude (I didn’t include in the example):

root
  |
  @---books
        |
        @----12345
               |
               |-----"title": "Harry Potter and the Sorcerer's Stone"
               |-----"author": "JK Rowling"
               |-----"isbn": "12345"
               |-----"publisher": "Bloomsbury Publishing"
               |
               @----userReviews...
               |
               |-----"filmTitle": "Harry Potter and the Sorcerer's Stone"
               |-----"filmStudio": "Warner Brother's"

I followed this algolia docs tutorial to create my app.js file

Using the example code from your docs, where would I exclude the userReviews, filmTitle, and filmStudio at?

Here is the app.js file:

 // Firebase-Admin Initialization
const admin = require("firebase-admin");
admin.initializeApp({
  credential: ...
  }),
  databaseURL: process.env.DATABASE_URL,
  databaseAuthVariableOverride: { uid: "my-service-worker" }
});

var db = admin.database();
var ref = db.ref('books');
ref.once("value", function(snapshot) {
  console.log(snapshot.val());
});

 // Firebase Initialization
const firebase = require('firebase');
var config = {
   apiKey: ...
};
firebase.initializeApp(config);

// Algolia Initialization
var algoliasearch = require('algoliasearch');
var client = algoliasearch(process.env.ALGOLIA_APP_ID, process.env.ALGOLIA_API_KEY);
var index = client.initIndex(process.env.INDEX_NAME);

const booksRef = firebase.database().ref(process.env.INDEX_NAME);

booksRef.on('child_added', addOrUpdateIndexRecord);
booksRef.on('child_changed', addOrUpdateIndexRecord);
booksRef.on('child_removed', deleteIndexRecord);

function addOrUpdateIndexRecord(contact) {
    // Get Firebase object
    const record = contact.val();
    // Specify Algolia's objectID using the Firebase object key
    record.objectID = contact.key;
    // Add or update object
    index
      .saveObject(record)
      .then(() => {
        console.log('Firebase object indexed in Algolia', record.objectID);
      })
      .catch(error => {
        console.error('Error when indexing contact into Algolia', error);
        process.exit(1);
      });
}

function deleteIndexRecord(contact) {
    // Get Algolia's objectID from the Firebase object key
    const objectID = contact.key;
    // Remove the object from Algolia
    index
      .deleteObject(objectID)
      .then(() => {
        console.log('Firebase object deleted from Algolia', objectID);
      })
      .catch(error => {
        console.error('Error when deleting contact from Algolia', error);
        process.exit(1);
      });
}

I found something to get me on the correct track. This seems like it would exclude the fields I don’t want to index but I’m not clear of the benefits of using restrictSearchableAttributes since I’m also using unretrievableAttributes at the same time.

I tried this and it’s not working.

index.setSettings({
    searchableAttributes: ['title', 'author','isbn', 'publisher'],
    attributesToRetrieve: ['title', 'author','isbn', 'publisher'],
    restrictSearchableAttributes: ['title', 'author', 'isbn', 'publisher'],
    attributesForFaceting:  ['title', 'author', 'isbn', 'publisher'],
    unretrievableAttributes: ['userReviews', 'filmTitle', 'filmStudio']
  });

Hi,

unretrievableAttributes won’t help reduce the size of your record since those fields are still in the index. See https://www.algolia.com/doc/api-reference/api-parameters/unretrievableAttributes/.

In your case, it seems that your records contain fields that you have no use for. You’d have to strip those from your records before saving them to your index.

From your specific code sample it looks like you’d have to start from your record variable and either strip unwanted fields or just create a new variable that only contains the fields you want before calling saveObject()

e.g:

    const record = contact.val();
    record.objectID = contact.key;
    delete record.userReviews;
    index.saveObject(record)...

Hi

In the addOrUpdateIndexRecord function you should modify record to only use the fields that you want to upload to Algolia. Something like

function addOrUpdateIndexRecord(contact) {
    // Get Firebase object
    const contactValue = contact.val();
    // Restrict what is send to Algolia
    const recordToSend = {
      objectID: contactValue.key,
      attribute: contactValue.attribute,
      otherAttribute: contactValue.otherAttribute,
    };
    // Add or update object
    index
      .saveObject(recordToSend)
      .then(() => {
        console.log('Firebase object indexed in Algolia', record.objectID);
      })
      .catch(error => {
        console.error('Error when indexing contact into Algolia', error);
        process.exit(1);
      });
}

hey thanks for the help. Actually this did work. I kept trying, I realized I made a typographical error. Thanks!!! This is perfect.

I would still like to know why @Bobylito’s answer didn’t work though.

@Bobylito thanks for the assistance. I tried the code but I keep getting an error response of:

Firebase object indexed in Algolia undefined

Here’s what I used:

function addOrUpdateIndexRecord(contact) {
  
  // Get Firebase object
  const contactValue = contact.val();

  // Restrict what is sent to Algolia
  const recordTosend = {
    objectID: contactValue.key,
    title: contactValue.title,
    author: contactValue.author,
    isbn: contactValue.isbn,
    publisher: contactValue.publisher
  };

  index
    .saveObject(recordTosend) // I also tried replacing this with contactValue but that just indexed everything including the unwanted data again
    .then(() => {
      console.log('Firebase object indexed in Algolia', recordTosend.objectID);
    })
    .catch(error => {
      console.error('Error when indexing contact into Algolia', error);
      process.exit(1);
    });
  }
}

Btw nice name. My son is a rapper and his name is Lanctio :slight_smile:

Is that an error of just the log message from:

    .then(() => {
      console.log('Firebase object indexed in Algolia', recordTosend.objectID);
    })

Which is actually the success path. undefined may not be problematic also as it just means that contactValue.key is undefined.

Btw nice name. My son is a rapper and his name is Lanctio :slight_smile:

Thanks, I wish I was a musician :slight_smile: