Can't upload JSON. What is 'extended quota'?

Hi,

I’m new to Algolia and I’d like to evaluate the free community tier. I just created my first index and I’m trying to upload a 10MB JSON file, which contains about 3500 records in valid JSON format, but to no avail. I keep getingt the following message:

{
“message”: “Record at the position 0 is too big size=9740073 bytes. Contact us if you need an extended quota”,
“position”: 0,
“status”: 400
}

Can someone please explain what this means? How can I best upload this JSON file using the free community tier?

Cheers,

Peter

Yes, records are not expected to go beyond 10KB of minified JSON. You’ll get the Record is too big error if you try to index a record bigger than the limit.

You may find more information here 1

1 Like

@geopetrus4 Indeed, when indexing records you should make sure your records are not too big and exceeding the maximum record size, an example of bad practice is storing long strings of text as values on your record, such as maybe a base64 encoded image or a very long string of text.

Let me know how it goes and if you have more issues we’ll be glad to help you get setup :slight_smile:

Hi Jonas,

Do you mean that the entire JSON file must be less than 10KB or each record in the JSON file must be less than 10KB?

I stripped out all of the non-essential fields in each of the ~3500 records in my JSON file, and reduced it from 10,233KB to 5,388KB. I still get the same error message, but now with a smaller number:

Record at the position 0 is too big size=5078267 bytes.

Each of my records is less than 10k, so is your system viewing my JSON (which I’ve validated) as a single record?

Cheers,

Peter

@geopetrus4 a record is a single entry inside your JSON and not the complete file. Can you check that your JSON schema is valid and make sure that you are importing the records as an Array type? See below a valid schema

  [
    {
	   "name": "Record_1"
    },
    {
       "name": "Record_2"
    }
  ]

Let me know if that helps :slight_smile:

Hi Jonas,

My JSON is valid according to https://jsonformatter.curiousconcept.com/, and a subset of it (3 records) looks as such:

{
“repositories”:{
“repository”:[
{
@rID”:926,
“rName”:{
":"Landsp\u00edtali University Hospital Research Archive" }, "rAcronym":{ "”:“Hirsla”
},
“rNamePreferred”:{
":"N" }, "rUrl":{ "”:“http://www.hirsla.lsh.is/
},
“rOaiBaseUrl”:{
“$”:“http://www.hirsla.lsh.is/lsh-oai/request
},
“uName”:{

        },
        "uAcronym":{

        },
        "uNamePreferred":{

        },
        "uUrl":{

        },
        "oName":{
           "$":"Landsp\u00edtali University Hospital"
        },
        "oAcronym":{

        },
        "oNamePreferred":{
           "$":"Y"
        },
        "oUrl":{
           "$":"http://www.landspitali.is/"
        },
        "postalAddress":{

        },
        "country":{
           "cIsoCode":{
              "$":"IS"
           },
           "cCountry":{
              "$":"Iceland"
           }
        },
        "paLatitude":{
           "$":64.1353
        },
        "paLongitude":{
           "$":-21.8952
        },
        "rDescription":{
           "$":"This site is an institutional repository for materials produced by the scholars working at the University Hospital. It is notable that the vast majority of materials listed on this site are metadata (bibliographic) only, and that full-text items are a small subset of the whole. The site is well supported with guidance material for users. The interface is in English. An RSS feed is available."
        },
        "rRemarks":{

        },
        "rNumOfItems":{
           "$":6040
        },
        "rDateHarvested":{
           "$":"2017-05-18"
        },
        "rYearEstablished":{

        },
        "repositoryType":{
           "$":"Institutional"
        },
        "operationalStatus":{
           "$":"Operational"
        },
        "rSoftWareName":{
           "$":"Open Repository"
        },
        "rSoftWareVersion":{

        },
        "classes":{
           "class":{
              "clCode":{
                 "$":"Ce"
              },
              "clTitle":{
                 "$":"Health and Medicine"
              }
           }
        },
        "languages":{
           "language":[
              {
                 "lIsoCode":{
                    "$":"en"
                 },
                 "lName":{
                    "$":"English"
                 }
              },
              {
                 "lIsoCode":{
                    "$":"is"
                 },
                 "lName":{
                    "$":"Icelandic"
                 }
              }
           ]
        },
        "contentTypes":{
           "contentType":[
              {
                 "@ctID":1,
                 "$":"Journal articles"
              },
              {
                 "@ctID":4,
                 "$":"Bibliographic references"
              },
              {
                 "@ctID":7,
                 "$":"Unpublished reports and working papers"
              }
           ]
        }
     },
     {
        "@rID":3712,
        "rName":{
           "$":"Opin visindi"
        },
        "rAcronym":{

        },
        "rNamePreferred":{
           "$":"Y"
        },
        "rUrl":{
           "$":"https://opinvisindi.is/"
        },
        "rOaiBaseUrl":{
           "$":"https://opinvisindi.is/oai/request"
        },
        "uName":{

        },
        "uAcronym":{

        },
        "uNamePreferred":{

        },
        "uUrl":{

        },
        "oName":{
           "$":"National and University Library of Iceland"
        },
        "oAcronym":{

        },
        "oNamePreferred":{
           "$":"Y"
        },
        "oUrl":{
           "$":"http://www.landsbokasafn.is/"
        },
        "postalAddress":{
           "$":"Reykjavik"
        },
        "country":{
           "cIsoCode":{
              "$":"IS"
           },
           "cCountry":{
              "$":"Iceland"
           }
        },
        "paLatitude":{
           "$":64.14102
        },
        "paLongitude":{
           "$":-21.95274
        },
        "rDescription":{
           "$":"This site provides access to the research output of the Agricultural University of Iceland, the Iceland Academy of the Arts, Tte National and University Library, the University of Akureyri, the University of Bifr\u00f6st, the University of Iceland, H\u00f3lar University College and the University of Reykjavik. The interface is available in Icelandic and English."
        },
        "rRemarks":{
           "$":"The national and University Library of Iceland comprises of the national library of Iceland and the library of the University of Iceland."
        },
        "rNumOfItems":{
           "$":150
        },
        "rDateHarvested":{
           "$":"2017-05-18"
        },
        "rYearEstablished":{

        },
        "repositoryType":{
           "$":"Institutional"
        },
        "operationalStatus":{
           "$":"Operational"
        },
        "rSoftWareName":{
           "$":"DSpace"
        },
        "rSoftWareVersion":{
           "$":5.4
        },
        "classes":{
           "class":{
              "clCode":{
                 "$":"C"
              },
              "clTitle":{
                 "$":"Multidisciplinary"
              }
           }
        },
        "languages":{
           "language":[
              {
                 "lIsoCode":{
                    "$":"is"
                 },
                 "lName":{
                    "$":"Icelandic"
                 }
              },
              {
                 "lIsoCode":{
                    "$":"en"
                 },
                 "lName":{
                    "$":"English"
                 }
              }
           ]
        },
        "contentTypes":{
           "contentType":{
              "@ctID":1,
              "$":"Journal articles"
           }
        }
     },
     {
        "@rID":1031,
        "rName":{
           "$":"Skemman"
        },
        "rAcronym":{

        },
        "rNamePreferred":{
           "$":"Y"
        },
        "rUrl":{
           "$":"http://skemman.is/"
        },
        "rOaiBaseUrl":{

        },
        "uName":{

        },
        "uAcronym":{

        },
        "uNamePreferred":{
           "$":"Y"
        },
        "uUrl":{

        },
        "oName":{
           "$":"National and University Library of Iceland"
        },
        "oAcronym":{

        },
        "oNamePreferred":{
           "$":"Y"
        },
        "oUrl":{
           "$":"http://www.landsbokasafn.is/"
        },
        "postalAddress":{
           "$":"Reykjavik"
        },
        "country":{
           "cIsoCode":{
              "$":"IS"
           },
           "cCountry":{
              "$":"Iceland"
           }
        },
        "paLatitude":{
           "$":64.14102
        },
        "paLongitude":{
           "$":-21.95274
        },
        "rDescription":{
           "$":"This is a consortium repository for Icelandic institutions. Providing access to the research output of both institutions. The site interface is available in Icelandic and English. Some items are only available to registered users only."
        },
        "rRemarks":{
           "$":"This site is the collective repository for The Agricultural University of Iceland, The Iceland Academy of the Arts, The National and University Library, The University of Akureyri, The University of Bifr\u00f6st, The University of Iceland, H\u00f3lar University College and The University of Reykjavik."
        },
        "rNumOfItems":{
           "$":26651
        },
        "rDateHarvested":{
           "$":"2017-05-18"
        },
        "rYearEstablished":{

        },
        "repositoryType":{
           "$":"Institutional"
        },
        "operationalStatus":{
           "$":"Operational"
        },
        "rSoftWareName":{
           "$":"DSpace"
        },
        "rSoftWareVersion":{

        },
        "classes":{
           "class":{
              "clCode":{
                 "$":"C"
              },
              "clTitle":{
                 "$":"Multidisciplinary"
              }
           }
        },
        "languages":{
           "language":[
              {
                 "lIsoCode":{
                    "$":"is"
                 },
                 "lName":{
                    "$":"Icelandic"
                 }
              },
              {
                 "lIsoCode":{
                    "$":"en"
                 },
                 "lName":{
                    "$":"English"
                 }
              }
           ]
        },
        "contentTypes":{
           "contentType":[
              {
                 "@ctID":1,
                 "$":"Journal articles"
              },
              {
                 "@ctID":5,
                 "$":"Conference and workshop papers"
              },
              {
                 "@ctID":6,
                 "$":"Theses and dissertations"
              },
              {
                 "@ctID":8,
                 "$":"Books, chapters and sections"
              }
           ]
        }
     }
  ]

}
}

Peter

I should have mentioned that I constructed this JSON structure by calling an API in Python that returns XML. I then converted that XML to JSON using BadgerFish.

Hey Peter, it seems that your json schema is indeed valid, but not of correct type - Array. You need to be specifying an array of objects and not an object containing an array as property and that’s why your record size is being rejected.

In other words, your json file should start with an array bracket syntax - [ insead of { and contain records(individual objects) inside it.