Algolia Crawler unable to find any link or records

Open Q&A
,
1

Hello,
for the past few days I’ve been trying to integrate the Algolia Docsearch in our app. But so far I didn’t manage to get it to work properly. I believe that our pages are structured properly, and I tried my best to follow the guidelines provided by Algolia, but no results so far.
The entry point for our documentation is Elumity | Documentation
The Algolia Crawler config properties are set to default values, but I’ll provide that as well

new Crawler({
  rateLimit: 8,
  maxDepth: 10,
  maxUrls: 5000,
  startUrls: ["https://elumity.com/resources/documentation/"],
  renderJavaScript: false,
  sitemaps: [],
  ignoreCanonicalTo: false,
  discoveryPatterns: ["https://elumity.com/resources/documentation/**"],
  schedule: "at 13:51 on Friday",
  actions: [
    {
      indexName: "elumity",
      pathsToMatch: ["https://elumity.com/resources/documentation/**"],
      recordExtractor: ({ helpers }) => {
        return helpers.docsearch({
          recordProps: {
            lvl1: ["header h1", "article h1", "main h1", "h1", "head > title"],
            content: ["article p, article li", "main p, main li", "p, li"],
            lvl0: {
              selectors: "",
              defaultValue: "Documentation",
            },
            lvl2: ["article h2", "main h2", "h2"],
            lvl3: ["article h3", "main h3", "h3"],
            lvl4: ["article h4", "main h4", "h4"],
            lvl5: ["article h5", "main h5", "h5"],
            lvl6: ["article h6", "main h6", "h6"],
          },
          aggregateContent: true,
          recordVersion: "v3",
        });
      },
    },
  ],
  safetyChecks: { beforeIndexPublishing: { maxLostRecordsPercentage: 30 } },
  initialIndexSettings: {
    elumity: {
      attributesForFaceting: ["type", "lang"],
      attributesToRetrieve: [
        "hierarchy",
        "content",
        "anchor",
        "url",
        "url_without_anchor",
        "type",
      ],
      attributesToHighlight: ["hierarchy", "content"],
      attributesToSnippet: ["content:10"],
      camelCaseAttributes: ["hierarchy", "content"],
      searchableAttributes: [
        "unordered(hierarchy.lvl0)",
        "unordered(hierarchy.lvl1)",
        "unordered(hierarchy.lvl2)",
        "unordered(hierarchy.lvl3)",
        "unordered(hierarchy.lvl4)",
        "unordered(hierarchy.lvl5)",
        "unordered(hierarchy.lvl6)",
        "content",
      ],
      distinct: true,
      attributeForDistinct: "url",
      customRanking: [
        "desc(weight.pageRank)",
        "desc(weight.level)",
        "asc(weight.position)",
      ],
      ranking: [
        "words",
        "filters",
        "typo",
        "attribute",
        "proximity",
        "exact",
        "custom",
      ],
      highlightPreTag: '<span class="algolia-docsearch-suggestion--highlight">',
      highlightPostTag: "</span>",
      minWordSizefor1Typo: 3,
      minWordSizefor2Typos: 7,
      allowTyposOnNumericTokens: false,
      minProximity: 1,
      ignorePlurals: true,
      advancedSyntax: true,
      attributeCriteriaComputedByMinProximity: true,
      removeWordsIfNoResults: "allOptional",
    },
  },
  appId: "***",
  apiKey: "***",
});

When I test the entry URL via the URL Tester, it returns the correct HTML, but the number of links found is always 0. When I test any of the docs pages, there are no records extracted either.
I’m really not sure what I’ve missed and I would really appreciate any help or guidance.
Thanks in advance.

2

Update
I’ve noticed that we haven’t done any domain verification, I wanted to do it but for some reason I have the Domain Verification is disabled on application "APP_NAME". Contact support to have one added manually. message displayed. But the contact support button doesn’t do anything. I’ll try to add the robots.txt file and see how it goes.
In the table I see one entry
Domain: elumity.com, Added by: system, Verified: YES

3

Yep that didn’t help either. I don’t get any information back from the URL tester I don’t know where/what to look for. It’s always success but nothing on the page, no links no records extracted no nothing.

4

I’m not sure, but to me it seems that the recordExtractor function doesn’t even get called when I test a URL. For example in my config I have the following:

recordExtractor: ({ helpers }) => {
        console.log('HELLO');
        return helpers.docsearch({
          recordProps: {
            lvl1: ["header h1", "article h1", "main h1", "h1", "head > title"],
            content: ["article p, article li", "main p, main li", "p, li"],
            lvl0: {
              selectors: "",
              defaultValue: "Documentation",
            },
            lvl2: ["article h2", "main h2", "h2"],
            lvl3: ["article h3", "main h3", "h3"],
            lvl4: ["article h4", "main h4", "h4"],
            lvl5: ["article h5", "main h5", "h5"],
            lvl6: ["article h6", "main h6", "h6"],
          },
          aggregateContent: true,
          recordVersion: "v3",
        });

After the test is run, there are no logs, in the console i get Nothing was logged from recordExtractor

5

Even if I do the following:

recordExtractor: ({ url, $, contentLength, fileType }) => {
        console.log('STA GOD');
        return [
          {
              url: url.href,
              title: $("head > title").text(),
              description: $("meta[name=description]").attr("content"),
              type: $('meta[property="og:type"]').attr("content"),
          }
        ];
      },

I get No records extracted either :frowning: