Help assigning variables to subpaths

Open Q&A
, ,
#1

Hello,

I’m using the docsearch scraper to index my site. I’m trying to use variables to create facet filters. It mostly works, except when a variable is a subpath. I’ve pasted my config code below:

  {
      "url": "https://xxx.pages.xxx.io/microservices/",
        "tags": [
            "Microservices"
        ]
    },
    {
      "url": "https://xxx.pages.xxx.io/microservices/(?P<content_type>.*?)/",
        "variables": {
            "content_type": ["blog"]
        }
    },
    {
      "url": "https://xxx.pages.xxx.io/microservices/docs/graphql/(?P<content_type>.*?)/graphql-faq/",
        "variables": {
            "content_type": ["faq"]
        }
    },
    {
      "url": "https://xxx.pages.xxx.io/microservices/(?P<content_type>.*?)/",
        "variables": {
            "content_type": ["docs"]
        }
    }

In this case, I can create a facet filter for blog and docs, but not one for faq which is a subpath of docs. When I remove the docs variable, of course faq works. When I switch the order, nothing changes.

I get why it’s happening, I’m just trying to understand if what I’m trying to do is possible.

Thanks in advance!

#2

Bumping this up.

I’m wondering if regex isn’t supported beyond capture groups for variables? This is where I’m at currently:

{
  "url": "https://XXX.pages.XXX.io/microservices/(?P<content_type>.*?)\/(?!.*faq).*/",
    "variables": {
        "content_type": ["docs"]
    }
}

Seems like my negative lookahead isn’t supported?

The docs have an example of these subpaths for selectors, but it looks like they’re not supported for variables?