It seems like the crawler is not able to analyze the Shadow root content of web components on a page for URLs. Based on the this discussion thread.
It would be helpful for the crawler to be able to index inside of shadow roots on a page running with renderJavaScript: true so that Web Component based sites can be indexed properly.