Ignore specific facets for certain products

I’m working on a site which has Brand and Material as facet filter. I want to have some “universal” products that will be filtered by user’s search query or Brand facet, but always show disregard the Color facet selected. One way to do that is include all Materials in those product records but we have hundreds of thousands Materials so doing that will exceed the record size limit.

What’s correct way to achieve it?