How to sort by array

If i have a bunch of objects, some with an array of integers, how can I sort by them?

For example, I have this data set

{ name: "Jeff", arrayToSortBy: [1,2,3,4,5] }
{ name: "Allen", arrayToSortBy: [3] }
{ name: "Linda" }

And I want to sort by, arrayToSortBy, where looks at either the lowest value for asc or highest value for desc. In this case, Jeff would come up on top for both. But if I changed Allen to have a 0, then he would come up first for ascending.

Is there any way to do this right now?