How to test methods with user defined functions in production?

I was asked a to solve a question in an interview, and the problem was such that the user could change the output of the function if we gave the option of passing a function as a parameter (in C++ this would be by passing a function pointer the preformed the desired operation). After, the interviewer asked me how would I test this in production. I was not completely sure of the answer, since I am interviewing for my first SWE job. My immediate solution was to have assertions that were dependent on the input of the user, or let the user be responsible of testing his own method.

My question is, how would you do it? Or where can I get information on the best practices for this?

In case my question wasn't clear, imagine you are writing the linux quicksort function (not the actual question but hopefully it will serve for illustration purposes):

void qsort(void *base, size_t nmemb, size_t size,
       int (*compar)(const void *, const void *));

How would you test for correctness? Since the user of the function is the one defining the (*compar) function, it seems to me that the user should take care of verifying correctness while using the function.