Unit Deviance on test set R

I am evaluating a set of predictions coming from different models against a set of actual values in a regression problem.
I do not want to use the mean squared error as my evaluation metric because my values are not normally distributed. I would like to actually use the deviance which extends this concept.
For example, the poisson deviance is defined as 2(y * log(y/mu) - y + mu) where y are the actual values and the predicted ones.
In R, there is the deviance function, however this is applied to a glm object and therefore it will be calculated on the training set. Moreover I am not necessarily dealing with GLMs only. Do I need to define these functions manually for each distribution I need, or, these functions are already coded in some libraries?

Thank you