How should I test experimental code that changes often?

At work we'll have some piece of mathematical code and run it with some input and examine the output. Sometimes, we realize that the output doesn't make sense with the input we gave it and we have to go back and debug some simple (probably non-mathematical) mistake in the code. This seems like a decent job for unit testing, since we could write relatively straight forward tests for the non-math parts of the code. The problem is that we also often realize that the result isn't quite what we wanted and we'll change a lot of the code, meaning the unit tests would also have to change. Having the change a unit test every time we change the code isn't particularly realistic, so how can we go about testing code that changes often like this?