Should I unit test my /src files or /build files?

When setting up unit tests (in my case, using Jasmine for JavaScript) should the un-minified/in-uglified src files be tested? Or should the end-user build files (minified and uglified) be tested?

In my grunt config:

jasmine: {
    test: {
        src: ['src/file.js'] /* or ['build/file.min.js'] */
    }
}

On one hand, it's nice to test the src files since it doesn't remove debuggers and is easier to inspect when needed.

On the other hand, I can test the src files as much as I like but it's not true to what the end users will be running, as the build files are uglified and minified.