Should I avoid hitting the database on tests as a shortcut to fill object with valid data?

I have a test suite which works against a dedicated copy of the real database. In genereal I try to avoid accessing the DB when doing unit testing especially because unit tests should run fast (I don't care so much if it tests more than one class/method because my tests became more of a functional tests after becoming acquainted with Ian Coopers approach on TDD).

However I have a code that creates a complex object by filling it from the database and it would be a lot of work do create one "manually" from code or mocking one to bring it to a valid state.

I was trying to bring it once from the database and then just do a deep clone of it and work from memory instead of from DB whenever a tests requires this object (in order to optimize performance).

However I've noticed that the first time I run the query against the db takes less than 100 ms and the following times it takes 0 ms (probably because the database/c# has some cache or something) I don't even experience a penalty for the round trips to the DB (which sits on the same network but not on my local machine).

So was my assumption that I should avoid hitting the database whenever possible on my unit tests wrong? I mean on cases when I have a DB on the same network and not actually testing the DB, just using it as a shortcut to fill objects with valid data?