Tests that don't include spec_helper being ignored in rspec runs

When testing lib code there is rarely any need to require spec_helper and load all of rails. This is why I have been removing require "spec_helper" in favour of require_relative "../../lib/my_lib.rb".

These tests pass when called directly (rspec spec/lib/my_lib.rb) and are blazingly fast. Winner.

My issue comes when I try and run these tests as a group.

When I call rspec spec/lib it runs any lib specs that have a require "spec_helper" line but not any tests that don't.

I have played with spec_helper.rb to load in these tests, and that kind of works only it means that when I run rspec spec/models/blah.rb it will also run these lib tests, which obviously isn't what I want.

Is there a different way I should be calling my tests? Or is there a way I can get them added to the test run?

Notes

My spec_helper is configured to run the tests in a random order, I wonder if this has anything to do with it?