How to isolate fixtures to specific Rails tests

How can I isolate the usage of fixtures to specific tests?

In my setup some of my tests rely on fixture data and some don't, so the default of loading all my fixtures in test_helper.rb with fixtures :all breaks my tests.

Example integration test that requires empty behaviorists table:

require 'test_helper'

class WelcomeFlowTest < ActionDispatch::IntegrationTest
  test "when no user is found start welcome flow" do
    get "/"
    follow_redirect!
    assert_response :success

    post "/setup", {
      behaviorist: { name: "Andy", email: "andy.bettisworth@accreu.com" },
      habit: { name: "Interval running", on_monday: false, on_tuesday: true, \
               on_wednesday: false, on_thursday: true, on_friday: false, \
               on_saturday: true, on_sunday: false }
    }
    assert_response :success
    assert_equal 1, Behaviorist.count
    assert_equal 1, Habit.count
  end
end

My unit test that requires behaviorist fixtures:

require 'test_helper'

class BehavioristTest < ActiveSupport::TestCase
  test "validates uniqueness of :name" do
    andy = Behaviorist.new(name: "Andy", remote_ip: "127.0.0.1")
    assert_not run.valid?
    assert_match /has already been taken/, andy.errors[:name].join
  end
end