Should I inject a HashSet for storing listeners to achieve testability?

I've seen this thread which may look like a duplicate but it doesn't explain two problems that I face: testability and mutability.

I have a class that implements the observable pattern. To make it simple, let's imagine it like this:

public class Observable {
    private final Set<Listener> listeners;

    public Observable(final Set<Listener> listeners) {
        this.listeners = listeners;
    }

    public Observable() {
        listeners = new HashSet<>();
    }

    public void registerListener(Listener listener) {
        listeners.add(listener);
    }

    public void unregisterListener(Listener listener) {
        listeners.remove(listener);
    }
}

I created two constructors to show two options I am trying to choose from. I want to test this implementation and I cannot achieve it if I don't inject a set for my listeners from the outside. On the other hand, if I inject it I make it possible to mutate this object elsewhere and my implementation is not that safe.

Is there a way to achieve both testability and safety in this scenario?