Fake timers don't trigger setTimeout call correctly in Jest

I have a simple React UI-less component that shows status of a passed action. There are 4 states - pending, running, finished, failed. The main catch here is that running and finished states have to last at least some time (for example 2 seconds). I'm trying to test this behavior using Jest (26.6.3) and its fake timers, but I can't get it working.

Component code:

export const TaskState = {
    PENDING: "pending",
    RUNNING: "running",
    FINISHED: "finished",
    FAILED: "failed",
};

export default function Task({ action, minDelay = 2000, children }) {
    const [state, setState] = useState(TaskState.PENDING);
    const [error, setError] = useState(null);
    const timeoutHandle = useRef(null);

    useEffect(() => {
        return () => {
            if (timeoutHandle.current) {
                clearTimeout(timeoutHandle.current);
            }
        };
    }, []);

    const resetState = () => {
        setState(TaskState.PENDING);
        setError(null);
    };

    const onSuccess = () => {
        setState(TaskState.FINISHED);
        timeoutHandle.current = setTimeout(resetState, minDelay);
    };

    const onError = errMsg => {
        setState(TaskState.FAILED);
        setError(errMsg);
    };

    const startAction = () => {
        setState(TaskState.RUNNING);
        const start = performance.now();
        let err = null;
        try {
            action();
        } catch (e) {
            err = e.message;
        } finally {
            const end = performance.now();
            const elapsedTime = end - start;
            const delayTime = minDelay - elapsedTime;
            if (elapsedTime < minDelay && minDelay > 0) {
                timeoutHandle.current = setTimeout(() => {
                    err ? onError(err) : onSuccess();
                }, delayTime);
            } else {
                err ? onError(err) : onSuccess();
            }
        }
    };

    return children({
        state,
        error,
        startAction,
    });
}
Task.propTypes = {
    action: PropTypes.func.isRequired,
    minDelay: PropTypes.number,
    children: PropTypes.func.isRequired,
};

Problematic testcase:

// sample UI implementation of our UI-less Task component
const renderBasicButton = ({ state, error, startAction }) => {
    return (
        <button onClick={startAction}>
            <span data-testid="state">{state}</span>
            <span data-testid="error">{error}</span>
        </button>
    );
};

test("changes state to failed after delay when clicked and action throws error", () => {
    jest.useFakeTimers("modern");
    const minDelay = 1000;
    render(
        <Task
            minDelay={minDelay}
            action={() => {
                throw new Error("Test error");
            }}
        >
            {props => renderBasicButton(props)}
        </Task>
    );
    userEvent.click(screen.getByRole("button"));
    expect(screen.getByTestId("state")).toHaveTextContent(TaskState.RUNNING);
    expect(screen.getByTestId("error")).toHaveTextContent("");
    act(() => jest.advanceTimersByTime(minDelay * 2));
    screen.debug(); // state is still TaskState.RUNNING instead of TaskState.FAILED
    expect(screen.getByTestId("state")).toHaveTextContent(TaskState.FAILED);
    expect(screen.getByTestId("error")).toHaveTextContent("Test error");
    jest.useRealTimers();
});

Action callback is called, state transition from pending to running is performed, but next state transition from running to failed is not performed. It works when I test it manually.

Minimal example - https://codesandbox.io/s/heuristic-khorana-7vygv?file=/src/App.js