Android Espresso: Flaky tests due to random ViewNotFoundException

I have a few Espresso tests that fail randomly when trying to click a button. The tests fail with a ViewNotFoundException. It appears that the Activity state during this exception is one step behind the expected state.

Note that inserting a Thread.sleep() does not solve the issue.

What are possible workarounds to fix those tests?