Espresso: How to test SwipeRefreshLayout?

My app reloads data when a down-swipe is done on a SwipeRefreshLayout. Now I try to test this with Android Test Kit / Espresso like this:

onView(withId(R.id.my_refresh_layout)).perform(swipeDown());

Unfortunately this fails with

android.support.test.espresso.PerformException: Error performing 'fast swipe'
on view 'with id: my.app.package:id/my_refresh_layout'.
...
Caused by: java.lang.RuntimeException: Action will not be performed because the target view
does not match one or more of the following constraints:
at least 90 percent of the view's area is displayed to the user.
Target view: "SwipeRefreshLayout{id=2131689751, res-name=my_refresh_layout, visibility=VISIBLE,
width=480, height=672, has-focus=false, has-focusable=true, has-window-focus=true,
is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, 
is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0,
child-count=2}"

Of course, the layout is visible and swiping manually works, but I'm unsure if I'm doing something wrong? The layout spans the whole screen, so it should really be possible for Espresso to do some action on it.