NoMatchingViewException when closing a dialog with Espresso

I got a test case where I open a date picker dialog, selects a date an press "Okay". After that, it must fill out a field, but like every 3rd time, I run the test, the dialog stays open and I get NoMatchingViewException on the field. ~2/3 test runs succeeds. I can get by the problem by using Thread.sleep, but I try to avoid it.

I've also tried IdlingResource in this way with no luck:

onView(withId(R.id.date_picker_dialog_button)).perform(scrollTo(), click());
IdlingResource idlingResource = new DialogFragmentIdlingResource( mActivityRule.getActivity().getFragmentManager(), "datePickerDialog");
onView(withText("OK")).check (matches(isDisplayed())).perform(click());
Espresso.unregisterIdlingResources(idlingResource);    
onView(allOf(withId(R.id.field_to_write))).check (matches(isDisplayed())).perform(typeText("text"));

With this class:

public class DialogFragmentIdlingResource implements IdlingResource {
private final FragmentManager manager;
private final String tag;
private ResourceCallback resourceCallback;

public DialogFragmentIdlingResource(FragmentManager manager, String tag) {
    this.manager = manager;
    this.tag = tag;
}

@Override
public String getName() {
    return DialogFragmentIdlingResource.class.getName() + ":" + tag;
}

@Override
public boolean isIdleNow() {
    boolean idle = (manager.findFragmentByTag(tag) == null);
    if (idle) {
        resourceCallback.onTransitionToIdle();
    }
    return idle;
}

@Override
public void registerIdleTransitionCallback(ResourceCallback resourceCallback) {
    this.resourceCallback = resourceCallback;
}
}