Android espresso test setPortraitOrientation stuck

I am testing some scenarios where I rotate the screen to Landscape mode and then brings back to Portrait mode but in my tests once I turn the screen to Landscape it never turn back to Portrait and the test framework is stuck and never fails. Seems like there is some activity going on. Used mainActivity in the test setup

activity.setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE);
activity.setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_PORTRAIT);