Selenium: Close button doesn't throw error but popup still doesn't close

I have a Selenium test with a create-new-folder pop-up which I want to close. At first I thought the button was not visible or not clickable or something like that, but that doesn't seem to be the problem. I checked for all of that. So, the actually click action goes through without error, but the click still doesn't seem to work as the pop-up remains afterwards.

WebElement button = Utils.DRIVER.findElement(By.xpath("//input[@class='lotusFormButton' and @value='Cancel' and not(@disabled)]")); wait.until(ExpectedConditions.elementToBeClickable(button)); new Actions(Utils.DRIVER).moveToElement(button).click().perform(); wait.until(ExpectedConditions.invisibilityOfElementLocated(By.className("dijitDialogUnderlay")));

I have no idea what I did wrong, I tried regular clicks

button.click();

I tried javascript clicks

JavascriptExecutor executor = (JavascriptExecutor) Utils.DRIVER; executor.executeScript("arguments[0].click();", button);

but none of it worked. Any help would be much appreciated.