Writing tests with testing-library depends on implementation details of my UI library

I am using @testing-library to test my React project which uses Material-UI as it's main UI library. As @testing-library suggests, it's main philosophy is to test the application in the exact same way a user would use it. e.g click the button which says Submit or edit the input whose label is Username, etc... . And the testing-library gives me some helper methods like getByText, getByLabel, etc... .

My problem is that for some tests, forms' input tests for instance, I need to query an input's value or change it's current value. I need to getByLabel to query my label and go to parent component to query the input that is contained within. The library provides a closest DOM query helper, but is of no use in some situations, like the one I just described.

Writing my tests this way feels good, I mean writing tests as a user's point of view. But the problem is that I depend too much on implementation details of Material-UI. Maybe in next upgrade, the inputs and their corresponding label are not children of ancestor nodes, etc... .

Is there something I am doing wrong? Or is it the pitfall of the testing philosophy that @testing-library provides?

This is my packages stack:

{
    "@material-ui/core": "^4.7.1",
    "@material-ui/icons": "^4.5.1",
    "@material-ui/lab": "^4.0.0-alpha.34",
    "@testing-library/jest-dom": "^4.2.4",
    "@testing-library/react": "^9.4.0",
    "react": "^16.11.0",
    "react-dom": "^16.11.0",
    "react-scripts": "3.3.0-next.62"
}