Creating a generic UI in React that wraps UI library

I will give a real-life example to give an idea of the problem I am trying to solve.

My team was responsible for a React app that was built with Semantic UI as its UI library. Here is a list of tests that were written (note we have our own definition for test types)

• Unit Tests: test per component using Enzyme.
• System Tests: done mainly on container components. Enzyme and sinon and fetch-mock where used for mocking.
• End to end tests: Done with puppeteer.

Later on, our product managers wanted a Manerial UI based theme. To make it possible we need to

1. Replace semantic UI components with Material UI components
2. Write new test helpers for our system & unit. Then update all our tests to use the new test helpers
3. Write new helpers for out End to End tests.

It was a painful process to make it possible, and it is still partially done. I thought that one of our mistakes were not defining components that are more generic and wrap our UI logic. However, after reviewing component API of both Semantic UI and Material UI I m wondering if this was even possible. Since:

1. Each Library has different implementation of the component
2. Components have different attributes
3. HTML structure is different so End to End test still need to be changed when UI Library is swapped.

Now my question: Has anyone faced the same issues and how did you tackle this problem?