Recommendation for a test environment for proprietary network protocol

I have a system I want to test, the system handles a proprietary network protocol (this need to be checked). The protocol is over TCP or UDP.

I'm looking for a software I could install on my OS-X/Linux/Windows computer connect it to the system I want to test and inject protocol messages and analyse feedback (software should have the ability to compare for each received message if it is the expected one etc.).

Prefer a software with a nice looking GUI and open-source so I could add features if needed (could be a paid software).

Where should I begin?