Strategy for test pipeline with .NetCore and KAFKA

After having set up a cloud service prototype based on Kafka and .NetCore applications (webapi as producer and console as consumers), I am designing the profile of my continuous integration and testing environment. Starting from scratch, I am interested to not reinvent the wheel, and use state-of-the-art tools for that. What are the major pre-requesite I have to work with? I speaking about tools, frameworks...

I use open-source applications, so Jenkins sounds like a "must use". But how to deal with the global environment? Do I need to have a always running kafka cluster to support continous testing jobs?

My thoughts were at this stage:

• Always running KAFKA and DB

• jobs

  • installing a test DataSet test DB
  • checking out the latest consumer component to test
  • building the related package into docker
  • starting the docker image in the test environment
  • starting producer/consumer test scripts (I would have run python? other best way to do?) to make the component under test live and verify it does the job, measuring performances and so on...

Am I missing something here? Is there any better way to do? Is there another way to test consumer/producer IO without a running Kafka cluster?

Thanks,