mercredi 30 mai 2018

How to mock a FileWriter in spark structured streaming unit tests?

I'm trying to write unit tests for a spark structured streaming app. I'm reading some data from a stream and storing it in RDS. To do this I'm using the ForeachWriter:

ds.writeStream
  .queryName("query")
  .foreach(writer)
  .option("checkpointLocation", CHECKPOINT)
  .outputMode("append")
  .start()

Is there a way to mock it in order to not write the data in the database?

Aucun commentaire:

Enregistrer un commentaire