lundi 23 mars 2020

Spark Structured Streaming test late data

Hi i was wondering if it is possible to manipulate processing time in unit-tests to allow testing for late data in Spark Structured Streaming.

I am inserting data as sequence into a MemoryStream in form of: case class LineWithTimestamp(line: String, timestamp: Timestamp). When inserting LineWithTimestamp("a b b a", from(now.plus(ofSeconds(4)))) i want this event to be a few seconds later, so i can test allowed lateness/watermarks.

Thanks in advance.

Aucun commentaire:

Enregistrer un commentaire