Hi i was wondering if it is possible to manipulate processing time in unit-tests to allow testing for late data in Spark Structured Streaming.
I am inserting data as sequence into a MemoryStream in form of: case class LineWithTimestamp(line: String, timestamp: Timestamp). When inserting LineWithTimestamp("a b b a", from(now.plus(ofSeconds(4)))) i want this event to be a few seconds later, so i can test allowed lateness/watermarks.
Thanks in advance.
Aucun commentaire:
Enregistrer un commentaire