I want to test a function f(dstream: DStream[Int]).
For this, I found the library StreamingSuiteBase @https://mvnrepository.com/artifact/com.holdenkarau/spark-testing-base.
But the following code:
class DStreamClassTestSuite extends FunSuite with StreamingSuiteBase {
test("test f(DStream)") {
assert(true)
}
throws:
java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.hdfs.DistributedFileSystem could not be instantiated
.
All the articles showcasing the use if the library date back from 2016.
The spark version at the time was 1.6. I am using 2.2.
I guess it is compatibility error between the version of the library and the version of spark.
Did anyone find a workaround this error? If no, how can I test my function that tale a DStream[T]?
Aucun commentaire:
Enregistrer un commentaire