I want to test a function f(dstream: DStream[Int])
.
For this, I found the library StreamingSuiteBase
@https://mvnrepository.com/artifact/com.holdenkarau/spark-testing-base
.
But the following code:
class DStreamClassTestSuite extends FunSuite with StreamingSuiteBase {
test("test f(DStream)") {
assert(true)
}
throws:
java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.hdfs.DistributedFileSystem could not be instantiated
.
All the articles showcasing the use if the library date back from 2016.
The spark
version at the time was 1.6
. I am using 2.2
.
I guess it is compatibility error between the version of the library and the version of spark.
Did anyone find a workaround this error? If no, how can I test my function that tale a DStream[T]
?
Aucun commentaire:
Enregistrer un commentaire