jeudi 5 décembre 2019

Testing on DStream throws java.util.ServiceConfigurationError

I want to test a function f(dstream: DStream[Int]).

For this, I found the library StreamingSuiteBase @https://mvnrepository.com/artifact/com.holdenkarau/spark-testing-base.

But the following code:

class DStreamClassTestSuite extends FunSuite with StreamingSuiteBase {

  test("test f(DStream)") {

    assert(true)
}

throws:

java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.hdfs.DistributedFileSystem could not be instantiated

.

All the articles showcasing the use if the library date back from 2016.

The spark version at the time was 1.6. I am using 2.2.

I guess it is compatibility error between the version of the library and the version of spark.

Did anyone find a workaround this error? If no, how can I test my function that tale a DStream[T]?

Aucun commentaire:

Enregistrer un commentaire