lundi 30 septembre 2019

MiniDFSCluster: can't get accessRight of self created directories

I would like to test my code evoking an org.apache.hadoop.hdfs.MiniDFSCluster class example. However at the time I call an example of it I get an error informing me that some file is not to be accessed by the process. I followed a stack trace and during debugging I could understand that I cannot get access to folders named "name1", "name2", those that MiniDFSCluster actually had created at the base directory during instantiating.

I'm trying to run a Scala testSuite test file containing a class invocation from Intellij IDEA 2019.2, Windows 7. Some time before I had the same problem as described at Hadoop Mini Cluster Mock (MiniDFSCluster), but fixed it and faced this one left by last user. I saw there are some more users facing the same problem, but could not extract correct solution: error using miniDFSCluster on windows. Stack trace returned to me is very similar to MiniDFSCluster UnsatisfiedLinkError org.apache.hadoop.io.nativeio.NativeIO$Windows.access0

Also I found some guide advising just to fix locally a method which throws an exception, but it seems awkward. https://www.cs.helsinki.fi/u/jilu/paper/hadoop_on_win.pdf

Here is my test:

val HADOOP_HOME="C:\\Local\\hadoop-common-2.2.0-bin-master"
val conf = new Configuration()
conf.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR, HADOOP_HOME)
System.setProperty("hadoop.home.dir", HADOOP_HOME)

val builder = new Builder(conf)
builder.build()

What I get as a result is following stack trace:

java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:557)
at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:490)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:308)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)

I lust for using this tool to test my code, so please, any info can be provided and any help would be appreciated.

If you are to help with the case - just try instantiate a class in Windows environment and tell me what it will return.

Aucun commentaire:

Enregistrer un commentaire