Home:ALL Converter>Running Java Native libraries on Hadoop

Running Java Native libraries on Hadoop

Ask Time:2014-06-18T23:29:20         Author:Anonim33

Json Formatter

I am trying to run Gurobi optimizer on the local run of hadoop. Gurobi uses JNI. To test how to run it on hadoop I've written the following code.

  public static void main(String[] args) throws Exception {
    try {
      GRBEnv env2 = new GRBEnv();
      env2.set(GRB.IntParam.OutputFlag, 0);
    } catch (Exception e) {
      e.printStackTrace();
    }

    Configuration conf = new Configuration();
    int res = ToolRunner.run(conf, new GurobiTest(), args);
    System.exit(res);
}

Before using, Gurobi requires to set some environmental variables (where LD_LIBRARY_PATH points to Gurobi native library):

export LD_LIBRARY_PATH=/home/username/System/gurobi563/linux64/lib
export GRB_LICENSE_FILE=/home/username/gurobi.lic

When I package my project and run it using jvm

java -cp ./target/GurobiOnHadoop-0.0.1-SNAPSHOT-jar-with-dependencies.jar mpi.de.test.GurobiTest

everything seems to work fine

But when I am trying to run it on hadoop

hadoop jar ./target/GurobiOnHadoop-0.0.1-SNAPSHOT-jar-with-dependencies.jar mpi.de.test.GurobiTest -libjars ./target/GurobiOnHadoop-0.0.1-SNAPSHOT-jar-with-dependencies.jar

I get an error

Exception in thread "main" java.lang.UnsatisfiedLinkError: no GurobiJni56 in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
    at java.lang.Runtime.loadLibrary0(Runtime.java:849)
    at java.lang.System.loadLibrary(System.java:1088)
    at gurobi.GurobiJni.<clinit>(GurobiJni.java:193)
    at gurobi.GRBEnv.<init>(GRBEnv.java:16)
    at gurobi.GRBEnv.<init>(GRBEnv.java:11)
    at mpi.de.test.GurobiTest.main(GurobiTest.java:89)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:197)

I've tried many different things. I've added "-Djava.library.path=/home/username/System/gurobi563/linux64/lib" to the hadoop running command. I've set required environmental variables in hadoop-env.sh. I try to load *.so files from the code level (using e.g. ystem .load("/home/username/System/gurobi563/linux64/lib/libGurobiJni56.so");). I've also tried to send those libraries to the hadoop distributed file system, load files from that place or point enviromental variables to that place.

I am using:

java version "1.7.0_60"

hadoop-0.20.2-cdh3u6

Debian 7.5 wheezy

Author:Anonim33,eproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article:https://stackoverflow.com/questions/24289535/running-java-native-libraries-on-hadoop
yy