I am trying to run Gurobi optimizer on the local run of hadoop. Gurobi uses JNI. To test how to run it on hadoop I've written the following code.
public static void main(String[] args) throws Exception {
try {
GRBEnv env2 = new GRBEnv();
env2.set(GRB.IntParam.OutputFlag, 0);
} catch (Exception e) {
e.printStackTrace();
}
Configuration conf = new Configuration();
int res = ToolRunner.run(conf, new GurobiTest(), args);
System.exit(res);
}
Before using, Gurobi requires to set some environmental variables (where LD_LIBRARY_PATH points to Gurobi native library):
export LD_LIBRARY_PATH=/home/username/System/gurobi563/linux64/lib
export GRB_LICENSE_FILE=/home/username/gurobi.lic
When I package my project and run it using jvm
java -cp ./target/GurobiOnHadoop-0.0.1-SNAPSHOT-jar-with-dependencies.jar mpi.de.test.GurobiTest
everything seems to work fine
But when I am trying to run it on hadoop
hadoop jar ./target/GurobiOnHadoop-0.0.1-SNAPSHOT-jar-with-dependencies.jar mpi.de.test.GurobiTest -libjars ./target/GurobiOnHadoop-0.0.1-SNAPSHOT-jar-with-dependencies.jar
I get an error
Exception in thread "main" java.lang.UnsatisfiedLinkError: no GurobiJni56 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
at java.lang.Runtime.loadLibrary0(Runtime.java:849)
at java.lang.System.loadLibrary(System.java:1088)
at gurobi.GurobiJni.<clinit>(GurobiJni.java:193)
at gurobi.GRBEnv.<init>(GRBEnv.java:16)
at gurobi.GRBEnv.<init>(GRBEnv.java:11)
at mpi.de.test.GurobiTest.main(GurobiTest.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
I've tried many different things. I've added "-Djava.library.path=/home/username/System/gurobi563/linux64/lib" to the hadoop running command. I've set required environmental variables in hadoop-env.sh. I try to load *.so files from the code level (using e.g. ystem
.load("/home/username/System/gurobi563/linux64/lib/libGurobiJni56.so");). I've also tried to send those libraries to the hadoop distributed file system, load files from that place or point enviromental variables to that place.
I am using:
java version "1.7.0_60"
hadoop-0.20.2-cdh3u6
Debian 7.5 wheezy