I am building a spark application using aws sdk to access an S3 source. I am getting the below error:
java.lang.NoSuchMethodError:
org.apache.http.conn.ssl.SSLConnectionSocketFactory.(Ljavax/net/ssl/SSLContext;Ljavax/net/ssl/HostnameVerifier;)V
Looked up online for solution and it appears my spark application is using a wrong httpclient. The following thread seems to offer a solution but not sure how I can override the default default httpclient.
What version of httpclient is compatible with the Amazon SDK v 1.11.5?
Here are the different httpclients that I have in my system.
./Applications/IBM Notes.app/Contents/MacOS/shared/eclipse/plugins/org.apache.wink_1.1.2.20150826-0855/lib/httpclient-4.0.1.jar
./Users/XXXXX/.ivy2/cache/org.apache.httpcomponents/httpclient/jars/httpclient-4.1.2.jar
./Users/XXXXX/.ivy2/cache/org.apache.httpcomponents/httpclient/jars/httpclient-4.5.1.jar
./Users/XXXXX/.m2/repository/org/apache/httpcomponents/httpclient/4.0.2/httpclient-4.0.2.jar
./Users/XXXXX/.m2/repository/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar
./Users/XXXXX/Downloads/aws-java-sdk-1.11.110/third-party/lib/httpclient-4.5.2.jar
./usr/local/aws-java/aws-java-sdk-1.11.109/third-party/lib/httpclient-4.5.2.jar
./usr/local/spark/spark-2.1.0-bin-hadoop2.7/jars/httpclient-4.5.2.jar
./usr/local/zeppelin/interpreter/alluxio/httpclient-4.3.6.jar
./usr/local/zeppelin/interpreter/bqsql/httpclient-4.3.6.jar
./usr/local/zeppelin/interpreter/elasticsearch/httpclient-4.3.6.jar
./usr/local/zeppelin/interpreter/hbase/httpclient-4.3.6.jar
./usr/local/zeppelin/interpreter/kylin/httpclient-4.3.6.jar
./usr/local/zeppelin/interpreter/lens/httpclient-4.3.6.jar
./usr/local/zeppelin/interpreter/livy/httpclient-4.3.4.jar
./usr/local/zeppelin/interpreter/pig/httpclient-4.3.6.jar
./usr/local/zeppelin/lib/httpclient-4.3.6.jar
./usr/local/zeppelin/lib/interpreter/httpclient-4.3.6.jar
I do not have classpath specified, so I am not sure which httpclient it is picking. How do I override such that it always picks up ./usr/local/aws-java/aws-java-sdk-1.11.109/third-party/lib/httpclient-4.5.2.jar?