Home:ALL Converter>Outdated Hadoop dependency for reading/writing Parquet in Java

Outdated Hadoop dependency for reading/writing Parquet in Java

Ask Time:2021-03-15T00:03:54         Author:Eugene Marin

Json Formatter

I have a Java service writing and reading Parquet files using parquet-avro 1.11.1.

Now as you probably know this requires another dependency (which is not included in parquet-avro) for some hadoop classes, for example:

org.apache.hadoop.conf.Configuration
org.apache.hadoop.fs.Path

I ended up using hadoop-core 1.2.1, which works. The thing is, this library is from 2013, and it's the latest version! I wonder if there's a newer alternative? I tried the following:

Both compile but don't work without a native Hadoop installation, which I'd like to avoid - I only want the classes needed for working with Parquet.

Author:Eugene Marin,eproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article:https://stackoverflow.com/questions/66626628/outdated-hadoop-dependency-for-reading-writing-parquet-in-java
yy