When running a Hadoop client on Windows to reach a kerberized cluster, you need a specific \"native library\" (i.e. DLL).\nAs far as I can tell there is no good reason for that, because that lib is not actually used outside of some automated regression tests (!?!) so it's a pain inflicted to Hadoop users by Hadoop committers.\n\nTo add extra pain, there is no official build of that DLL (and of the Windows \"stub\" that enable its use from Java). You must either (a) build it yourself from source code -- good luck -- or (b) search the internet for a downloadable Hadoop-for-Windows runtime, and pray that is does not contain any malware.\nThe best option (for 64-bit Windows) is here: https://github.com/steveloughran/winutils\n...and the ReadMe explains why you can reasonably trust that run-time. But if you are stuck with an older 32-bit Windows, then you are on your own.\n\nNow let's assume you deployed that run-time on your Windows box under\n C:\\Some Dir\\hadoop\\bin\\(the final bin is required; the embedded space is just extra fun)\n\nYou must point the Hadoop client to that run-time with a couple of Java properties:\n \"-Dhadoop.home.dir=C:/Some Dir/hadoop\" \"-Djava.library.path=C:/Some Dir/hadoop/bin\"\n(note the double quotes around Windows args as a whole, to protect embedded spaces in the paths, which have been translated to Java style for extra fun)(in Eclipse, just stuff these props under \"VM Arguments\", quotes included)\n\nNow, there's the Kerberos config. If your KDC is your corporate Active Directory server, then Java should find the config parameters automatically. But if your KDC is a standalone \"MIT Kerberos\" install on Linux, then you have to find a valid /etc/krb5.conf file on the cluster, copy it on your Windows box, and have Java use it with an additional property...\n \"-Djava.security.krb5.conf=C:/Some Other Dir/krb5.conf\"\n\nThen let's assume you have created your keytab file on a Linux box, using ktutil (or an Active Directory admin created it for you with some AD command) and you dropped the file under C:\\Some Other Dir\\foo.keytab\nBefore anything else, if the keytab is for a real Windows account -- i.e. your own account -- or a Prod service account, then make sure that keytab is secure!! Use the Windows Security dialog box to restrict access to your account only (and maybe System, for backups). Because that file could enable anyone, on any machine, to authenticate on the cluster (and any Kerberos-enabled system, including Windows).\n\nNow you can try to authenticate using\nUserGroupInformation.loginUserFromKeytab(\"
[email protected]\", \"C:/Some Other Dir/foo.keytab\");\n\nIf it does not work, enable the Kerberos debug traces with both an environment variable\n set HADOOP_JAAS_DEBUG=true\n...and a Java property\n -Dsun.security.krb5.debug=true\n(in Eclipse, set these in \"Environment\" and \"VM Arguments\" respectively)",
2017-03-21T12:36:07