Hadoop DistributedCache 在 Windows 使用失败

最近有同学反馈，在windows下使用 Hadoop DistributedCache 失败.

17/02/23 12:13:12 INFO mapred.LocalDistributedCacheManager: Creating symlink: \tmp\hadoop-Administrator\mapred\local\1487823192632\student.dat <- D:\workspace\projects\mrdemo/student.dat
17/02/23 12:13:12 WARN fs.FileUtil: Command 'D:\hadoop-2.7.3\bin\winutils.exe symlink D:\workspace\projects\mrdemo\student.dat \tmp\hadoop-Administrator\mapred\local\1487823192632\student.dat' failed 1 with: CreateSymbolicLink error (1314): ???????????

17/02/23 12:13:12 WARN mapred.LocalDistributedCacheManager: Failed to create symlink: \tmp\hadoop-Administrator\mapred\local\1487823192632\student.dat <- D:\workspace\projects\mrdemo/student.dat
17/02/23 12:13:12 INFO mapred.LocalDistributedCacheManager: Localized file:/D:/data/test_dat/student/student.dat as file:/tmp/hadoop-Administrator/mapred/local/1487823192632/student.dat
17/02/23 12:13:12 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
17/02/23 12:13:12 INFO mapreduce.Job: Running job: job_local657929120_0001
17/02/23 12:13:12 INFO mapred.LocalJobRunner: OutputCommitter set in config null
17/02/23 12:13:12 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/02/23 12:13:12 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
17/02/23 12:13:13 INFO mapred.LocalJobRunner: Waiting for map tasks
17/02/23 12:13:13 INFO mapred.LocalJobRunner: Starting task: attempt_local657929120_0001_m_000000_0
17/02/23 12:13:13 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/02/23 12:13:13 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
17/02/23 12:13:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@56081a5
17/02/23 12:13:13 INFO mapred.MapTask: Processing split: file:/D:/data/test_dat/exam/exam.dat:0+87
17/02/23 12:13:13 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/02/23 12:13:13 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/02/23 12:13:13 INFO mapred.MapTask: soft limit at 83886080
17/02/23 12:13:13 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/02/23 12:13:13 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/02/23 12:13:13 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
file:///D:/data/test_dat/student/student.dat#student.dat
17/02/23 12:13:13 INFO mapred.MapTask: Starting flush of map output
17/02/23 12:13:13 INFO mapred.LocalJobRunner: map task executor complete.
17/02/23 12:13:13 WARN mapred.LocalJobRunner: job_local657929120_0001
java.lang.Exception: java.io.FileNotFoundException: student.dat (系统找不到指定的文件。)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.io.FileNotFoundException: student.dat (系统找不到指定的文件。)
    at java.io.FileInputStream.open0(Native Method)
    at java.io.FileInputStream.open(FileInputStream.java:195)
    at java.io.FileInputStream.<init>(FileInputStream.java:138)
    at java.io.FileInputStream.<init>(FileInputStream.java:93)
    at com.hainiubl.hadoop.mrdemo.SemiJoinJob$JoinMapper.setup(SemiJoinJob.java:63)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
17/02/23 12:13:13 INFO mapreduce.Job: Job job_local657929120_0001 running in uber mode : false
17/02/23 12:13:14 INFO mapreduce.Job:  map 0% reduce 0%
17/02/23 12:13:14 INFO mapreduce.Job: Job job_local657929120_0001 failed with state FAILED due to: NA
17/02/23 12:13:14 INFO mapreduce.Job: Counters: 0

仔细看一下是创建链接失败， failed 1 with: CreateSymbolicLink error (1314)，这个是由windows的权限控制造成的，我们可以用一个简单粗暴的解决办法就是关闭权限控制，默认以管理员身份运行应用，当然你对windows权限控制比较熟悉的话可以控制的更详细。

主要操作步骤：