本文共 3353 字，大约阅读时间需要 11 分钟。

HDFS?Hadoop???????

HDFS?Hadoop Distributed File System??Hadoop????????????????????????????????????????????????????????????????????????????Java API?????????HDFS???????

??????

??VMware 15???????????CentOS 7?????????Apache Hadoop 3.1.4???????Windows?Mac???????IDEA?IntelliJ IDEA????????????????????????Hadoop?????????????????

??????

??????

??????

1. Hadoop?????

Hadoop????????Apache Nutch???????????????????????????????????????Nutch???????????????2003??????????????????????????GFS?Google File System??MapReduce???????????Hadoop???????????

2004??2005??Nutch?????????GFS?MapReduce?????????????????Hadoop???2008?1??Hadoop????Apache??????????????????

Hadoop?????????????????????????????????????????

2. Hadoop????

Hadoop?????????????

• 0.x???Hadoop?????????????????
• 1.x????????0.x??????bug???????????
• 2.x??????YARN??????????????????????????????????
• 3.x????2.x????????HDFS????????????????????

3. Hadoop????????

Hadoop??????????????Apache Hadoop?Cloudera?Hortonworks?MapR?

• Apache Hadoop?????????????????
• Cloudera????????????????Impala???????????
• Hortonworks????????Windows??????????
• MapR????????Hadoop??????????????

??Hadoop??????

Hadoop??????????

1. HDFS??

HDFS????????

• NameNode????????????????????????????????
• DataNode??????????????????????
• Secondary NameNode???NameNode????????????????

2. YARN??

YARN????????????

• ResourceManager?????????????????
• NodeManager??????????MapReduce???

??HDFS????

1. ?????????

HDFS???????????????128MB?????????????????????????????3????????????????

• ???????hdfs-site.xml???dfs.blocksize??????????
• ???????????dfs.replication?????????????????????????

2. ?????????

• ??????????????HDFS??????????????????????
• ????????????????????????????
• ??????????????????????????

3. HDFS??

HDFS????NameNode?DataNode?Secondary NameNode?

• NameNode??????????????????
• DataNode????????????????????
• Secondary NameNode???NameNode??????????????????????

4. ?????

• ?????????????????????
• ?????????Unix???????????????

??HDFS????

HDFS?????????hadoop fs?hdfs dfs?????????????????????????????????

• ?????
  hdfs dfs -mkdir /path
• ?????
  hdfs dfs -put local_file hdfs_path
• ?????
  hdfs dfs -get hdfs_path local_file
• ?????
  hdfs dfs -rm hdfs_path

??HDFS????

HDFS?????????????????????????????????????????????????

• ????????????????99.9%?????????????
• ????????????????????hdfs dfsadmin -safemode enter | leave | get | wait?

??Java API??

1. ????

• Windows?????Hadoop??????????????????
• Mac??????????????Hadoop???

2. ?????????

??Maven???????Hadoop?????

       
            
     
      org.apache.hadoop
             
     
      hadoop-client
             
     
      ${hadoop.version}
         
        
    
   

3. Java API????

@Testpublic void createDirectory() throws IOException {    Configuration configuration = new Configuration();    configuration.set("fs.defaultFS", "hdfs://node01:8020");    FileSystem fileSystem = FileSystem.get(configuration);    Path dirPath = new Path("/kaikeba/dir1");    FsPermission permission = new FsPermission(FsAction.ALL, FsAction.READ, FsAction.WRITE);    fileSystem.mkdirs(dirPath, permission);    fileSystem.close();}

?????????

???????????????HDFS????????????????

转载地址：http://fg.baihongyu.com/