[Hadoop] pom.xml로 maven 컴파일하기

 Hadoop을 설치하였을 때 사용자의 이해를 돕기 위해 별도로 예제를 마련해 두고 있습니다. maven을 사용해 Hadoop 소스코드를 컴파일 하기 위해서는 pom.xml을 사용해야 하는데 간단한 Hadoop 예제를 pom.xml로 컴파일 하는 방법을 알아보도록 하겠습니다.

1. 먼저 실행하고자 하는 Hadoop 소스코드를 구현합니다. 본 예제는 하둡의 예제 소스코드인 WordCount를 사용하였습니다.

/hadoop-example/src/main/java/elecs/tistory/com/WordCount.java

package elecs.tistory.com;
 
import java.io.IOException;
import java.util.StringTokenizer;
 
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
 
public class WordCount {
 
  public static class TokenizerMapper 
       extends Mapper<Object, Text, Text, IntWritable>{
    
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();
      
    public void map(Object key, Text value, Context context
                    ) throws IOException, InterruptedException {
      StringTokenizer itr = new StringTokenizer(value.toString());
      while (itr.hasMoreTokens()) {
        word.set(itr.nextToken());
        context.write(word, one);
      }
    }
  }
  
  public static class IntSumReducer 
       extends Reducer<Text,IntWritable,Text,IntWritable> {
    private IntWritable result = new IntWritable();
 
    public void reduce(Text key, Iterable<IntWritable> values, 
                       Context context
                       ) throws IOException, InterruptedException {
      int sum = 0;
      for (IntWritable val : values) {
        sum += val.get();
      }
      result.set(sum);
      context.write(key, result);
    }
  }
 
  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
    if (otherArgs.length < 2) {
      System.err.println("Usage: <in> [<in>...] <out>");
      System.exit(2);
    }
    Job job = new Job(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    for (int i = 0; i < otherArgs.length - 1; ++i) {
      FileInputFormat.addInputPath(job, new Path(otherArgs[i]));
    }
    FileOutputFormat.setOutputPath(job,
      new Path(otherArgs[otherArgs.length - 1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

2. 다음으로 소스코드를 컴파일 하기 위한 pom.xml을 만듭니다.

/hadoop-example/pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                      http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-project</artifactId>
    <version>2.6.0</version>
  </parent>
  <groupId>elecs.tistory.com</groupId>
  <artifactId>hadoop-wordcount-example</artifactId>
  <version>2.6.0</version>
  <description>Apache Hadoop Wordcount Example</description>
  <name>Apache Hadoop Wordcount Example</name>
  <packaging>jar</packaging>
 
  <dependencies>
    <dependency>
      <groupId>commons-cli</groupId>
      <artifactId>commons-cli</artifactId>
    </dependency>
    <dependency>
      <groupId>commons-logging</groupId>
      <artifactId>commons-logging</artifactId>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
      <version>${project.version}</version>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
      <scope>test</scope>
      <type>test-jar</type>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <scope>test</scope>
      <type>test-jar</type>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-hdfs</artifactId>
      <scope>runtime</scope>
    </dependency>
    <dependency>
       <groupId>org.apache.hadoop</groupId>
       <artifactId>hadoop-hdfs</artifactId>
       <scope>test</scope>
       <type>test-jar</type>
     </dependency>
     <dependency>
       <groupId>org.apache.hadoop</groupId>
       <artifactId>hadoop-yarn-server-tests</artifactId>
       <scope>test</scope>
       <type>test-jar</type>
     </dependency>
     <dependency>
       <groupId>org.apache.hadoop</groupId>
       <artifactId>hadoop-mapreduce-client-app</artifactId>
       <scope>provided</scope>
     </dependency>
     <dependency>
       <groupId>org.apache.hadoop</groupId>
       <artifactId>hadoop-mapreduce-client-app</artifactId>
       <type>test-jar</type>
       <scope>test</scope>
     </dependency>
    <dependency>
      <groupId>com.sun.jersey.jersey-test-framework</groupId>
      <artifactId>jersey-test-framework-grizzly2</artifactId>
      <scope>test</scope>
    </dependency>
     <dependency>
       <groupId>org.apache.hadoop</groupId>
       <artifactId>hadoop-mapreduce-client-hs</artifactId>
       <scope>test</scope>
     </dependency>
     <dependency>
       <groupId>org.hsqldb</groupId>
       <artifactId>hsqldb</artifactId>
       <scope>provided</scope>
     </dependency>
     <dependency>
      <groupId>com.google.guava</groupId>
      <artifactId>guava</artifactId>
      <scope>provided</scope>
     </dependency>
  </dependencies>
 
  <!--
  자신이 main으로 설정하고자 하는 class의 이름을 package경로와 함께 표기합니다.
  -->
  <build>
   <plugins>
    <plugin>
    <groupId>org.apache.maven.plugins</groupId>
     <artifactId>maven-jar-plugin</artifactId>
      <configuration>
       <archive>
         <manifest>
           <mainClass>elecs.tistory.com.WordCount</mainClass>
         </manifest>
       </archive>
     </configuration>
    </plugin>
   </plugins>
   </build>
</project>

3. 위 과정까지 진행하였다면 다음으로 maven을 사용해 Hadoop 소스코드를 컴파일합니다.

# mvn clean

 명령어 수행 후 target폴더가 생성되고 그 안에 jar 파일이 생성되었다면 컴파일에 성공한 것입니다.

 4. 다음으로 WordCount를 수행하기 위한 예제를 만듭니다.

# vi hello.txt

 5. hello.txt 예제 파일을 만든 후 hdfs에 파일을 전송합니다.

# hdfs dfs -mkdir /input

# hdfs dfs -copyFromLocal hello.txt /input

 6. hadoop 명령어를 실행하여 예제를 실행합니다.

# hadoop jar target/hadoop-wordcount-example-2.6.0.jar /input/hello.txt /output

7. 완성된 결과물을 Local 폴더에 저장합니다.

# hdfs dfs -getmerge /output result.txt

# cat result.txt

 아래와 같은 결과가 출력되면 Hadoop이 정상적으로 동작한 것입니다.