|
Hadoop小程序测试方法1、创建新的JAVA project项目:Hadoop
2、点击hadoop项目右键进入属性,在java build path中加载所需要的JAR包。
3、点击C+创建类,在类名:填写PutMerge,并打钩public static void main(String[] args]
4、编写java小程序
- import java.io.IOException;
- import org.apache.hadoop.conf.Configuration;
- import org.apache.hadoop.fs.FSDataInputStream;
- import org.apache.hadoop.fs.FSDataOutputStream;
- import org.apache.hadoop.fs.FileStatus;
- import org.apache.hadoop.fs.FileSystem;
- import org.apache.hadoop.fs.Path;
- public class PutMerge {
- public static void main(String[] args) throws IOException {
- Configuration conf = new Configuration();
- FileSystem hdfs = FileSystem.get(conf);
- FileSystem local = FileSystem.getLocal(conf);
- Path inputDir = new Path(args[0]); --输入本地文件系统的目录和文件
- Path hdfsFile = new Path(args[1]); --输出HDFS的文件名
- try {
- FileStatus[] inputFiles = local.listStatus(inputDir);
- FSDataOutputStream ut = hdfs.create(hdfsFile); --创建文件
- for (int i=0; i<inputFiles.length; i++) {
- System.out.println(inputFiles[i].getPath().getName());
- FSDataInputStream in = local.open(inputFiles[i].getPath());
- byte buffer[] = new byte[256];
- int bytesRead = 0;
- while( (bytesRead = in.read(buffer)) > 0) {
- out.write(buffer, 0, bytesRead);
- }
- in.close();
- }
- out.close();
- } catch (IOException e) {
- e.printStackTrace();
- }
- }
- }
复制代码 5、该小程序生成class类文件,我们需要对该类文件打成JAR包,然后上传到HADOOP平台
D:>cd d:\hadoop\project\HelloJava\bin
编写文本manifest.mf
Main-Class: PutMerge
D:>jar cvfm PutMerge.jar manifest.mf PutMerge.class
生成的JAR包上传到服务器中。
6、测试
- # vi /tmp/test1.txt
- adfsfs fasfasfeqe fwfqw
- # hadoop jar PutMerge.jar /tmp/test1.txt /tmp/1.txt (/tmp是HDFS创建目录)
- #hadoop fs -cat /tmp/1.txt
复制代码
|
|