mac下编译Hadoop

以前都是win下装个centos虚拟机编译hadoop,最近换了Mac,不用再装虚拟机了,直接编译Hadoop,很爽。但需要配置下hadoop的编译环境。在此记录下。

findbugs安装

下载findbugs-3.0.1.tar.gz,解压tar -zvxf findbugs-3.0.1.tar.gz,配置环境变量。
.bash_profile中添加FINDBUGS_HOME,并export

1
2
3
FINDBUGS_HOME=/Users/hadoop/soft/findbugs-3.0.1
PATH=$FINDBUGS_HOME/bin
export FINDBUGS_HOME

protobuf安装

安装protobuf之前验证下是否有gcc,命令gcc --version
进入protobuf-2.5.0目录执行./configure --prefix=/Users/hadoop/soft/protobuf-2.5.0
然后安装就可以了,命令make && make install
为了使用protoc命令方便,将bin目录添加到.bash_porfile
PATH=/Users/hadoop/soft/protobuf-2.5.0/bin

cmake安装

解压cmake-3.0.0,进入目录中,执行./bootstrap --prefix=/Users/hadoop/soft/cmake-3.0.0进行编译
执行make && make install进行
为了使用cmake命令方便,将bin目录添加到.bash_porfile
PATH=/Users/hadoop/soft/cmake-3.0.0/bin

mac环境中由于已经安装了xcode-select则,不用额外安装zlib-devel

openssl-devel安装

这个在mac上没有这个依赖包,只是装了openssl,但是编译不过去,报错,内容如下:

1
2
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec dir="/Users/hadoop/bigdata/hadoop-2.6.0-src-eclipse/hadoop-tools/hadoop-pipes/target/native" executable="cmake" failonerror="true">... @ 5:141 in /Users/hadoop/bigdata/hadoop-2.6.0-src-eclipse/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml

网上大部分都说是缺少openssl-devel依赖包,但苦逼的是mac上没有此依赖包,然后就下载了openssl源码进行自编译安装吧,可是就是编译不过去,下载了各个版本的openssl就是无法编译成功。然后看error信息,貌似是和ant和cmake有关,hadoop现在的版本已经不用ant了呀,而且contos上的编译环境也没有ant呀,难道是cmake的原因?首先想到cmake版本问题,换了个新版也不成功,最后快要绝望时搜到了一篇文章

上面说可以去build-main.xml中查看具体出错的行,用cmake单独执行下,看报什么错

当我执行cmake /Users/hadoop/bigdata/hadoop-2.6.0-src-eclipse/hadoop-tools/hadoop-pipes/src/ -DJVM_ARCH_DATA_MODEL = 64时,错误信息显示出来了,

1
Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the System variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR), suggesting that not found openssl suggested that the need to add environment variables

然后我就按照文章中介绍的将OPENSSL_INCLUDE_DIROPENSSL_ROOT_DIRexport出,再次编译就成功了。

由于我并不知道mac默认的openssl的安装目录,所以我使用brew install openssl重新安装了下,安装成功会提示openssl的安装目录

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
brew install openssl
==> Downloading https://homebrew.bintray.com/bottles/openssl-1.0.2k.sierra.bottle.tar.gz
Already downloaded: /Users/netease/Library/Caches/Homebrew/openssl-1.0.2k.sierra.bottle.tar.gz
==> Pouring openssl-1.0.2k.sierra.bottle.tar.gz
==> Using the sandbox
==> Caveats
A CA file has been bootstrapped using certificates from the SystemRoots
keychain. To add additional certificates (e.g. the certificates added in
the System keychain), place .pem files in
/usr/local/etc/openssl/certs
and run
/usr/local/opt/openssl/bin/c_rehash
This formula is keg-only, which means it was not symlinked into /usr/local.
Apple has deprecated use of OpenSSL in favor of its own TLS and crypto libraries
If you need to have this software first in your PATH run:
echo 'export PATH="/usr/local/opt/openssl/bin:$PATH"' >> ~/.zshrc
For compilers to find this software you may need to set:
LDFLAGS: -L/usr/local/opt/openssl/lib
CPPFLAGS: -I/usr/local/opt/openssl/include
For pkg-config to find this software you may need to set:
PKG_CONFIG_PATH: /usr/local/opt/openssl/lib/pkgconfig
==> Summary
🍺 /usr/local/Cellar/openssl/1.0.2k: 1,696 files, 12MB
If you need to have this software first in your PATH run:
echo 'export PATH="/usr/local/opt/openssl/bin:$PATH"' >> ~/.zshrc

编译hadoop

所需的依赖组件都安装好之后就可以正确的编译hadoop了,在hadoop_home目录下执行命令
mvn clean package -DskipTests -Pdist,native -Dtar -X

这里还有个小插曲,就是会报个找不类的错,

1
Exception in thread “main” Java.lang.AssertionError: Missing tools.jar at: /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/Classes/classes.jar. Expression: file.exists()

解决办法就是在JAVA_HOME,也即/Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home路径下新建目录Class,并执行创建软符号链接命令:
sudo ln -s $JAVA_HOME/lib/tools.jar $JAVA_HOME/Classes/classes.jar

您的肯定,是我装逼的最大的动力!