AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-1-
NOTE: Most of the install steps are performed as root -------------------------------------------------------------------------------------------------------------------CREATE A VM USING RED HAT (6.5) X64 (e.g. AMI: m3.large, 32G/Disk, 7.5 G/RAM) -------------------------------------------------------------------------------------------------------------------DISABLE IPTABLES (as root) chkconfig iptables off service iptables stop -------------------------------------------------------------------------------------------------------------------INSTALL JAVA 1.6 (as root) yum install java-1.6.0-openjdk-devel echo "export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk.x86_64; export PATH=\$JAVA_HOME/bin:\$PATH" > /etc/profile.d/java.sh source /etc/profile.d/java.sh -------------------------------------------------------------------------------------------------------------------INSTALL EPEL REPO (as root) rpm -ivH http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm -------------------------------------------------------------------------------------------------------------------INSTALL RPMS AND NPM MODULES (as root) yum install -y git nodejs npm libuuid-devel libtool zip unzip npm install -g inherits bower grunt grunt-cli -------------------------------------------------------------------------------------------------------------------INSTALL MAVEN 3.0.5 (as root) mkdir ~/downloads cd ~/downloads wget http://archive.apache.org/dist/maven/binaries/apache-maven-3.0.5-bin.zip sudo unzip apache-maven-3.0.5-bin.zip -d /usr/share/ sudo ln -s /usr/share/apache-maven-3.0.5/bin/mvn /usr/bin/mvn -------------------------------------------------------------------------------------------------------------------INSTALL CLOUDERA 4.8.4 (as root) * Disable SE Linux sed -i -e "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config * Reboot (or run 'setenforce 0' to temporarily turn SE Linux off until reboot) reboot * Install Cloudera Manager 4.8.3 cd ~/downloads wget http://archive.cloudera.com/cm4/installer/latest/cloudera-manager-installer.bin AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-1-
AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-2-
chmod u+x cloudera-manager-installer.bin ./cloudera-manager-installer.bin * Configure Cloudera Console: Browse to: http://{HOST}:7180 (e.g. http://172.31.0.50:7180) Use Credentials: admin/admin Select the free edition Select the following components: CDH, Solr, ZooKeeper, HDFS, HBase, MapReduce, Hive, Oozie, Sqoop, Impala, Hue, Cloudera Management Services * Test the Installation using a PI Estimator Job sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100 -------------------------------------------------------------------------------------------------------------------INSTALL ACCUMULO 1.5.1 (as root) cd ~/downloads wget http://www.us.apache.org/dist/accumulo/1.5.1/accumulo-1.5.1-bin.tar.gz cd /usr/lib tar xzf ~/downloads/accumulo-1.5.1-bin.tar.gz ln -s accumulo-1.5.1 accumulo groupadd supergroup useradd -g supergroup accumulo chown -R accumulo:supergroup accumulo-1.5.1/ cp accumulo/conf/examples/1GB/standalone/* accumulo/conf cp accumulo/conf/accumulo-env.sh accumulo/conf/accumulo-env.sh.orig vi accumulo/conf/accumulo-env.sh ... if [ -z "$HADOOP_HOME" ] then test -z "$HADOOP_PREFIX" && export HADOOP_PREFIX=/opt/cloudera/parcels/CDH/lib/hadoop else HADOOP_PREFIX="$HADOOP_HOME" unset HADOOP_HOME fi test -z "$HADOOP_CONF_DIR" && export HADOOP_CONF_DIR="$HADOOP_PREFIX/etc/hadoop" test -z "$JAVA_HOME" && export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk.x86_64 test -z "$ZOOKEEPER_HOME" && export ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH/lib/zookeeper export HADOOP_HDFS_HOME=/opt/cloudera/parcels/CDH/lib/hadoop-hdfs export HADOOP_MAPREDUCE_HOME=/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce ... cp accumulo/conf/accumulo-site.xml accumulo/conf/accumulo-site.xml.orig vi accumulo/conf/accumulo-site.xml ...
instance.dfs.uri hdfs://token1:8020 master.port.client AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-2-
AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-3-
8995 tserver.port.client 8997 general.classpaths $ACCUMULO_HOME/server/target/classes/, $ACCUMULO_HOME/lib/accumulo-server.jar, $ACCUMULO_HOME/core/target/classes/, $ACCUMULO_HOME/lib/accumulo-core.jar, $ACCUMULO_HOME/start/target/classes/, $ACCUMULO_HOME/lib/accumulo-start.jar, $ACCUMULO_HOME/fate/target/classes/, $ACCUMULO_HOME/lib/accumulo-fate.jar, $ACCUMULO_HOME/proxy/target/classes/, $ACCUMULO_HOME/lib/accumulo-proxy.jar, $ACCUMULO_HOME/lib/[^.].*.jar, $ACCUMULO_HOME/lib/accumulo-examples-simple.jar, $ZOOKEEPER_HOME/zookeeper[^.].*.jar, $HADOOP_CONF_DIR, $HADOOP_PREFIX/[^.].*.jar, $HADOOP_PREFIX/lib/[^.].*.jar, $HADOOP_PREFIX/client/[^.].*.jar, $HADOOP_HDFS_HOME/.*.jar, $HADOOP_HDFS_HOME/lib/.*.jar ... ip_address=$(ip addr show eth0 | awk '/inet / {print $2}' | cut -d / -f 1) sed -i -e "s|
secret|
password|" accumulo/conf/accumulo-site.xml sed -i -e "s/token1/${ip_address}/" accumulo/conf/accumulo-site.xml sed -i -e "s/localhost/${ip_address}/" accumulo/conf/monitor d. Initialize Hadoop & Accumulo sudo -u hdfs hadoop fs -mkdir /accumulo sudo -u hdfs hadoop fs -chown -R accumulo:supergroup /accumulo sudo -u hdfs hadoop fs -chmod -R 775 /user sudo -u accumulo /usr/lib/accumulo/bin/accumulo init --instance-name lumify --password password cd /usr/lib/accumulo/bin ln -s /usr/lib/accumulo/bin/etc_initd_accumulo /etc/init.d/accumulo chkconfig accumulo on cp etc_initd_accumulo etc_initd_accumulo.orig vi /etc/init.d/accumulo ... start) /bin/su - accumulo -c "$ACCUMULO_HOME/bin/start-here.sh" RETVAL=$? ;; AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-3-
AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-4-
... stop) /bin/su - accumulo -c "$ACCUMULO_HOME/bin/stop-here.sh" RETVAL=$? ;; ... su - accumulo vi ~/.bashrc ... export ACCUMULO_HOME=/usr/lib/accumulo ... exit /etc/init.d/accumulo start * Check Install Check Console, Browse to: http://{HOST}:50095 (e.g. http://172.31.0.50:50095) Check Console for errors in the logs * Test Accumulo Shell su - accumulo $ACCUMULO_HOME/bin/accumulo shell -u root -p password * Shell prompt "root@lumify>" should appear * Test Accumulo Example su - accumulo $ACCUMULO_HOME/bin/accumulo shell -u root -p password createtable hellotable exit $ACCUMULO_HOME/bin/accumulo org.apache.accumulo.examples.simple.helloworld.InsertWithBatchWriter -i lumify -u root -p password -t hellotable $ACCUMULO_HOME/bin/accumulo shell -u root -p password maxrow -t hellotable * Should display 'row_9999' -------------------------------------------------------------------------------------------------------------------INSTALL ELASTICSEARCH 1.1.2 (as root) cd ~/downloads wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.2.noarch.rpm rpm -i elasticsearch-1.1.2.noarch.rpm cd /usr/share chown -R elasticsearch:supergroup elasticsearch ./elasticsearch/bin/plugin install org.securegraph/securegraph-elasticsearch-plugin/0.6.0 ./elasticsearch/bin/plugin install mobz/elasticsearch-head chkconfig elasticsearch on /etc/init.d/elasticsearch start * Test Service Browse to http://{HOST}:9200 (e.g.http://172.31.0.52:9200) Some JSON about Elasticsearch should display -------------------------------------------------------------------------------------------------------------------INSTALL STORM 0.9.0.1 (as root) AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-4-
AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-5-
cd ~/downloads wget http://download.zeromq.org/zeromq-2.1.7.zip unzip zeromq-2.1.7.zip cd zeromq-2.1.7 ./configure make make install ldconfig cd ~/downloads git clone https://github.com/zeromq/jzmq.git cd jzmq ./autogen.sh ./configure make make install cd ~/downloads wget http://www.us.apache.org/dist/commons/io/binaries/commons-io-2.4-bin.zip cd ~/downloads wget https://dl.dropboxusercontent.com/s/dj86w8ojecgsam7/storm-0.9.0.1.zip cd /opt unzip ~/downloads/storm-0.9.0.1.zip -d . mv storm-0.9.0.1/ storm cd /opt/storm/lib mv commons-io-1.4.jar commons-io-1.4.jar.orig unzip -j ~/downloads/commons-io-2.4-bin.zip commons-io-2.4/commons-io-2.4.jar useradd -m storm -G supergroup chown -R storm:supergroup /opt/storm cp /opt/storm/conf/storm.yaml /opt/storm/conf/storm.yaml.orig vi /opt/storm/conf/storm.yaml ... storm.zookeeper.servers: [token1] nimbus.host: token1 supervisor.slots.ports: [6700, 6701, 6702, 6703] ui.port: 8081 ... ip_address=$(ip addr show eth0 | awk '/inet / {print $2}' | cut -d / -f 1) sed -i -e "s/token1/${ip_address}/" /opt/storm/conf/storm.yaml cd /opt/storm/bin wget https://raw.githubusercontent.com/fhd/init-script-template/master/template -O storm-nimbus ln -s /opt/storm/bin/storm-nimbus /etc/init.d/storm-nimbus wget https://raw.githubusercontent.com/fhd/init-script-template/master/template -O storm-ui ln -s /opt/storm/bin/storm-ui /etc/init.d/storm-ui wget https://raw.githubusercontent.com/fhd/init-script-template/master/template -O storm-supervisor AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-5-
AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-6-
ln -s /opt/storm/bin/storm-supervisor /etc/init.d/storm-supervisor chown -R storm:supergroup /opt/storm/bin/storm-* chmod -R 755 /opt/storm/bin/storm-* * Edit each storm-* fvi ile to start its own storm sub-process accordingly (e.g. nimbus, ui, supervisor). See example below: dir="/opt/storm/bin" user="storm" cmd="./storm nimbus" chkconfig storm-nimbus on chkconfig storm-ui on chkconfig storm-supervisor on /etc/init.d/storm-nimbus start /etc/init.d/storm-ui start /etc/init.d/storm-supervisor start * Check Admin Console Browse to: http://{HOST}:8081 (e.g. http://172.31.0.52:8081) -------------------------------------------------------------------------------------------------------------------INSTALL RABBITMQ 3.2.3 (as root) yum install erlang cd ~/downloads wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.2.3/rabbitmq-server-3.2.3-1.noarch.rpm rpm -ivH rabbitmq-server-3.2.3-1.noarch.rpm /usr/sbin/rabbitmq-plugins enable rabbitmq_management chkconfig rabbitmq-server on /etc/init.d/rabbitmq-server start *Check Admin Console Browse to: http://{HOST}:15672 (e.g. http://172.31.0.50:15672) Use Credentials: guest/guest -------------------------------------------------------------------------------------------------------------------INSTALL JETTY 8.1.15 (as root) cd ~/downloads wget http://central.maven.org/maven2/org/mortbay/jetty/dist/jetty-rpm/8.1.15.v20140411/jetty-rpm-8.1.15.v2014041 1.rpm rpm -ivh jetty-rpm-8.1.15.v20140411.rpm useradd -m jetty -G supergroup vi /etc/default/jetty ... JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk.x86_64 JETTY_HOME=/opt/jetty JETTY_USER=jetty JETTY_LOGS=/opt/jetty/logs ... AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-6-
AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-7-
mkdir -p /opt/lumify/config keytool -genkeypair -keysize 2048 -keyalg RSA -keypass password -storepass password -dname CN=lumify -keystore /opt/lumify/config/jetty.jks cp /opt/jetty/etc/jetty.xml /opt/jetty/etc/jetty.xml.orig vi /opt/jetty/etc/jetty.xml ...
/opt/lumify/config/jetty.jks password /opt/lumify/config/jetty.jks password false 300000 2 false 20000 5000 ... mkdir -p /opt/jetty/contexts-DISABLED mv /opt/jetty/contexts/* /opt/jetty/contexts-DISABLED mkdir -p /opt/jetty/webapps-DISABLED mv /opt/jetty/webapps/* /opt/jetty/webapps-DISABLED chown -R jetty:supergroup /opt/jetty* chmod -R 775 /opt/jetty* chkconfig jetty on /etc/init.d/jetty start -------------------------------------------------------------------------------------------------------------------CREATE A LUMIFY USER (as root) sed -i 's/^# %wheel/%wheel/' /etc/sudoers useradd -g supergroup -G wheel lumify passwd lumify * Use Credentials lumify/lumify cd /home/lumify mkdir .ssh chmod 700 .ssh/ cat /home/ec2-user/.ssh/authorized_keys >> .ssh/authorized_keys chmod 600 .ssh/authorized_keys chown -R lumify:supergroup .ssh/
AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-7-
AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-8-
-------------------------------------------------------------------------------------------------------------------CLONE THE LUMIFY PROJECT (as lumify) su - lumify git clone https://github.com/lumifyio/lumify.git -------------------------------------------------------------------------------------------------------------------CONFIGURE LUMIFY APPLICATION (as lumify) su - lumify sudo mkdir -p /opt/lumify/{config,lib,logs} sudo chown -R lumify:supergroup /opt/lumify sudo chmod -R g+w /opt/lumify sudo find /opt/lumify -type d -exec chmod g+s {} \; -------------------------------------------------------------------------------------------------------------------BUILD LUMIFY APPLICATION (as lumify) su - lumify cd ~/lumify cp config/{lumify.properties,log4j.xml} /opt/lumify/config ip_address=$(ip addr show eth0 | awk '/inet / {print $2}' | cut -d / -f 1) sed -i -e "s/192.168.33.10/${ip_address}/" /opt/lumify/config/lumify.properties sudo -u hdfs hadoop fs -mkdir -p /lumify/libcache sudo -u hdfs hadoop fs -chown -R lumify:supergroup /lumify sudo -u hdfs hadoop fs -chmod -R 775 /lumify mvn package -P web-war -pl web/war -am -Dmaven.test.skip=true mvn package -pl web/plugins/auth-username-only -am -Dmaven.test.skip=true mvn package -pl web/plugins/dev-tools -am -Dmaven.test.skip=true -------------------------------------------------------------------------------------------------------------------DEPLOY LUMIFY APPLICATION (as lumify) su - lumify cd ~/lumify sudo -u jetty cp web/war/target/lumify-web-war-0.4.0-SNAPSHOT.war /opt/jetty/webapps/ROOT.war sudo -u lumify cp web/plugins/auth-username-only/target/lumify-web-auth-username-only-0.4.0-SNAPSHOT.jar /opt/lumify/lib sudo -u lumify cp web/plugins/dev-tools/target/lumify-web-dev-tools-0.4.0-SNAPSHOT.jar /opt/lumify/lib *Check Lumify Console Browse to: http://{HOST}:8443 (e.g. http://172.31.0.50:8443) Use Credentials: Enter any username = password if using the lumify-web-auth-username-only plugin -------------------------------------------------------------------------------------------------------------------UPLOAD THE DEV ONTOLOGY (as lumify) su - lumify cd ~/lumify/examples zip -r ontology-dev.zip ontology-dev Browse to Lumify Application (e.g. https://172.31.0.52:8443) Enter http://lumify.io/dev as the Document IRI choose ontology-dev.zip as the OWL File AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-8-
AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-9-
click Submit -------------------------------------------------------------------------------------------------------------------CONFIGURE OPENNLP (as lumify) su - lumify cd ~/lumify hadoop fs -mkdir -p /lumify/config hadoop fs -put config/opennlp /lumify/config -------------------------------------------------------------------------------------------------------------------BUILD AND DEPLOY THE LUMIFY STORM TOPOLOGY (as lumify) su - lumify cd ~/lumify mvn package -pl storm/storm -am -Dmaven.test.skip=true * Configure Variables Exclude plugins that require optional dependencies that these instructions have not installed (see https://github.com/lumifyio/lumify/blob/master/docs/dependencies-by-feature.md)s excluded_plugins=$(cat docs/dependencies-by-feature.md | awk -F '|' '$2 ~ /text|media/ && $4 !~ /n\/a/ {print $4}' | sed -e 's/^ *//' -e 's/ *$//' -e 's| *
*| |') find_opts=$(echo $(echo $excluded_plugins) | sed -E 's/^| / ! -name /g') plugins=$(echo $(find storm/plugins -mindepth 1 -maxdepth 1 -type d ! -name target ${find_opts} ) | sed -e 's/ /,/g') mvn package -pl ${plugins} -am -Dmaven.test.skip=true jars=$(for t in $(find storm/plugins -mindepth 2 -maxdepth 2 -type d -name target); do find ${t} -name '*.jar' ! -name '*-sources.jar' | sort | tail -1; done) hadoop fs -put ${jars} /lumify/libcache /opt/storm/bin/storm jar storm/storm/target/lumify-storm-0.4.0-SNAPSHOT-jar-with-dependencies.jar io.lumify.storm.StormRunner
AWS Red Hat 6.5 Lumify Installation Guide RHEL6 CDH4 & Accumulo 1.5.1.txt
-9-
