POC Prerequisites
This page describe which packages are needed for the POC, how install and configure them.
Prerequisites
System
yum -y install epel-release
yum update
yum upgrade
vi nano /etc/selinux/config
#Change SELINUX value to
SELINUX=disabled
systemctl stop firewalld
systemctl disable firewalld
reboot
Elastic repo
Download and install Elastic public signing key :
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Create a new yum repo file and add the following lines :
vi /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
Java
Installation
sudo yum install java-1.8.0-openjdk
Elasticsearch
Installation
yum install elasticsearch
Configuration
Modify/add the following lines in Elasticsearch configuration file :
sudo vi /etc/elasticsearch/elasticsearch.yml
cluster.name: BOTES
node.name: Glooper
# If you have to change default data path
# Don't forget to change permissions for data folder
# chown -R elasticsearch:elasticsearch /opt/data/elasticsearch/
path.data: /opt/data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: localhost
http.port: 9200
cluster.initial_master_nodes: ["$Your_Server_IP"]
Configure system and Elasticsearch for JVM memory usage :
sudo vi /etc/elasticsearch/jvm.options
-Xms2g
# Xmx must but set no more than 50% of total memory and no more than 32Gb
-Xmx2g
sudo vi /etc/security/limits.conf
# Add the following lines
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
sudo vi /etc/sysconfig/elasticsearch
# Modify the folloing lines
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited
sudo vi /usr/lib/systemd/system/elasticsearch.service
# Add the following line
LimitMEMLOCK=infinity
Then start and enable Elasticsearch :
systemctl start elasticsearch
systemctl enable elasticsearch
Logstash
Installation
yum install logstash
Configuration
Logstash output configuration for Kafka can be downloaded here : https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/output-kafka/output-kafka.conf
Logstash input configurations for BOTES JSON files can be downloaded on Logstash Configuration section here: BOTES Prerequisites.
Then start and enable Logstash :
systemctl start logstash
systemctl enable logstash
Kibana
Installation
yum install kibana
Configuration
Modify/add the following lines in Kibana configuration file :
sudo vi /etc/kibana/kibana.yml
server.port: 5601
server.host: "$Your_Server_IP"
server.name: "BOTES"
elasticsearch.hosts: ["http://localhost:9200"]
# Node can be slow if in Raspberry Pi for example
elasticsearch.requestTimeout: 300000
Then start and enable Kibana :
systemctl start kibana
systemctl enable kibana
Zookeeper
Installation
cd /opt
wget http://apache-mirror.8birdsvideo.com/zookeeper/stable/apache-zookeeper-3.5.5.tar.gz
tar zxf apache-zookeeper-3.5.5.tar.gz
rm -f apache-zookeeper-3.5.5.tar.gz
ln -s /opt/apache-zookeeper-3.5.5/ /opt/zookeeper
mkdir /opt/zookeeper/logs
mkdir /opt/zookeeper/data
sudo useradd zk -m
sudo usermod --shell /bin/bash zk
sudo usermod -aG sudo zk
sudo chown -R zk:zk /opt/zookeeper/logs/
sudo chown -R zk:zk /opt/zookeeper/data/
Configuration
Create and add the following lines to Zookeeper configuration file :
vi /opt/zookeeper/conf/server.configuration
tickTime=2000
dataDir=/opt/zookeeper/data
dataLogDir/opt/zookeeper/logs
clientPort=2181
clientPortAddress=localhost
Systemd script
Create and add the following lines to Zookeeper systemd script :
[Unit]
Description=Zookeeper Daemon
Documentation=http://zookeeper.apache.org
Requires=network.target
After=network.target
[Service]
Type=forking
WorkingDirectory=/opt/zookeeper
User=zk
Group=zk
ExecStart=/opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/server.configuration
ExecStop=/opt/zookeeper/bin/zkServer.sh stop /opt/zookeeper/conf/server.configuration
ExecReload=/opt/zookeeper/bin/zkServer.sh restart /opt/zookeeper/conf/server.configuration
TimeoutSec=30
Restart=on-failure
[Install]
WantedBy=default.target
Then start and enable Zookeeper :
systemctl start zookeeper
systemctl enable zookeeper
Kafka
Installation
cd /opt
wget http://apache-mirror.8birdsvideo.com/kafka/2.3.0/kafka_2.11-2.3.0.tgz
tar zxf kafka_2.11-2.3.0.tgz
rm -f kafka_2.11-2.3.0.tgz
ln -s /opt/kafka_2.11-2.3.0/ /opt/kafka
Configuration
Modify/add the following lines in Kafka configuration file :
vi /opt/kafka/config/server.properties
broker.id=0
listeners=PLAINTEXT://localhost:9092
log.retention.hours=24
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
Systemd script
sudo vi /etc/systemd/system/kafka.service
[Unit]
Description=Apache Kafka server (broker)
Documentation=http://kafka.apache.org/documentation.html
Requires=network.target remote-fs.target
After=network.target remote-fs.target zookeeper.service
[Service]
Type=simple
User=zk
Group=zk
Environment=JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
[Install]
WantedBy=multi-user.target
Then start and enable Kafka :
systemctl start kafka
systemctl enable kafka
Maven
Installation
cd /opt
wget http://apache.mirrors.ionfish.org/maven/maven-3/3.6.2/binaries/apache-maven-3.6.2-bin.tar.gz
tar xzf apache-maven-3.6.2-bin.tar.gz
rm -f apache-maven-3.6.2-bin.tar.gz
ln -s /opt/apache-maven-3.6.2/ /opt/maven
export M2_HOME=/opt/maven
export PATH=${M2_HOME}/bin:${PATH}
Flink
Installation
cd /opt
wget http://apache-mirror.8birdsvideo.com/flink/flink-1.9.0/flink-1.9.0-bin-scala_2.11.tgz
tar zxf flink-1.9.0-bin-scala_2.11.tgz
rm -f flink-1.9.0-bin-scala_2.11.tgz
ln -s /opt/flink-1.9.0/ /opt/flink
export FLINK_HOME=/opt/flink/
export PATH=$PATH:$FLINK_HOME/bin
Redis
Installation
yum install redis
Configuration
Modify/add the following lines in Redis configuration file :
cp /etc/redis.conf /etc/redis.conf.orig
sudo vi /etc/redis.conf
bind 127.0.0.1
port 6379
Modify system parameters :
sudo vi /etc/sysctl.conf
vm.overcommit_memory=1
# Then launch
echo never > /sys/kernel/mm/transparent_hugepage/enabled
Then start and enable Redis :
systemctl start redis
systemctl enable redis
Last updated
Was this helpful?