POC Prerequisites

This page describe which packages are needed for the POC, how install and configure them.

Note: For this demo, infrastructure is pretty small and simple, there is absolutely no security or reliability functionnalities configured. Don't use those configurations in production.

Demo has been tested on the latest CentOS 8 version (currently 8.0-1905).

Prerequisites

System

yum -y install epel-release
yum update
yum upgrade

vi nano /etc/selinux/config
#Change SELINUX value to
SELINUX=disabled

systemctl stop firewalld
systemctl disable firewalld

reboot

Elastic repo

Download and install Elastic public signing key :

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Create a new yum repo file and add the following lines :

vi /etc/yum.repos.d/elasticsearch.repo

[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Java

Installation

sudo yum install java-1.8.0-openjdk

Elasticsearch

Installation

yum install elasticsearch

Configuration

Modify/add the following lines in Elasticsearch configuration file :

sudo vi /etc/elasticsearch/elasticsearch.yml


cluster.name: BOTES
node.name: Glooper
# If you have to change default data path 
# Don't forget to change permissions for data folder
# chown -R elasticsearch:elasticsearch /opt/data/elasticsearch/
path.data: /opt/data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: localhost
http.port: 9200
cluster.initial_master_nodes: ["$Your_Server_IP"]

Configure system and Elasticsearch for JVM memory usage :

sudo vi /etc/elasticsearch/jvm.options


-Xms2g
# Xmx must but set no more than 50% of total memory and no more than 32Gb
-Xmx2g

sudo vi /etc/security/limits.conf

# Add the following lines
elasticsearch   soft  memlock   unlimited
elasticsearch   hard  memlock   unlimited


sudo vi /etc/sysconfig/elasticsearch

# Modify the folloing lines
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited


sudo vi /usr/lib/systemd/system/elasticsearch.service

# Add the following line
LimitMEMLOCK=infinity

Then start and enable Elasticsearch :

systemctl start elasticsearch
systemctl enable elasticsearch

Logstash

Installation

yum install logstash

Configuration

Logstash output configuration for Kafka can be downloaded here : https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/output-kafka/output-kafka.conf

Logstash input configurations for BOTES JSON files can be downloaded on Logstash Configuration section here: BOTES Prerequisites.

Then start and enable Logstash :

systemctl start logstash
systemctl enable logstash

Kibana

Installation

yum install kibana

Configuration

Modify/add the following lines in Kibana configuration file :

sudo vi /etc/kibana/kibana.yml


server.port: 5601
server.host: "$Your_Server_IP"
server.name: "BOTES"
elasticsearch.hosts: ["http://localhost:9200"]
# Node can be slow if in Raspberry Pi for example
elasticsearch.requestTimeout: 300000

Then start and enable Kibana :

systemctl start kibana
systemctl enable kibana

Zookeeper

Installation

cd /opt
wget http://apache-mirror.8birdsvideo.com/zookeeper/stable/apache-zookeeper-3.5.5.tar.gz
tar zxf apache-zookeeper-3.5.5.tar.gz
rm -f apache-zookeeper-3.5.5.tar.gz
ln -s /opt/apache-zookeeper-3.5.5/ /opt/zookeeper

mkdir /opt/zookeeper/logs
mkdir /opt/zookeeper/data

sudo useradd zk -m
sudo usermod --shell /bin/bash zk
sudo usermod -aG sudo zk
sudo chown -R zk:zk /opt/zookeeper/logs/
sudo chown -R zk:zk /opt/zookeeper/data/

Configuration

Create and add the following lines to Zookeeper configuration file :

vi /opt/zookeeper/conf/server.configuration


tickTime=2000
dataDir=/opt/zookeeper/data
dataLogDir/opt/zookeeper/logs
clientPort=2181
clientPortAddress=localhost

Systemd script

Create and add the following lines to Zookeeper systemd script :

[Unit]
Description=Zookeeper Daemon
Documentation=http://zookeeper.apache.org
Requires=network.target
After=network.target


[Service]
Type=forking
WorkingDirectory=/opt/zookeeper
User=zk
Group=zk
ExecStart=/opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/server.configuration
ExecStop=/opt/zookeeper/bin/zkServer.sh stop /opt/zookeeper/conf/server.configuration
ExecReload=/opt/zookeeper/bin/zkServer.sh restart /opt/zookeeper/conf/server.configuration
TimeoutSec=30
Restart=on-failure


[Install]
WantedBy=default.target

sudo vi /etc/systemd/system/zookeeper.service


[Unit]
Description=Zookeeper Daemon
Documentation=http://zookeeper.apache.org
Requires=network.target
After=network.target


[Service]
Type=forking
WorkingDirectory=/opt/zookeeper
User=zk
Group=zk
ExecStart=/opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/server.configuration
ExecStop=/opt/zookeeper/bin/zkServer.sh stop /opt/zookeeper/conf/server.configuration
ExecReload=/opt/zookeeper/bin/zkServer.sh restart /opt/zookeeper/conf/server.configuration
TimeoutSec=30
Restart=on-failure


[Install]
WantedBy=default.target

Then start and enable Zookeeper :

systemctl start zookeeper
systemctl enable zookeeper

Kafka

Installation

cd /opt
wget http://apache-mirror.8birdsvideo.com/kafka/2.3.0/kafka_2.11-2.3.0.tgz
tar zxf kafka_2.11-2.3.0.tgz
rm -f kafka_2.11-2.3.0.tgz
ln -s /opt/kafka_2.11-2.3.0/ /opt/kafka

Configuration

Modify/add the following lines in Kafka configuration file :

vi /opt/kafka/config/server.properties


broker.id=0
listeners=PLAINTEXT://localhost:9092
log.retention.hours=24
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000

Systemd script

sudo vi /etc/systemd/system/kafka.service


[Unit]
Description=Apache Kafka server (broker)
Documentation=http://kafka.apache.org/documentation.html
Requires=network.target remote-fs.target
After=network.target remote-fs.target zookeeper.service


[Service]
Type=simple
User=zk
Group=zk
Environment=JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh


[Install]
WantedBy=multi-user.target

Then start and enable Kafka :

systemctl start kafka
systemctl enable kafka

Maven

Installation

cd /opt
wget http://apache.mirrors.ionfish.org/maven/maven-3/3.6.2/binaries/apache-maven-3.6.2-bin.tar.gz
tar xzf apache-maven-3.6.2-bin.tar.gz
rm -f apache-maven-3.6.2-bin.tar.gz
ln -s /opt/apache-maven-3.6.2/ /opt/maven
export M2_HOME=/opt/maven
export PATH=${M2_HOME}/bin:${PATH}

Flink

Installation

cd /opt
wget http://apache-mirror.8birdsvideo.com/flink/flink-1.9.0/flink-1.9.0-bin-scala_2.11.tgz
tar zxf flink-1.9.0-bin-scala_2.11.tgz
rm -f flink-1.9.0-bin-scala_2.11.tgz
ln -s /opt/flink-1.9.0/ /opt/flink
export FLINK_HOME=/opt/flink/
export PATH=$PATH:$FLINK_HOME/bin

Redis

Installation

yum install redis

Configuration

Modify/add the following lines in Redis configuration file :

cp /etc/redis.conf /etc/redis.conf.orig
sudo vi /etc/redis.conf


bind 127.0.0.1
port 6379

Modify system parameters :

sudo vi /etc/sysctl.conf

vm.overcommit_memory=1


# Then launch
echo never > /sys/kernel/mm/transparent_hugepage/enabled

Then start and enable Redis :

systemctl start redis
systemctl enable redis

PreviousStreaming architecture NextPOC Code

Last updated 5 years ago

Was this helpful?