1. Home
  2. Notes
  3. Building a Graphite Server on Debian 11

Building a Graphite Server on Debian 11

Trying to build a Graphite server with custom location for install. The steps in the Graphite documentation are not working for me.
It installs most of the things in /usr/local and some other paths. So I came up with some other steps of installing it and it seems to be working well until now.

The installtion will be in /data/graphite on a dedicated drive. It will use Python virtual environment at that location.

Using

  • Debian 11
  • Python 3.9 (from debian)
  • Postgres 13 for Django database
  • Graphite installed with pip from the git repo master

Installing and configuring postgres 

apt install postgresql postgresql-client postgresql-contrib

Changing the password for the postgres database user:

su - postgres
psql -d template1 -c "ALTER USER postgres WITH PASSWORD 'PostgresPassword';"

/etc/postgresql/13/main/pg_hba.conf

# To allow remote connection
host    all         all         192.168.122.0/24    trust
host    all         all         192.168.123.0/24    trust

/etc/postgresql/13/main/postgresql.conf

Adding

listen_addresses = '*'

Then

systemctl enable postgresql
systemctl start postgresql && systemctl status postgresql

pg_isready

Creating the database for graphite-web 

su - postgres 
psql

CREATE DATABASE graphite;
CREATE USER graphite WITH PASSWORD 'GraphiteDBPassword';
ALTER ROLE graphite SET client_encoding TO 'utf8';
ALTER ROLE graphite SET default_transaction_isolation TO 'read committed';
ALTER ROLE graphite SET timezone TO 'America/Toronto';
GRANT ALL PRIVILEGES ON DATABASE graphite TO graphite;

Installing dependencies for graphite 

apt install python3-dev python3-pip libcairo2-dev libffi-dev build-essential

Installing graphite 

apt install git -y
apt install python3-venv

cd /data
python3 -m venv graphite
cd graphite
source bin/activate

pip install wheel

Installing psycopg2

apt install libpq-dev

pip install psycopg2

mkdir downloads
cd downloads

git clone https://github.com/graphite-project/whisper.git
git clone https://github.com/graphite-project/carbon.git
git clone https://github.com/graphite-project/graphite-web.git

Modifying the setup.py files for carbon and graphite-web.

For carbon setup.py commenting out

if os.environ.get('GRAPHITE_NO_PREFIX'):
    cf.remove_section('install')
else:
    print('#' * 80)
    print('')
    print('Carbon\'s default installation prefix is "/opt/graphite".')
    print('')
    print('To install Carbon in the Python\'s default location run:')
    print('$ GRAPHITE_NO_PREFIX=True python setup.py install')
    print('')
    print('#' * 80)
    try:
        cf.add_section('install')
    except DuplicateSectionError:
        pass
    if not cf.has_option('install', 'prefix'):
        cf.set('install', 'prefix', '/opt/graphite')
    if not cf.has_option('install', 'install-lib'):
        cf.set('install', 'install-lib', '%(prefix)s/lib')

with open('setup.cfg', 'w') as f:
    cf.write(f)

For graphite-web setup.py file commenting out

if os.environ.get('GRAPHITE_NO_PREFIX') or os.environ.get('READTHEDOCS'):
    cf.remove_section('install')
else:
    try:
        cf.add_section('install')
    except DuplicateSectionError:
        pass
    if not cf.has_option('install', 'prefix'):
        cf.set('install', 'prefix', '/opt/graphite')
    if not cf.has_option('install', 'install-lib'):
        cf.set('install', 'install_lib', '%(prefix)s/webapp')

with open('setup.cfg', 'w') as f:
    cf.write(f)

Then from within the downloads directory, installing all three components

pip install ./whisper
pip install ./carbon
pip install ./graphite-web

In webapp directory, creating a symlink to graphite

cd ../webapp
ln -s /data/graphite/lib/python3.9/site-packages/graphite graphite

Configuring carbon

Working in /data/graphite/conf directory

cd ../conf

cp carbon.conf.example carbon.conf
cp storage-schemas.conf.example storage-schemas.conf
cp storage-aggregation.conf.example storage-aggregation.conf
cp relay-rules.conf.example relay-rules.conf

Creating user and group carbon

groupadd -g 497 carbon
useradd -c "carbon user" -g 497 -u 497 -s /bin/false carbon

carbon.conf

LOCAL_DATA_DIR = /data/graphite/storage/whisper/

Creating a startup script for carbon

For systemd

/etc/systemd/system/carbon-cache.service

[Unit]
Description=Graphite Carbon Cache
After=network.target

[Service]
Type=forking
ExecStart=/data/graphite/bin/carbon-cache.py --config=/data/graphite/conf/carbon.conf --pidfile=/var/run/carbon-cache.pid start
ExecReload=/bin/kill -USR1 $MAINPID
PIDFile=/var/run/carbon-cache.pid

[Install]
WantedBy=multi-user.target

Configuring graphite-web 

pip install gunicorn

systemd files

# This is /etc/systemd/system/graphite-web.socket
[Unit]
Description=graphite-web socket

[Socket]
ListenStream=/run/graphite-api.sock
ListenStream=127.0.0.1:8080

[Install]
WantedBy=sockets.target

# This is /etc/systemd/system/graphite-web.service
[Unit]
Description=graphite-web service
Requires=graphite-web.socket

[Service]
ExecStart=/data/graphite/bin/gunicorn wsgi --pythonpath=/data/graphite/webapp/graphite --bind 127.0.0.1:8080
Restart=on-failure
#User=graphite
#Group=graphite
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

local_settings.py

(graphite) root@graphite-01:/data/graphite# cd ../webapp/graphite
(graphite) root@graphite-01:/data/graphite/webapp/graphite# cp local_settings.py.example local_settings.py

SECRET_KEY = 'secretkey'
ALLOWED_HOSTS = [ '*' ]
TIME_ZONE = 'America/Toronto'

LOG_ROTATION = True
LOG_ROTATION_COUNT = 1
LOG_RENDERING_PERFORMANCE = True
LOG_CACHE_PERFORMANCE = True

GRAPHITE_ROOT = '/data/graphite'

CONF_DIR = '/data/graphite/conf'
STORAGE_DIR = '/data/graphite/storage'
STATIC_ROOT = '/data/graphite/static'
LOG_DIR = '/data/graphite/storage/log/webapp'
INDEX_FILE = '/data/graphite/storage/index'

DASHBOARD_CONF = '/data/graphite/conf/dashboard.conf'
GRAPHTEMPLATES_CONF = '/data/graphite/conf/graphTemplates.conf'

CERES_DIR = '/data/graphite/storage/ceres'
WHISPER_DIR = '/data/graphite/storage/whisper'
RRD_DIR = '/data/graphite/storage/rrd'
STANDARD_DIRS = [WHISPER_DIR, RRD_DIR]

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'graphite',
        'USER': 'graphite',
        'PASSWORD': 'GraphiteDBPassword',
        'HOST': 'localhost',
        'PORT': '',
    }
}

mkdir /data/graphite/storage/whisper
mkdir /data/graphite/storage/rrd
mkdir /data/graphite/storage/ceres
mkdir /data/graphite/storage/index
mkdir /data/graphite/static

Doing migrations

(graphite) root@graphite-01:/data/graphite/webapp/graphite# cd ../../
(graphite) root@graphite-01:/data/graphite# pwd
/data/graphite
(graphite) root@graphite-01:/data/graphite#

(graphite) root@graphite-01:/data/graphite# mkdir -p storage/log/webapp
(graphite) root@graphite-01:/data/graphite# touch storage/log/webapp/info.log

(graphite) root@graphite-01:/data/graphite# django-admin migrate --settings=graphite.settings

Installing and configuring nginx 

apt install nginx -y

touch /var/log/nginx/graphite.access.log
touch /var/log/nginx/graphite.error.log
chmod 640 /var/log/nginx/graphite.*
chown www-data:www-data /var/log/nginx/graphite.*

/etc/nginx/sites-available/graphite

upstream graphite {
    server 127.0.0.1:8080 fail_timeout=0;
}

server {
    listen 80 default_server;

    server_name graphite-01.marthemed.org 192.168.122.120;

    root /data/graphite/webapp;

    access_log /var/log/nginx/graphite.access.log;
    error_log  /var/log/nginx/graphite.error.log;

    location = /favicon.ico {
        return 204;
    }

    # serve static content from the "content" directory
    location /static {
        alias /data/graphite/webapp/content;
        expires max;
    }

    location / {
        try_files $uri @graphite;
    }

    location @graphite {
        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_connect_timeout 10;
        proxy_read_timeout 10;
        proxy_pass http://graphite;
    }
}

ln -s /etc/nginx/sites-available/graphite /etc/nginx/sites-enabled
rm -f /etc/nginx/sites-enabled/default

systemctl restart nginx

Other steps 

systemctl daemon-reload
systemctl enable carbon-cache
systemctl enable graphite-web
systemctl enable nginx

Testing

Let's do some test using graphitesend

mkdir testing_graphite
cd testing_graphite/
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install wheel

pip install graphitesend

storage-schemas.conf already has a catch-all definition

[default]
pattern = .*
retentions = 60s:1d,5m:30d,1h:3y

Using some scripts (inspired from the examples from graphitesend)

import time
import graphitesend
import socket

server_hostname = socket.gethostname()

g = graphitesend.init(prefix="testing", \
    system_name=server_hostname, \
    group='loadavg_', \
    suffix='min', \
    graphite_server='192.168.122.120', \
    graphite_port=2003)

while True:
    (la1, la5, la15) = open('/proc/loadavg').read().strip().split()[:3]
    print(g.send_dict({'1': la1, '5': la5, '15': la15}))
    time.sleep(1)

import time
import graphitesend
import socket

server_hostname = socket.gethostname()

g = graphitesend.init(prefix="testing", \
    system_name=server_hostname, \
    group='netstat.', \
    lowercase_metric_names=True, \
    graphite_server='192.168.122.120', \
    graphite_port=2003)

while True:
    lines = open('/proc/net/netstat').readlines()
    tcp_metrics = lines[0].split()[1:]
    tcp_values = lines[1].split()[1:]
    ip_metrics = lines[2].split()[1:]
    ip_values = lines[3].split()[1:]

    data_list = zip(tcp_metrics + ip_metrics, tcp_values + ip_values)
    print(g.send_list(data_list))
    time.sleep(1)