Three-node Squid proxy with Keepalived

June 24, 2017
Category: Linux
Tags: Keepalived Squid

This is a configuration with three Squid cache servers using Keepalived for HA.

01. Servers and configurations
02. Installing Squid
03. Configuring the time server
04. Creating and configuring the cache folder for squid
05. Configuring squid
06. Configuring peering squid
07. Binding to non-local interfaces
08. Installing and Configuring Keepalived
09. SARG
10. Installing vnstat
11. Installing and configuring BandwidthD
12. Proxy settings on the client servers
13. Configuring Postfix for alert sending

01. Servers and configurations

srv28 - 192.168.122.127 - squid on srv2 - CentOs 7
srv29 - 192.168.122.126 - squid on srv5 - CentOs 7
srv30 - 192.168.122.125 - squid on srv7 - CentOs 7
vip - proxy.domain.dom with keepalived - 192.168.122.124

srv28 - 192.168.122.127

Create an A record in DNS
/squid = 31 GB - xfs

srv29 - 192.168.122.126

Create an A record in DNS
/squid = 31 GB - xfs

srv30 - 192.168.122.125

Create an A record in DNS
/squid = 31 GB - xfs

On all three servers:

yum update
yum install epel-release net-tools bind-utils mailx

02. Installing Squid

Adding the repo:

/etc/yum.repos.d/squid.repo

[squid]
name=Squid repo for CentOS Linux - $basearch
#IL mirror
baseurl=http://www1.ngtech.co.il/repo/centos/$releasever/$basearch/
failovermethod=priority
enabled=1
gpgcheck=0

Then installing:

yum install squid squid-helpers

03. Configuring the time server

Configured chrony with the local time server:

/etc/chrony.conf

server 192.168.122.151 iburst

systemctl enable chronyd

04. Creating and configuring the cache folder for squid

Creating the folder:

mkdir /squid/squid-cache
chmod -R 777 /squid/squid-cache
chown -R squid:squid /squid/squid-cache

Configuring squid to use it:

/etc/squid/squid.conf

cache_dir ufs /squid/squid-cache 29696 16 256

from: Squid configuration directive cache_dir

Usage:

        cache_dir ufs Directory-Name Mbytes L1 L2 [options]

'L1' is the number of first-level subdirectories which
will be created under the 'Directory'.  The default is 16.

'L2' is the number of second-level subdirectories which
will be created under each first-level directory.  The default
is 256.

05. Configuring squid

systemctl enable squid

Add to /etc/squid/squid.conf

acl SSL_ports port 9091         # openfire admin secure access
acl SSL_ports port 10000               # webmin


visible_hostname srv28.domain.dom
dns_nameservers 192.168.122.151
append_domain .domain.dom

# added for privacy
via off
forwarded_for off

request_header_access From deny all
request_header_access Server deny all
request_header_access WWW-Authenticate deny all
request_header_access Link deny all
request_header_access Cache-Control deny all
request_header_access Proxy-Connection deny all
request_header_access X-Cache deny all
request_header_access X-Cache-Lookup deny all
request_header_access Via deny all
request_header_access X-Forwarded-For deny all
request_header_access Pragma deny all
request_header_access Keep-Alive deny all


http_port 192.168.122.124:3128

06. Configuring peering squid

On each of the three squid servers:

/etc/squid/squid.conf

On 192.168.122.127 (srv28)

cache_peer 192.168.122.126 sibling 3128 3130
cache_peer 192.168.122.125 sibling 3128 3130

# peer communication
icp_port 3130

On 192.168.122.126 (srv29):

cache_peer 192.168.122.127 sibling 3128 3130
cache_peer 192.168.122.125 sibling 3128 3130

# peer communication
icp_port 3130

On 192.168.122.125 (srv30):

cache_peer 192.168.122.127 sibling 3128 3130
cache_peer 192.168.122.126 sibling 3128 3130

# peer communication
icp_port 3130

Then:

Starting squid to create directories

squid -z

07. Binding to non-local interfaces

echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind

permanent:

/etc/sysctl.conf

net.ipv4.ip_nonlocal_bind = 1

then:

sysctl -p
net.ipv4.ip_nonlocal_bind = 1

08. Installing and Configuring Keepalived

Installing keepalived

yum install keepalived
systemctl enable keepalived

Configuring keepalived

On srv28 (192.168.122.127 - MASTER):

/etc/keepalived/keepalived.conf:

! Configuration File for keepalived

global_defs {
   notification_email {
     alerts@domain.dom
   }
   notification_email_from srv28@domain.dom
   smtp_server 192.168.122.139
   smtp_connect_timeout 30
   router_id srv28
}

vrrp_script chk_squid {
        script "killall -0 squid"
        interval 5  # squid status check intervel
        fall 4
        rise 1
}

vrrp_instance squid_clus {
        state MASTER
        interface ens160
        virtual_router_id 61
        same group
        priority 101
        smtp_alert
        virtual_ipaddress {
            192.168.122.124
        }
        track_script {
                chk_squid
        }
}

On srv29 (192.168.122.126):

/etc/keepalived/keepalived.conf:

! Configuration File for keepalived

global_defs {
   notification_email {
     alerts@domain.dom
   }
   notification_email_from srv29@domain.dom
   smtp_server 192.168.122.139
   smtp_connect_timeout 30
   router_id srv29
}

vrrp_script chk_squid {
        script "killall -0 squid"
        interval 5  # squid status check intervel
        fall 4
        rise 1
}

vrrp_instance squid_clus {
        state BACKUP
        interface ens160
        virtual_router_id 61
        same group
        priority 100
        smtp_alert
        virtual_ipaddress {
            192.168.122.124
        }
        track_script {
                chk_squid
        }
}

On srv30 (192.168.122.125):

/etc/keepalived/keepalived.conf:

! Configuration File for keepalived

global_defs {
   notification_email {
     alerts@domain.dom
   }
   notification_email_from srv30@domain.dom
   smtp_server 192.168.122.139
   smtp_connect_timeout 30
   router_id srv30
}

vrrp_script chk_squid {
        script "killall -0 squid"
        interval 5  # squid status check intervel
        fall 4
        rise 1
}

vrrp_instance squid_clus {
        state BACKUP
        interface ens160
        virtual_router_id 61
        same group
        priority 99
        smtp_alert
        virtual_ipaddress {
            192.168.122.124
        }
        track_script {
                chk_squid
        }
}

09. SARG

Installing sarg for reports

mkdir sarg
cd sarg
yum install wget
wget http://sourceforge.net/projects/sarg/files/sarg/sarg-2.3.10/sarg-2.3.10.tar.gz
tar -xzf sarg-2.3.10.tar.gz
yum install autoconf automake gd-devel pcre-devel bzip2-devel openldap-devel gettext-devel xz xz-devel perl-GD httpd php
cd sarg-2.3.10
./configure --prefix=/usr --sysconfdir=/etc/sarg --with-gd --with-iconv --disable-nls --enable-extraprotection

It was failing to install because of the version of gettext:

*** error: gettext infrastructure mismatch: using a Makefile.in.in from gettext version 0.18 but the autoconf macros are from gettext version 0.19

To fix this I went in the unzipped folder /root/sarg/sarg-2.3.10/po and in the changed the version of gettext in Makefile.in.in (two in): vi po/Makefile.in.in

# Origin: gettext-0.18
GETTEXT_MACRO_VERSION = 0.19

Then it installs normally

yum install autoconf automake gd-devel pcre-devel bzip2-devel openldap-devel gettext-devel xz xz-devel perl-GD httpd php

./configure --prefix=/usr --sysconfdir=/etc/sarg --with-gd --with-iconv --disable-nls --enable-extraprotection
make
sudo make install

Changed the squid log path in /etc/sarg/sarg.conf to

access_log /var/log/squid/access.log

and the font size for reports:

font_size 16px
header_font_size 18px
title_font_size 20px

Then executed sarg the first time: sarg -x -z

Reports

Setting up a cron job for periodic reports

Creating directories:

cd /var/www/html/squid-reports
mkdir reports_ad-hoc reports_daily reports_weekly reports_monthy

Changing the reports settings in sarg.conf :

/etc/sarg/sarg.conf

overwrite_report yes
output_dir /var/www/html/squid-reports/reports_ad-hoc

Creating scripts for scheduled reports:

/etc/cron.daily/sarg_daily

#!/bin/bash

YESTERDAY=$(date --date "1 day ago" +%d/%m/%Y)

exec /usr/bin/sarg \
    -o /var/www/html/squid-reports/reports_daily \
    -d $YESTERDAY &>/dev/null
exit 0

then: chmod +x /etc/cron.daily/sarg_daily

/etc/cron.weekly/sarg_weekly

#!/bin/bash
LOG_FILES=
for FILE in /var/log/squid/access.log*; do
    LOG_FILES="$LOG_FILES -l $FILE"
done

# Get yesterday's date
YESTERDAY=$(date --date "1 day ago" +%d/%m/%Y)

# Get one week ago date
WEEKAGO=$(date --date "7 days ago" +%d/%m/%Y)

exec /usr/bin/sarg \
    $LOG_FILES \
    -o /var/www/html/squid-reports/reports_weekly \
    -d $WEEKAGO-$YESTERDAY &>/dev/null
exit 0

then: chmod +x /etc/cron.weekly/sarg_weekly

/etc/cron.monthly/sarg_monthly

#!/bin/bash
LOG_FILES=
for FILE in /var/log/squid/access.log*; do
    LOG_FILES="$LOG_FILES -l $FILE"
done

# Get yesterday's date
YESTERDAY=$(date --date "1 day ago" +%d/%m/%Y)

# Get 1 month ago date
MONTHAGO=$(date --date "1 month ago" +%d/%m/%Y)

exec /usr/bin/sarg \
    $LOG_FILES \
    -o /var/www/html/squid-reports/reports_monthly \
    -d $MONTHAGO-$YESTERDAY &>/dev/null
exit 0

then: chmod +x /etc/cron.monthly/sarg_monthly

Modifying the index.html in /var/www/html/sarg-reports

On srv28 (192.168.122.127):

<body>
<div class="logo"><a href="http://sarg.sourceforge.net"><img src="images/sarg.png" title="SARG, Squid Analysis Report Generator. Logo by Osamu Matsuzaki" alt="Sarg"></a> Squid Analysis Report Generator</div>
<div class="title"><table cellpadding="0" cellspacing="0">
<tr><th class="title_c">Squid User Access Report on srv28 (192.168.122.127)</th></tr>
<tr><td class="data3"><a href="./reports_ad-hoc/index.html"> Ad Hoc reports </a></td></tr>
<tr><td class="data3"><a href="./reports_daily/index.html"> Daily reports </a></td></tr>
<tr><td class="data3"><a href="./reports_weekly/index.html"> Weekly reports </a></td></tr>
<tr><td class="data3"><a href="./reports_monthly/index.html"> Monthly reports </a></td></tr>
</table></div>
<div class="title"><table cellpadding="0" cellspacing="0">
<tr><th class="title_c">The Other Squid Peers</th></tr>
<tr><td class="data3"><a href="http://192.168.122.126/squid-reports" target="_blank"> srv29 (192.168.122.126) </a></td></tr>
<tr><td class="data3"><a href="http://192.168.122.125/squid-reports" target="_blank"> srv30 (192.168.122.125) </a></td></tr>
</table></div>
</body>

On srv29 (192.168.122.126):

<body>
<div class="logo"><a href="http://sarg.sourceforge.net"><img src="images/sarg.png" title="SARG, Squid Analysis Report Generator. Logo by Osamu Matsuzaki" alt="Sarg"></a> Squid Analysis Report Generator</div>
<div class="title"><table cellpadding="0" cellspacing="0">
<tr><th class="title_c">Squid User Access Report on srv29 (192.168.122.126)</th></tr>
<tr><td class="data3"><a href="./reports_ad-hoc/index.html"> Ad Hoc reports </a></td></tr>
<tr><td class="data3"><a href="./reports_daily/index.html"> Daily reports </a></td></tr>
<tr><td class="data3"><a href="./reports_weekly/index.html"> Weekly reports </a></td></tr>
<tr><td class="data3"><a href="./reports_monthly/index.html"> Monthly reports </a></td></tr>
</table></div>
<div class="title"><table cellpadding="0" cellspacing="0">
<tr><th class="title_c">The Other Squid Peers</th></tr>
<tr><td class="data3"><a href="http://192.168.122.127/squid-reports" target="_blank"> srv28 (192.168.122.127) </a></td></tr>
<tr><td class="data3"><a href="http://192.168.122.125/squid-reports" target="_blank"> srv30 (192.168.122.125) </a></td></tr>
</table></div>
</body>

On srv30 (192.168.122.125):

<body>
<div class="logo"><a href="http://sarg.sourceforge.net"><img src="images/sarg.png" title="SARG, Squid Analysis Report Generator. Logo by Osamu Matsuzaki" alt="Sarg"></a> Squid Analysis Report Generator</div>
<div class="title"><table cellpadding="0" cellspacing="0">
<tr><th class="title_c">Squid User Access Report on srv30 (192.168.122.125)</th></tr>
<tr><td class="data3"><a href="./reports_ad-hoc/index.html"> Ad Hoc reports </a></td></tr>
<tr><td class="data3"><a href="./reports_daily/index.html"> Daily reports </a></td></tr>
<tr><td class="data3"><a href="./reports_weekly/index.html"> Weekly reports </a></td></tr>
<tr><td class="data3"><a href="./reports_monthly/index.html"> Monthly reports </a></td></tr>
</table></div>
<div class="title"><table cellpadding="0" cellspacing="0">
<tr><th class="title_c">The Other Squid Peers</th></tr>
<tr><td class="data3"><a href="http://192.168.122.127/squid-reports" target="_blank"> srv28 (192.168.122.127) </a></td></tr>
<tr><td class="data3"><a href="http://192.168.122.126/squid-reports" target="_blank"> srv29 (192.168.122.126) </a></td></tr>
</table></div>
</body>

Installing mailx for email reports

yum install mailx

Now to access the reports:

http://192.168.122.127/squid-reports
http://192.168.122.126/squid-reports
http://192.168.122.125/squid-reports

10. Installing vnstat

I am installing vnstat to monitor the traffic through these servers.

yum install vnstat
systemctl enable vnstat

vnstat --iflist
Available interfaces: lo ens160 (10000 Mbit) 

vnstat --iflist
Available interfaces: lo ens160 (10000 Mbit)

/etc/vnstat.conf

Interface "ens160"

Then initializing:

vnstat -u -i ens160

Error: Unable to read database "/var/lib/vnstat/ens160": No such file or directory
Info: -> A new database has been created.
chmod -R 777 /var/lib/vnstat/ens160
systemctl start vnstat && systemctl status vnstat

11. Installing and configuring BandwidthD

Also adding BandwidthD to get more info about each client's traffic.

/etc/yum.repos.d/nux-misc

[nux-misc]
name=Misc Nux.Ro EL RPMs
baseurl=http://li.nux.ro/download/nux/misc/el$releasever/$basearch/ http://mirror.li.nux.ro/li.nux.ro/nux/misc/el$releasever/$basearch/
enabled=0
gpgcheck=1
gpgkey=http://li.nux.ro/download/nux/RPM-GPG-KEY-nux.ro

Then checking: http://192.168.122.127/bandwidthd/index.html

12. Proxy settings on the client servers

Linux - system proxy settings

/etc/environment

http_proxy=http://192.168.122.124:3128
https_proxy=http://192.168.122.124:3128
ftp_proxy=http://192.168.122.124:3128
no_proxy="localhost,127.0.0.1,.domain.dom,192.168.0.0/16"
HTTP_PROXY=http://192.168.122.124:3128
HTTPS_PROXY=http://192.168.122.124:3128
FTP_PROXY=http://192.168.122.124:3128
NO_PROXY="localhost,127.0.0.1,.domain.dom,192.168.0.0/16"

yum.conf

/etc/yum.conf

proxy=http://192.168.122.124:3128

/etc/wgetrc

http_proxy=http://192.168.122.124:3128
https_proxy=http://192.168.122.124:3128
ftp_proxy=http://192.168.122.124:3128

13. Configuring Postfix for alert sending

Added to /etc/postfix/main.cf on all three of them:

mynetworks_style = host
# Setting postfix to only send email:
myhostname = srv28.domain.dom
myorigin = $mydomain
relayhost = $mydomain
inet_interfaces = loopback-only
mydestination =

mynetworks_style = host
# Setting postfix to only send email:
myhostname = srv29.domain.dom
myorigin = $mydomain
relayhost = $mydomain
inet_interfaces = loopback-only
mydestination =

mynetworks_style = host
# Setting postfix to only send email:
myhostname = srv30.domain.dom
myorigin = $mydomain
relayhost = $mydomain
inet_interfaces = loopback-only
mydestination =