Skip to main content

HAproxy logging

By OK -

ubuntu@ubuntu:/etc/haproxy$ cat /etc/rsyslog.d/haproxy.conf
# Create an additional socket in haproxy's chroot in order to allow logging via
# /dev/log to chroot'ed HAProxy processes


$ModLoad imudp
$UDPServerRun 514


local0.* -/var/log/haproxy-0.log
local1.* -/var/log/haproxy-1.log


### keep logs in localhost ##
$AddUnixListenSocket /var/lib/haproxy/dev/log

-------------------
ubuntu@ubuntu:/etc/haproxy$ cat /etc/haproxy/haproxy.cfg
global
chroot /var/lib/haproxy
user haproxy
group haproxy
log 127.0.0.1   local0
        log 127.0.0.1   local1 notice
daemon
frontend www
    bind 192.168.40.128:8282    # haproxy public IP
    default_backend as-backend    # backend used
backend as-backend
   balance leastconn
   mode http 
server as1 192.168.40.128:8082 check    # application srv 1
defaults
log global
mode http
option httplog
option dontlognull
        contimeout 5000
        clitimeout 50000
        srvtimeout 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
listen  google  192.168.40.128:8282
        mode http
        balance roundrobin
        cookie SERVERID insert indirect
        option httpchk HEAD /index.html HTTP/1.0
        server  google_1 209.85.225.103:80 cookie google1 check
ubuntu@ubuntu:/etc/haproxy$ 

---------------------
ubuntu@ubuntu:/etc/haproxy$ curl -s -I http://192.168.40.128:8282
HTTP/1.1 301 Moved Permanently
Location: http://www.google.com:8282/
Content-Type: text/html; charset=UTF-8
Date: Sun, 25 Dec 2016 17:59:35 GMT
Expires: Tue, 24 Jan 2017 17:59:35 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 224
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: SERVERID=google1; path=/


---------------------
ubuntu@ubuntu:/etc/haproxy$ cat /var/log/haproxy*
Dec 25 09:50:20 localhost haproxy[12909]: Proxy google started.
Dec 25 09:50:36 localhost haproxy[13010]: Proxy google started.
Dec 25 09:50:41 localhost haproxy[13011]: 127.0.0.1:41481 [25/Dec/2016:09:50:40.921] google google/google_1 0/0/135/193/464 301 327 - - --NI 0/0/0/0/0 0/0 "HEAD / HTTP/1.1"
Dec 25 09:51:28 localhost haproxy[13011]: 127.0.0.1:41525 [25/Dec/2016:09:51:28.156] google google/google_1 0/0/138/198/476 301 327 - - --NI 0/0/0/0/0 0/0 "HEAD / HTTP/1.1"
Dec 25 09:51:30 localhost haproxy[13011]: 127.0.0.1:41529 [25/Dec/2016:09:51:29.727] google google/google_1 0/0/149/216/516 301 327 - - --NI 0/0/0/0/0 0/0 "HEAD / HTTP/1.1"
Dec 25 09:52:02 localhost haproxy[13011]: Server google/google_1 is DOWN, reason: Layer4 timeout, check duration: 2004ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 25 09:52:02 localhost haproxy[13011]: proxy google has no server available!
Dec 25 09:52:07 localhost haproxy[13011]: Server google/google_1 is UP, reason: Layer7 check passed, code: 302, info: "Found", check duration: 325ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Dec 25 09:54:52 localhost haproxy[13197]: Proxy google started.
Dec 25 09:55:27 localhost haproxy[13198]: 192.168.40.128:56742 [25/Dec/2016:09:55:26.822] google google/google_1 3/0/137/181/458 301 326 - - --NI 0/0/0/0/0 0/0 "HEAD / HTTP/1.1"
Dec 25 09:55:33 localhost haproxy[13198]: 192.168.40.128:56750 [25/Dec/2016:09:55:32.818] google google/google_1 0/0/139/172/451 301 326 - - --NI 0/0/0/0/0 0/0 "HEAD / HTTP/1.1"
Dec 25 09:59:35 localhost haproxy[13198]: 192.168.40.128:56996 [25/Dec/2016:09:59:35.435] google google/google_1 0/0/135/177/466 301 326 - - --NI 2/2/2/2/0 0/0 "HEAD / HTTP/1.1"
2016-12-25T09:50:20-08:00 localhost haproxy[12909]: Proxy google started.
2016-12-25T09:50:36-08:00 localhost haproxy[13010]: Proxy google started.
2016-12-25T09:52:02-08:00 localhost haproxy[13011]: Server google/google_1 is DOWN, reason: Layer4 timeout, check duration: 2004ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-12-25T09:52:02-08:00 localhost haproxy[13011]: proxy google has no server available!
2016-12-25T09:52:07-08:00 localhost haproxy[13011]: Server google/google_1 is UP, reason: Layer7 check passed, code: 302, info: "Found", check duration: 325ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
2016-12-25T09:54:52-08:00 localhost haproxy[13197]: Proxy google started.


HAProxy Logging in Ubuntu Lucid



At Transloadit we use HAProxy "The Reliable, High Performance TCP/HTTP Load Balancer" so that we can offer different services on 1 port.
For instance, depending on the hostname, a requests to port 80 can be routed to either nodejs (in case of api.transloadit.com), or nginx (in case of www.transloadit.com).
HAProxy has been good to us and setting it up was a breeze. But getting HAProxy to log on Ubuntu Lucid was harder than I thought. All of the tutorials I found either didn't cover logging, or had deprecated information on it.
Google suddenly stopped being my friend.

HAProxy Wants to Log

For performance & maintenance reasons HAProxy doesn't log directly to files. Instead it wants to log against a syslog server. This is a separate Linux daemon that most servers are equiped with already, but HAProxy requires it to listen on UDP port 514, and usually that's not enabled.
A syslog server:
  • receives log entries
  • decides what's interesting
  • writes it to disk in a highly optimized way
these aspect can all be configured by you.
If we look at the top of your current /etc/haproxy/haproxy.cfg file, we may find something like:
global
        maxconn         10000
        ulimit-n        65536
        log             127.0.0.1 local1 notice
In your backends or default config, refer to global:
defaults
    log             global
As you can see 127.0.0.1 is where it will try to find a syslog server to log to. On Unbuntu Lucid the default syslog daemon is rsyslogd, so let's make it accept HAProxy log entries.

Rsyslogd Welcomes HAProxy

Most google hits I found on logging with HAProxy told me to change the /etc/default/rsyslog file, but that's completely ignored with the new upstartsystem.  And even if you make it adhere the defaults file (yep, I tried), it will make rsyslogd go down in compatibility mode. Which is not only a shame, but also unnecessary as it turns out.
Using these config lines:
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
# Thanks Joeri Blokhuis of DongIT, pointing out that UDPServerAddress needs to
# go before UDPServerRun, or the server will run on 0.0.0.0
rsyslogd will open up it's UDP port.
Where to put these lines you say? Well, if HAProxy is the only service you need the UDP syslog port for, you could put/uncomment the lot in just one /etc/rsyslog.d/49-haproxy.conf file (Thanks to Gilles for the '49-' prefix):
# .. otherwise consider putting these two in /etc/rsyslog.conf instead:
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514

# ..and in any case, put these two in /etc/rsyslog.d/49-haproxy.conf:
local1.* -/var/log/haproxy_1.log
& ~
# & ~ means not to put what matched in the above line anywhere else for the rest of the rules
# http://serverfault.com/questions/214312/how-to-keep-haproxy-log-messages-out-of-var-log-syslog
Now do a quick:
$ restart rsyslog
And you're done. Check for HAProxy logs in:
$ tail -f /var/log/haproxy*.log
Don't forget to tweak the debug level in /etc/haproxy/haproxy.cfg, and maybe set up a logrotate right away in /etc/logrotate.d/haproxy:
/var/log/haproxy*.log
{
    rotate 4
    weekly
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        reload rsyslog >/dev/null 2>&1 || true
    endscript
}
Happy logging!

Comments

Post a Comment

Popular posts from this blog

NFS mount add in fstab _netdev instead of default | firewall-cmd --list-all

By OK -
Redhat recommends us to add  _netdev  option for NFS file system in /etc/fstab. So that they can be mounted after starting the networking service. Alternative ways to refer to partitions: Label : LABEL=label Network ID Samba : //server/share NFS : server:/share SSHFS : sshfs#user@server:/share Device : /dev/sdxy (not recommended) Example #1 //192.168.1.134/share_name /mnt/share_name cifs username=server_user,password=server_password,_netdev 0 0 ^ Path to your share ^ Mountpoint ^ Username ^ Password ^ See note A Note A: The  _netdev  options makes sure the mount is only attempted AFTER a network connection has been established. Example #2 10.10.10.1:/vol1/fs1 /data nfs defaults,_netdev 0 0 ============================ [root@localhost ~]# firewall-cmd --list-all public   target: default   icmp-block-inversion: no   interfaces:   sources:   services: dhcpv6-client high-availability ssh   ports:   protocols:   masquera
Read more

teamcity Automatic Agent Start under Linux

By OK -
Automatic Agent Start under Linux To run agent automatically on the machine boot under Linux, configure daemon process with the  agent.sh start  command to start it and  agent.sh stop  command to stop it. Refer to an example procedure below: 1. Navigate to the services start/stop services scripts directory: cd /etc/init.d/ 2. Open the build agent service script: sudo vim buildAgent 3. Paste the following into the file : #!/bin/sh ### BEGIN INIT INFO # Provides: TeamCity Build Agent # Required-Start: $remote_fs $syslog # Required-Stop: $remote_fs $syslog # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Start build agent daemon at boot time # Description: Enable service provided by daemon. ### END INIT INFO #Provide the correct user name: USER="agentuser" case "$1" in start) su - $USER -c "cd BuildAgent/bin ; ./agent.sh start" ;; stop) su - $USER -c "cd BuildAgent/bin ; ./agent.
Read more