Getting live alerts when your website is visited with Apache, MQTT, and home-assistant

I have a website or two and sometime wish I could get notifications whenever someone visited them, just for fun. Well I did it, and now I can get beeps in my home whenever anyone visits. It’s kind of cool to hear it go off, though normally it will be annoying, so we need a switch for it.

Inventory: I already have set up an apache2 web server with my webpage on it, an MQTT broker, and a home-assistant based home-automation setup, hooked up to my stereo, lights, and other things. (I’m going to skip the installation of all that for this post.)

We all know you can live monitor your apache log with:

tail -f /var/log/apache2/access.log

Turns out, Apache supports Piped Logs which allow you to send your log data to anything. Let’s pipe the Apache log output to a Python script that can trigger events, while at the same time maintaining the regular old text apache log. In your sites-enabled apache config file, change the log line to this:

CustomLog "|$/usr/bin/tee -a ${APACHE_LOG_DIR}/access.website.log | /usr/local/bin/apacheLogAlert.py" combined

1	CustomLog "\|$/usr/bin/tee -a ${APACHE_LOG_DIR}/access.website.log \| /usr/local/bin/apacheLogAlert.py" combined

The script just has to monitor for regular expressions and publish to the MQTT broker when they’re found. I limited it to off-root html files for now, but will tune accordingly.

#!/usr/bin/python3
"""
Receive Apache log information and perform actions based on it. 

Set up using an apache2 piped log with tee:

"""
MOSQUITTO_LINE = 'mosquitto_pub -h myserver.com -p MYPORT -t {} -m "ON" -u USER -P PASSWORD --capath /etc/ssl/certs'

TOPIC = 'webserver/servername/hit'

import sys
import re
import subprocess
import time
import logging
import logging.handlers

LOG = logging.getLogger('ApacheMonitor')
LOG.setLevel(logging.DEBUG)
handler = logging.handlers.SysLogHandler(address = '/dev/log')
LOG.addHandler(handler)

PATTERNS = [r'GET /\S*html',
#            '"GET / HTTP/1.1" 200'
           ]

def monitor():
    
    while True:
        time.sleep(1)
        for line in sys.stdin:
            #LOG.debug('Monitoring: {}\n'.format(line))
            for pat in PATTERNS:
                if re.search(pat, line):
                    trigger()
    

def trigger():
    """
    Send MQTT message that there has been a hit.

    It should be reset by the receiver.
    """
    subprocess.call(MOSQUITTO_LINE.format(TOPIC), shell=True)
        

if __name__ == '__main__':
    LOG.debug('Beginning log monitor.')
    try:
        monitor()
    finally:
        LOG.debug('Closing log monitor.')

#!/usr/bin/python3

"""

Receive Apache log information and perform actions based on it.

Set up using an apache2 piped log with tee:

"""

MOSQUITTO_LINE = 'mosquitto_pub -h myserver.com -p MYPORT -t {} -m "ON" -u USER -P PASSWORD --capath /etc/ssl/certs'

TOPIC = 'webserver/servername/hit'

import sys

import re

import subprocess

import time

import logging

import logging.handlers

LOG = logging.getLogger('ApacheMonitor')

LOG.setLevel(logging.DEBUG)

handler = logging.handlers.SysLogHandler(address = '/dev/log')

LOG.addHandler(handler)

PATTERNS = [r'GET /\S*html',

# '"GET / HTTP/1.1" 200'

]

def monitor():

while True:

time.sleep(1)

for line in sys.stdin:

#LOG.debug('Monitoring: {}\n'.format(line))

for pat in PATTERNS:

if re.search(pat, line):

trigger()

def trigger():

"""

Send MQTT message that there has been a hit.

It should be reset by the receiver.

"""

subprocess.call(MOSQUITTO_LINE.format(TOPIC), shell=True)

if __name__ == '__main__':

LOG.debug('Beginning log monitor.')

try:

monitor()

finally:

LOG.debug('Closing log monitor.')

Note the main loop with sleeping was adapted from this SO post. You will only get a max of 1 alert per second.

Then you just have some simple configuring to do in home-assistant. Here’s my the relevant config. You just setup a MQTT client and a binary sensor based on the MQTT signal that gets flipped on by the above script. Then you make an automation that detects it flipping on, plays a sound and flips it back off, waiting for the next signal from the script. It’s awesome! You could alternatively make a sensor that measures the hit rate and turns a light redder when you are having more traffic, or anything! The possibilities are endless. With home-assistant you an also set up email alerts, SMS, etc. for other kinds of conditions, so this actually makes a really neat, if not convoluted, log monitoring setup. You can also turn the alert automation on and off with a GUI toggle on the frontend.

mqtt:
  broker: !secret remote_server
  port: !secret mqtt_port
  client_id: home-assistant
  keepalive: 60
  username: !secret user
  password: !secret mqtt_pass
  certificate: /etc/ssl/certs/DST_Root_CA_X3.pem
  protocol: 3.1
  birth_message:
    topic: 'home/status/hass'
    payload: 'online'
    qos: 1
    retain: true

binary_sensor:
  - platform: mqtt
    name: "Webpage Hit"
    state_topic: "webserver/webpage/hit"

shell_command:
  beep: mpg321 -g 50 /opt/sounds/beep.mp3
  reset_web_alert: mosquitto_pub -h server.com -p PORT -t webserver/webpage/hit -m "OFF" -u USER -P PASSWORD --capath /etc/ssl/certs

automation:
 - alias: Web traffic ding
   trigger:
     platform: state
     entity_id: binary_sensor.webpage_hit
     to: 'on'
     from: 'off'
   action:
     - service: shell_command.beep
     - service: shell_command.reset_web_alert

mqtt:

broker: !secret remote_server

port: !secret mqtt_port

client_id: home-assistant

keepalive: 60

username: !secret user

password: !secret mqtt_pass

certificate: /etc/ssl/certs/DST_Root_CA_X3.pem

protocol: 3.1

birth_message:

topic: 'home/status/hass'

payload: 'online'

qos: 1

retain: true

binary_sensor:

- platform: mqtt

name: "Webpage Hit"

state_topic: "webserver/webpage/hit"

shell_command:

beep: mpg321 -g 50 /opt/sounds/beep.mp3

reset_web_alert: mosquitto_pub -h server.com -p PORT -t webserver/webpage/hit -m "OFF" -u USER -P PASSWORD --capath /etc/ssl/certs

automation:

- alias: Web traffic ding

trigger:

platform: state

entity_id: binary_sensor.webpage_hit

to: 'on'

from: 'off'

action:

- service: shell_command.beep

- service: shell_command.reset_web_alert

You may want another one that calls reset_web_alert if it’s on for 1 minutes in case you reboot home-assistant and the MQTT topic is left in the ON position.

Filed to: Narcissism

One thought on “Getting live alerts when your website is visited with Apache, MQTT, and home-assistant”

Matthew Mansfield says:

2017-05-02 at 9:54 pm

Nick,
I’ve been working on some code so I can parse the Google Analytics “active users on site” number (from my Google Analytics reports overview account panel) and push a notification to my phone (via SMS). This would let me know if the number (of visitors) increases, so I would know if there were any new visitors to my site. I happened to come across your website during my research and wanted to pass along to you that I think your projects you’ve done are exciting! I ended up getting lost in your website here for a while reading just about everything you’ve posted!

I’ve also played around with home automation and written my own code for a voice recognition software program that I run in my office. I invite you to check out my voice recognition software post (here: http://matthewmansfield.me/index.php/2017/04/30/c/). If you would like to collaborate on projects or ideas in the future let me know!

Have a great day!

Matthew~

One thought on “Getting live alerts when your website is visited with Apache, MQTT, and home-assistant”

Leave a Reply Cancel reply