Adding multiple content filters to an email server with postfix and dovecot Pigeonhole/Sieve

A year ago, my friend Laura was wishing that email providers could do some tone filtering and reject messages that are too mean. Since I run my own email server, I thought it might be simple to set something like that up easily. Turns out, it’s not that hard, but it wasn’t exactly trivial to figure out.

My setup

I run have postfix running to receive messages from the internet. It passes them through SpamAssassin, which inspects the messages and adds a few headers that indicate whether or not it’s spam. Then it passes them on to dovecot , which stores the messages in mailboxes and then tells with my email client, Thunderbird, that I’ve got mail. I like this setup because I feel like I have a bit more control over my data. Besides, it’s fun!

The plan

The original request is here shown below. I figure, if I could just have the message go through a second filter after it goes through spamassassin, I could make it a custom script that counts swear words.

 

text post showing the idea
The Original Idea

 

How to do multiple content filters on postfix

Learning materials include:

A lot of answers were along the lines of “Just get amavisd to do the other filter.” Maybe that’s the right answer but I’m not running amavisd and didn’t feel like setting it up just for this.

Then I stared at the postfix config for a while, in particular at this line:

That tells postfix to pipe everything off to spamc for content filtering. Then, once the filtering is done, the -e sendmail thing (according to the man pages for spamc)  sends the filtered mail to sendmail, which delivers it to dovecot for filing. Aha! I bet you can just send the processed mail to an intermediate script that does its thing and then sends it on to sendmail! Turns out this works perfectly. Here’s my script (WARNING: it has swear words in it!)

As you can see, this just expects an email to come in on stdin, checks it out, and sends it to the stdin of sendmail.

This is not particularly efficient, because it will take some time for python to process some huge attachment looking for swearwords in binary. I wouldn’t put this on a large-scale system. But the pathway works so adjust to suit your needs!

Your postfix config line changes to:

Processing in Dovecot

Now that we have a header that indicates swearwords (‘X-Too-Many-Swearwords’), we can take actions with it. The Dovecot PigeonHole extension interprets the Sieve language which is used to do stuff based on email content. I use it to filter emails server-side into folders and stuff, and to reject emails flagged as spam into my spam folder. You can actually adjust the filters directly from Thunderbird with an extension, so that’s cool (uses ManageSieve Dovecot extension too).

Here’s the user-level Pigeonhole/Sieve filter that deals with swearwords and does some other stuff. The spam filtering is in the global one, which I configured to run before this.

That last clause simply rejects the email so I never even see it and tells the author to send a new email. Nice! Totally works. Gmail marks the response as spam though… typical!

Anyway, so that was fun, if not totally useless. I had enough trouble configuring that I hope this info helps someone do something actually useful! Good luck.

The Infopanel: a simple MQTT-connected display system for weather, traffic, pictures, animations…anything!

I got one of those RGB LED matrix things for my birthday and wasn’t sure what to do with it. Then I found this awesome library which has Python bindings and can control it nicely even from a Raspberry Pi. Conveniently I had a spare Raspberry Pi 1 B+  sitting around so I hooked it up. After playing around for a while, I got the demos working.

UPDATE: Full documentation of infopanel is now available.

But I needed to connect it to home-assistant to really make it valuable to myself. So I wrote a little program called Infopanel (available for free on github) that lets you do some things:

  • Get data directly from a MQTT broker for getting live data (e.g. travel times in traffic, weather conditions) and for command and control. This allows me to connect the screen to my home-assistant home automation system.
  • Assemble various built-in elements like giraffes, animated text, rainbow text, pictures, animated gifs into various scenes that rotate through on the screen to display the information in various fun and/or useful ways.
  • There are Temperature and Duration sprites that you can define high and low values of so they’re red when they’re bad and green when they’re good, and anywhere in between.

You can set the scenes to be just random or you can control them through MQTT.

It’s intended to be very configurable but since it’s brand new some extra development is needed to make everything perfect. Send in your ideas and requests and code changes!

A relatively complete example configuration file is in the repo. That demonstrates using MQTT, connecting MQTT topics to various sprites, building your own frames of animation by hand, and adding in gifs and images from file paths. Note that you have to set an environmental variable or two to get the fonts right and whatnot.

Jeep Cherokee Laredo 1991

Note: This is a thowback post, revitalized from the old partofthething.com where it was posted by me in like, 2004. I took out most last names.

My car:

My car is the ultimate car ever. It’s the greatest thing on Earth. No, really. It is. Sure, sure it’s old but that’s not what matters anymore. It’s the past that counts…and the stereo system. I grew up in this car. Yes sir I did. I always sat in the back right seat driving all over the place since I was seven years old. Usually Tom Petty was playing. you see, it used to be my mom’s car. When she got a new one, it became my sister’s car. And then, it became my car.

Continue reading Jeep Cherokee Laredo 1991

Getting live alerts when your website is visited with Apache, MQTT, and home-assistant

I have a website or two and sometime wish I could get notifications whenever someone visited them, just for fun. Well I did it, and now I can get beeps in my home whenever anyone visits. It’s kind of cool to hear it go off, though normally it will be annoying, so we need a switch for it.

Continue reading Getting live alerts when your website is visited with Apache, MQTT, and home-assistant

Running fortune in a Python virtualenv

I learned on reddit the other day about the sudo “insults” capability where it throws shade at you when you mistype the password. I configured it everywhere I could, but I wanted more, so I came across cowsay and fortune.

$ fortune | cowsay
/ SHIFT TO THE LEFT! SHIFT TO THE RIGHT! \
\ POP UP, PUSH DOWN, BYTE, BYTE, BYTE!   /
 ----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Needless to say… wow.

Continue reading Running fortune in a Python virtualenv

Getting IP camera motion events into Home Assistant to trigger things

Oh this is exciting! I’ve been trying to figure out how to get motion events from my IP camera into my home-assistant instance running on my Raspberry Pi, and I just did a successful test! It works! Hooray. Briefly, I set up an email server on the Pi, have the camera email the Pi, have the email server trigger a script which parses the email for key words and sends MQTT signals as appropriate, at which point the home-assistant MQTT client sees them and triggers automations (like blinking a light to scare people off). Here’s how I did it.

Camera-based motion sensors in home-assistant

Continue reading Getting IP camera motion events into Home Assistant to trigger things

Adding IR send and receive capabilities to my home-assistant based smart home

Or, how to control your A/C from your phone

You may have seen my earlier post about my new home automation setup. Well I just upgraded it! I just finished testing out a new infrared remote control send and receive capability and it is pure awesome. Here’s what I can do now from within home-assistant:

  • Turn on and off my new DeLonghi portable air conditioner from Costco, and set the temperature to whatever
  • Turn on and off the TV and press any buttons from its remote control
  • Turn on and off the stereo and virtually press any buttons from its remote control (which I never even had!)
  • Use an old random blueray remote control to set scenes, turn lights on and off, etc. (basically do anything under home-assistant’s control)

This post is about how I did it on my Raspberry Pi.

Update: A video!

Annotated setup with IR sensors

Continue reading Adding IR send and receive capabilities to my home-assistant based smart home

Using an Aeotech recessed door sensor in home-assistant

I went over to my friend’s house last night to help get home-assistant on a Raspberry Pi working for his z-wave door sensors and siren. The Ecotech sensor showed up fine but the Aeotech recessed one (Recessed Door Sensor Gen5 , ZW089-A) did not show up by default. After poking around in the logs for a while, we realized that it was sending BASIC_SET command classes instead of BinarySensor commands. Apparently (I thought I had fixed this already), home-assistant doesn’t recognize the BASIC_SET commands just yet. But looking at the config file for this device, we discovered that config item 121 changes which kind of report this sensor sends:
Continue reading Using an Aeotech recessed door sensor in home-assistant

Determine your location from relative FM radio signal strengths

In this post, we’ll take a brief measurement of regular old FM radio stations and try to determine where we are. It’s like a GPS but with local FM transmitters instead of satellites. I did this just for fun. I wonder if it could be used for indoor location and stuff? It is nowhere near as accurate as GPS. But whatever.

calibrated_result1Reading FM radio signal power

The first step is to get a reading of all the nearby radio stations. I used gnuradio and a HackRF One software defined radio. A simple flow-chart that takes the FFT and dumps it to a file is all I needed. I had to throttle the I/O or else my computer would freeze. I used 16M sample rate to have as wide a bandwidth as possible.

The gnuradio flow-chart used to gather data
The gnuradio flow-chart used to gather data

The file sink saves the FFT results in 4-byte integers and just has one 1024-length vector after the other. In a few seconds, I had a 50 MB of data. I did all the post-processing in an external Python script.

Continue reading Determine your location from relative FM radio signal strengths

Home automation with Z-Wave, Home-Assistant, Aeon Multisensor, HUE lights, and a Raspberry Pi 2

I’ve always dreamed of having a “smart-home.” There’s just something cool about being able to flip switches and read sensors and have a program turn a light on when you open the door, but only if it’s dark. This post is about home automation.

UPDATE: I made a video demoing everything:

Continue reading Home automation with Z-Wave, Home-Assistant, Aeon Multisensor, HUE lights, and a Raspberry Pi 2

Sending and receiving Morse code with a laser and a Raspberry Pi

Sending and receiving text with Morse code light pulses across the room (or to your neighbor) is a fun and cheap project you can do on a Raspberry Pi or Arduino or any other microcontroller.  This post explains how I did it, and how you can do it too.

Laser blinking
It’s working!

Hardware

The hardware is simple and cheap. Here’s my parts list:

  • Raspberry Pi B+ as the controller. This does the sending, receiving, and signal processing.
  • Photoresistor – Just a little guy that has variable resistance based on how much light is hitting it
  • A 220 Ohm resistor – to make a voltage divider with the photoresistor for reading the input signal
  • MCP3008 10-bit Analog-to-Digital Converter (ADC) – Since the RPi doesn’t come with an ADC, this is required for converting analog voltage from the photoresistor into a signal I can process on the RPi.
  • Laser module – to transmit with laser light. I got one from sunfounder.com for like 3 bucks.
  • Breadboard, wires

You can learn how to use the ADC at this Adafruit tutorial. I decided to talk to the ADC with the RPi’s hardware SPI interface, which I had already enabled. I wanted to be able to go very fast. (You can alternatively do SPI off of GPIO ports with software, if you prefer.) The laser just hooks directly between a GPIO and +5V. Here’s the layout:

Wiring diagram
The wiring diagram

And what it actually looks like:

The actual laserComm hardware.
The actual laserComm hardware.

Continue reading Sending and receiving Morse code with a laser and a Raspberry Pi

Data acquisition and live-plotting over the network with a Raspberry PI and Python

Sensors are fun! I wanted to keep an eye on the temperature of one of my home physics projects and realized I needed a thermocouple and a convenient way to read it. This post is about how I got one and wrote a Python program to send the data from a Raspberry PI over the network to my laptop, which plots the results in real time.

Thermocouple in fridge
Thermocouple in fridge

I bought a K-type thermocouple for several dollars that ranges from -100°C to 500°C. The Seebeck effect that makes it work only gives microvolts per degree so I needed a good amplifier for it. And since I wanted to read it on a Raspberry PI (which only has digital inputs), I’d need an analog-to-digital converter (ADC). Conveniently, the MAX31855 does amplification and ADC and is like $15, so I got one. Sweet. There’s even a Python library for it that makes it easy to access. Double-sweet.

The MAX31855 thermocouple amp and ADC
The MAX31855 thermocouple amp and ADC

Continue reading Data acquisition and live-plotting over the network with a Raspberry PI and Python

Reading out a Crookes radiometer (light-mill) with Python and OpenCV

Since I’m on a late 1800s physics kick, I’ve been staring at my Crookes radiometer a lot.

My Crookes radiometer, or light-mill
My Crookes radiometer, or light-mill

You might remember these from the hands-on museum. Today, these are just novelty items, but back in the 1800’s, there were actually scientific instruments that could quantitatively read out the amount of incoming radiation (like visible or infrared light). They were generally read out using a calibrated spinning disk with slots in it. It’s like when you have a fan spinning in a dark room, and you adjust a strobe light until it looks like the fan isn’t spinning. Then, if you know how fast the strobe is blinking, you know how fast the object is spinning. This is called a stroboscope.

Well, I don’t have a stroboscope. But I do have a digital camera and a computer. As it turns out, I can read the Crookes radiometer using that!

Continue reading Reading out a Crookes radiometer (light-mill) with Python and OpenCV

Remote ham radio operation through a Raspberry Pi

This post is about how I set up remote operation on my ham radio through a Wifi network, over a VPN, and around the world using a Raspberry Pi, like this:

radio-link-global

This could be useful for people who travel a lot and want to be able to use their rig while away. Or for people who want to set up their radio and antenna far away from where they actually live (e.g. apartment dwellers who have friends or parents who live in the boonies and wouldn’t mind you erecting an antenna). If you want to skip the VPN and just control your ham shack from your kitchen, that’s even easier.

I did this all with an ICOM-7100 all-mode radio, a Raspberry Pi B+, a Buffalo DD-WRT router/VPN server, and a laptop running Ubuntu 15.04. This post focuses on the radio control and networked audio but not on setting up the VPN.

Continue reading Remote ham radio operation through a Raspberry Pi

You can use open-source Linux for most stuff these days

Linux is an open-source operating system that you can download and install on your computer to complement (or even replace) your current operating system (Windows or MacOS). It started out when some guy wrote some code and gave it to the world community for free. It has become huge, and now provides the engine of Android phones and tablets, runs about 40% of the servers that make up the internet, and powers 97% of the world’s supercomputers. It’s a stunning success in the social open-source experiment.

For home use, it is just now ditching a reputation of being only for nerds, scientists, and hobbyists like me, with lots of headaches getting software and hardware working nicely with it. Really, I’ve been using it at home on my laptop since 2007 and it’s evolved to a point where I think it’s totally usable for most people. It also has a focus on privacy, security, and stability. Since the code is open (everyone can see and change how and why it works), there’s less mystery about what anyone is doing with your computer and more control to get it to do what you want. It was the speed and low-resource requirements that brought me to Linux originally, and it’s the quality and open philosophy that has made me a real proponent.

My desktop
My freshly-updated Ubuntu 15.10 desktop

Continue reading You can use open-source Linux for most stuff these days

Import precedence for different types of python extensions

I was doing some speed tests with cython in Windows and started wondering what the import precedence was between py, pyc, and pyd files (not to mention pyo). So I did a simple little test with two files, test.py and test2.py. Test.py just had a print statement in it and test2 just imported test1 (so it would create the pyc). I compiled test.py with cython into test.pyd and then changed test.py to print something else.

Conclusion: The pyd file was always imported if it was present. If it wasn’t there, then the new print statement from the new test.py was printed out. I’m sure this is documented somewhere but I’m not sure where. The conclusion holds if there are “optimized” pyo files present as well.

how to fix: ctrl-c closes ipython within console2

Just FYI, I was having a problem with ipython 0.12 running within Console, where pressing Ctrl-C to cancel an operation in a Python would actually just close the whole ipython window! It was horrible, since before I reinstalled everything it wouldn’t do that. Anyway, there’s an email chain talking about this here, but it’s irrelevant to this particular problem, since I’m using mainline (non-dev repo) versions and don’t know what an ETS is.

Thankfully, there’s a very simple solution (assuming you made my same mistake). Rather than setting the console tab to C:python27scriptsipython.exe, instead set it to C:python27python.exe C:python27scriptsipython-script.py and Ctrl+C will give a KeyboardInterrupt and leave you in ipython, just like the good old days!

Building a python extension with Trilinos 10.10 on 64-bit Windows 7

Continuing the journey, the folks at work upgraded me to 64-bit Windows 7 and the Trilinos folks updated to 10.10. Needless to say, I had to rebuild. I can’t definitively say which of these steps is necessary but I can tell you that if you follow them, you can compile a C++ python extension that uses many aspects of Trilinos on this system. And not just a few packages from Trilinos either. In fact, I’m currently using all of the following:

  • Epetra
  • AztecOO
  • Anasazi
  • Teuchos
  • epetraext
  • Amesos

To do this, first run the trilinos cmake system without shared libraries. Then build. This builds things like epetra.lib. Now, go back and turn on shared libraries and build again, this time with shared libraries turned on. This builds the dlls, which I believe are required to build a python extension. If one of the ones you needs fails, just go in to Visual Studio 2008 and open up the Trilinos solution. Right click the project. Change from a static library to a dynamic librray. Change Use of MFC to “Use MFC if a Shared DLL.” Change Use of ATL to “Dynamic Link to ATL.” The rebuild the package. The DLL should be created.

Build trilinos shared library
VS 2008 options to build shared libraries

Once all the static and shared libraries are built, I didn’t even mess with environmental variables (being a grad student and all) so I just copied them all into the build directory of my Python extension. On import, it worked just fine. Amazing!

I have yet to try linking to MSMPI and running large reactor problems on Microsoft HPC but that will be coming someday.

favorite programming language jokes

These started coming up at work and people kept chiming in new ones. Here’s a list. Sorry in advance for the offensive one.

  • What’s a pirate’s favorite programming language?
  • What’s a miner’s favorite programming language?
  • What’s a hispanic’s favorite programming language?
  • What’s a diver’s favorite programming language?
  • What’s a mongoose’s favorite programming language?
  • What’s a Seattleite’s favorite programming language?
  • What’s the army’s favorite scripting language?
  • What’s the little mermaid’s favorite scripting language?
  • What’s Beethoven’s favorite programming language?
  • What’s a Frenchman’s favorite programming language?

Hilarious. Answers below.

Continue reading favorite programming language jokes