{"id":1318,"date":"2017-05-21T16:41:36","date_gmt":"2017-05-21T23:41:36","guid":{"rendered":"https:\/\/partofthething.com\/thoughts\/?p=1318"},"modified":"2017-07-07T20:15:37","modified_gmt":"2017-07-08T03:15:37","slug":"adding-multiple-content-filters-to-an-email-server-with-postfix-and-dovecot-pigeonholesieve","status":"publish","type":"post","link":"https:\/\/partofthething.com\/thoughts\/adding-multiple-content-filters-to-an-email-server-with-postfix-and-dovecot-pigeonholesieve\/","title":{"rendered":"Adding multiple content filters to an email server with postfix and dovecot Pigeonhole\/Sieve"},"content":{"rendered":"<p>A year ago, my friend Laura was wishing that email providers could do some tone filtering and reject messages that are too mean. Since I run my own email server, I thought it might be simple to set something like that up easily. Turns out, it&#8217;s not that hard, but it wasn&#8217;t exactly trivial to figure out.<\/p>\n<h3>My setup<\/h3>\n<p>I run have postfix running to receive messages from the internet. It passes them through SpamAssassin, which inspects the messages and adds a few headers that indicate whether or not it&#8217;s spam. Then it passes them on to dovecot , which stores the messages in mailboxes and then tells with my email client, Thunderbird, that I&#8217;ve got mail. I like this setup because I feel like I have a bit more control over my data. Besides, it&#8217;s fun!<\/p>\n<h3>The plan<\/h3>\n<p>The original request is here shown below. I figure, if I could just have the message go through a second filter after it goes through spamassassin, I could make it a custom script that counts swear words.<\/p>\n<p><!--more--><\/p>\n<figure id=\"attachment_1321\" aria-describedby=\"caption-attachment-1321\" style=\"width: 709px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/partofthething.com\/thoughts\/wp-content\/uploads\/laura-holewa-server-reject-idea-swearfilter.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-1321 size-full\" src=\"https:\/\/partofthething.com\/thoughts\/wp-content\/uploads\/laura-holewa-server-reject-idea-swearfilter.png\" alt=\"text post showing the idea\" width=\"709\" height=\"529\" srcset=\"https:\/\/partofthething.com\/thoughts\/wp-content\/uploads\/laura-holewa-server-reject-idea-swearfilter.png 709w, https:\/\/partofthething.com\/thoughts\/wp-content\/uploads\/laura-holewa-server-reject-idea-swearfilter-300x224.png 300w\" sizes=\"auto, (max-width: 709px) 100vw, 709px\" \/><\/a><figcaption id=\"caption-attachment-1321\" class=\"wp-caption-text\">The Original Idea<\/figcaption><\/figure>\n<p>&nbsp;<\/p>\n<h3>How to do multiple content filters on postfix<\/h3>\n<p>Learning materials include:<\/p>\n<ul>\n<li><a href=\"http:\/\/www.postfix.org\/FILTER_README.html\">Postfix After-Queue Content Filtering<\/a><\/li>\n<li><a href=\"https:\/\/www.thecodingmachine.com\/triggering-a-php-script-when-your-postfix-server-receives-a-mail\/\">Dude who triggered a script via email but didn&#8217;t forward it to the mailbox<\/a><\/li>\n<li><a href=\"https:\/\/lists.amavis.org\/pipermail\/amavis-users\/2016-March\/004099.html\">Dude on maillist who asked the right question but got mediocre answers<\/a><\/li>\n<li><a href=\"http:\/\/postfix.1071664.n5.nabble.com\/multiple-content-filter-settings-td42296.html\">Another person who tried to figure it out<\/a><\/li>\n<\/ul>\n<p>A lot of answers were along the lines of &#8220;Just get amavisd to do the other filter.&#8221; Maybe that&#8217;s the right answer but I&#8217;m not running amavisd and didn&#8217;t feel like setting it up just for this.<\/p>\n<p>Then I stared at the postfix config for a while, in particular at this line:<\/p>\n<pre class=\"remarkup-code\">spamassassin unix -     n       n       -       -       pipe user=spamd argv=\/usr\/bin\/spamc -f -e  \/usr\/sbin\/sendmail -oi -f ${sender} ${recipient}<\/pre>\n<p>That tells postfix to pipe everything off to spamc for content filtering. Then, once the filtering is done, the -e sendmail thing (according to the man pages for spamc)\u00a0 sends the filtered mail to sendmail, which delivers it to dovecot for filing. Aha! I bet you can just send the processed mail to an intermediate script that does its thing and then sends it on to sendmail! Turns out this works perfectly. Here&#8217;s my script (<span style=\"color: #ff0000;\">WARNING<\/span>: it has swear words in it!)<\/p>\n<pre class=\"lang:python decode:true\" title=\"My custom filter\">#!\/usr\/bin\/python3\r\n\"\"\"\r\nCheck some stuff and modify header of emails coming in from the output of SpamAssassin.\r\n\r\nThis is for Laura H's swearword detecting email system. \r\n\r\n\"\"\"\r\nimport email\r\nimport sys\r\nimport subprocess\r\n\r\nsender, recipient = sys.argv[1], sys.argv[2]\r\n\r\nsys.stdin = sys.stdin.detach()\r\nmsg = email.message_from_binary_file(sys.stdin)\r\nmsg_st = msg.as_string()\r\n\r\ncount = 0\r\nfor swearword in ['shit','fuck','cunt','bitch','bastard']:\r\n   count += msg_st.count(swearword)\r\nif count&gt;10:\r\n    msg.add_header('X-Too-Many-Swearwords', 'YES')\r\n\r\n# relay the modified message back to postfix for delivery to dovecot\r\np = subprocess.Popen(['\/usr\/sbin\/sendmail', '-oi', '-f', sender, recipient], stdin=subprocess.PIPE, stderr=subprocess.PIPE)\r\nstdout = p.communicate(input=msg.as_bytes())\r\n<\/pre>\n<p>As you can see, this just expects an email to come in on stdin, checks it out, and sends it to the stdin of sendmail.<\/p>\n<p>This is not particularly efficient, because it will take some time for python to process some huge attachment looking for swearwords in binary. I wouldn&#8217;t put this on a large-scale system. But the pathway works so adjust to suit your needs!<\/p>\n<p>Your postfix config line changes to:<\/p>\n<pre class=\"remarkup-code\">spamassassin unix -     n       n       -       -       pipe user=spamd argv=\/usr\/bin\/spamc -f -e  \/usr\/local\/bin\/swearfilter.py ${sender} ${recipient}<\/pre>\n<h3>Processing in Dovecot<\/h3>\n<p>Now that we have a header that indicates swearwords (&#8216;X-Too-Many-Swearwords&#8217;), we can take actions with it. The Dovecot <a href=\"https:\/\/wiki.dovecot.org\/Pigeonhole\/Sieve\/\">PigeonHole<\/a> extension interprets the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Sieve_(mail_filtering_language)\">Sieve language<\/a> which is used to do stuff based on email content. I use it to filter emails server-side into folders and stuff, and to reject emails flagged as spam into my spam folder. You can actually adjust the filters directly from Thunderbird with <a href=\"https:\/\/github.com\/thsmi\/sieve\">an extension<\/a>, so that&#8217;s cool (uses ManageSieve Dovecot extension too).<\/p>\n<p>Here&#8217;s the user-level Pigeonhole\/Sieve filter that deals with swearwords and does some other stuff. The spam filtering is in the global one, which I configured to run before this.<\/p>\n<pre class=\"\">require [\"fileinto\", \"envelope\", \"mailbox\",\"reject\"];\r\nif header :contains \"From\" [\"notifications@github.com\"] {\r\n\u00a0 fileinto \"INBOX.Lists\";\r\n}\r\nelsif header :contains \"From\" [\"paypal.com\",\"facebookmail.com\"] {\r\n\u00a0 fileinto \"INBOX.Lists\";\r\n}\r\nelsif header :contains \"X-Too-Many-Swearwords\" \"YES\" {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 reject text:\r\nWhoops! Your message was a bit too vulgar for the email server to process.\r\nPlease adjust and resend. Thanks!\r\n.\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 ;\r\n}<\/pre>\n<p>That last clause simply rejects the email so I never even see it and tells the author to send a new email. Nice! Totally works. Gmail marks the response as spam though&#8230; typical!<\/p>\n<p>Anyway, so that was fun, if not totally useless. I had enough trouble configuring that I hope this info helps someone do something actually useful! Good luck.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A year ago, my friend Laura was wishing that email providers could do some tone filtering and reject messages that are too mean. Since I run my own email server, I thought it might be simple to set something like that up easily. Turns out, it&#8217;s not that hard, but it wasn&#8217;t exactly trivial to &hellip; <a href=\"https:\/\/partofthething.com\/thoughts\/adding-multiple-content-filters-to-an-email-server-with-postfix-and-dovecot-pigeonholesieve\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Adding multiple content filters to an email server with postfix and dovecot Pigeonhole\/Sieve<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":4,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"","footnotes":""},"categories":[3],"tags":[],"class_list":["post-1318","post","type-post","status-publish","format-standard","hentry","category-computers"],"_links":{"self":[{"href":"https:\/\/partofthething.com\/thoughts\/wp-json\/wp\/v2\/posts\/1318","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/partofthething.com\/thoughts\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/partofthething.com\/thoughts\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/partofthething.com\/thoughts\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/partofthething.com\/thoughts\/wp-json\/wp\/v2\/comments?post=1318"}],"version-history":[{"count":2,"href":"https:\/\/partofthething.com\/thoughts\/wp-json\/wp\/v2\/posts\/1318\/revisions"}],"predecessor-version":[{"id":1387,"href":"https:\/\/partofthething.com\/thoughts\/wp-json\/wp\/v2\/posts\/1318\/revisions\/1387"}],"wp:attachment":[{"href":"https:\/\/partofthething.com\/thoughts\/wp-json\/wp\/v2\/media?parent=1318"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/partofthething.com\/thoughts\/wp-json\/wp\/v2\/categories?post=1318"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/partofthething.com\/thoughts\/wp-json\/wp\/v2\/tags?post=1318"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}