Scraps
Magic spells
My fdm-maildir-neomutt setup

My fdm-maildir-neomutt setup

Warning

This article is from 2021. I went back to Thunderbird in early 2024 so, if something below stopped being current after that, I wouldn’t be able to tell.

So in my quest of complicating my life in the hopes of eventually simplifying it, I switched from Thunderbird to Neomutt. In this post I will try to re-walk the steps I took in order to have a functioning Neomutt setup, in case I need it for future reference but also in case this eases the life of someone trying to do the same.

Ingredients

Apart from the mentioned fdm and neomutt, these tools are involved in this setup:

  • Spamassassin for the spam.
  • Mairix for the searches.
  • Pass for the credentials (which needs gpg to work, but that’s far out of scope for this article).
  • Khard can be used as neomutt’s address book, but I won’t go into detail. I also sync khard’s contacts with my phone’s using vdirsyncer and davx5 on each side, with Nextcloud in the middle.
  • Python, for a couple scripts to help with the initial setup.
  • pdftotext for the PDF attachments.
  • w3m for the html emails.

My starting point

Somewhere between 2008 and 2018 I had switched from Apple’s Mail.app to Thunderbird. I had three accounts from which I downloaded the messages (I like to have them in my disk) and had managed to classify most of my mail into directories and subdirectories of what Thunderbird calls “Local folders”… Of course I still had 2000 or 3000 unsorted mails in the inboxes.

Around 2018 I kept hearing marvels about Maildir and, taking the opportunity that I was also switching to GNU/Linux, I configured Thunderbird to make use of it following, if I recall correctly, these instructions. When I finally jumped to Neomutt this year, Thunderbird’s storage was looking like this:

    • 2019 - Things.msf
    • Oldstuff.msf
      • Something 2018.msf
    • Worktuff.msf
      • Company1.msf
      • Company2.msf
    • Trash.msf
    • Same as above, with Inbox, Sent, Trash, etc
    • Same as above, with Inbox, Sent, Trash, etc

    • The above example is just an excerpt with some names changed, plus each CompanyN has subfolders to classify mail on a per project basis.

    The new folder structure

    Choosing it was almost the hardest part because the documentation I found online came off as very confusing, but maybe it was just me overthinking it. To keep things short, this is what mutt’s documentation has to say, and this is the structure I ended up with inside my ~/Mail folder:

    • The above structure has a Trash folder for all accounts. Then, an inbox for each of my three accounts (plus sent and drafts for two of them, my Sdf plan doesn’t include SMTP). Then comes the “Personal Folders” structure (which I now call “Archive”) flattened in a way that each dot represents one level deeper. I don’t know if I will encounter problems if I ever change my MUA again, but for the moment everything semms to be working alright. By the way, I learned that you can create those repetitive three subdirectories in a single command: mkdir somedir/{tmp,cur,new}, with -p if somedir doesn’t exist.

    Problem: Local Folders has too much stuff

    I created the inboxes manually because I wanted a fresh start, but as for the “Local Folders”, I had more than 170 folders and subfolders. How to import them into the new structure without too much work? I came up with a very crude Python script, available at https://gitlab.com/-/snippets/2090166.

    It’s dirty and it’s hacky, and took some dry runs to adjust, but did the work for me.

    Warning

    Please have an emergency plan if you follow anything in this page. This script wasn’t enough for reasons I’ll explain further down.

    fdm

    In short, fdm connects to the mailserver, fetches the messages, and does something with them according to a set of rules. In my case, something means:

    • having them analised by spamassassin
    • storing them in folders depending on
      • what spamassassin thought about it
      • what the sender and/or subject is
    • storing them in the inbox if none of the two above applied
    • deleting them from the server (I like to just store them locally).

    The configuration is relatively simple. I have a .fdm.conf dotfile for a couple of general settings and the accounts, and from there a separate file for the rules is included, because it can get long:

    set verify-certificates
set maximum-size 20M
set unmatched-mail keep

$path = "%h/Mail"

account "sdf" imaps server "mx.sdf.org"
    # I got pass installed, 
    # so this instructs fdm to ask me to 
    # unblock the password file, instead of having
    # it here in clear text:
    user "mysdfusername" pass $(pass sdf)

account "gmxch" imaps server "imap.gmx.net"
    user "mygmxchusername" pass $(pass gmxch)

account "gmxnet" imaps server "imap.gmx.net"
    user "mygmxnetusername" pass $(pass gmxnet)

include "~/.config/fdm/filtering_rules.conf"

    The filtering is made in the form of regex “matches” that trigger certain “actions”:

    # SPAM FILTERING
# This action sends the message to spamassassin, which
# will add a header the spammy ones:
action "spamcheck"   rewrite "/usr/bin/vendor_perl/spamassassin"
# And this action sends a message to our Spam local folder:
action "spamsort"    maildir "${path}/.Archive.Spam"

# FOLDER ACTIONS
# These just send the affected message to one of the inboxes:
action "sdf"         maildir "${path}/Sdf"
action "gmxch"       maildir "${path}/GMXch"
action "gmxnet"      maildir "${path}/GMXnet"
# And these sort into more specific folders, one of them
# with an extra step as example:
action "company1" {
                     add-header "Importance" value "high"
                     add-header "X-Priority" value "1"
                     maildir "${path}/.Archive.Company1"
                }
action "company2"    maildir "${path}/.Archive.Company2"
# More actions like the ones above for other folders
# ...
# ...

# SPAM CHECKING
# This matches all messages in my three accounts
# I specify them with an "and" because I actually have a 4th
# account I will speak about later :)
match all and accounts { "sdf" "gmxch" "gmxnet" } action "spamcheck" continue
# The "continue" means that the messages keep falling down 
# the rules until they are matched again.
# And the next rule matches messages that come out from 
# the previous action carrying the spamassassin mark on them
# as per https://github.com/rspamd/rspamd/issues/824
match "^X-Spam-Status:.*[Yy][Ee][Ss]" in headers  action "spamsort"

# FOLDER SORTING RULES
match "^From:.*company1\\.ch"         in headers  action "company1"

match "^From:.*person@company2\\.ch"  in headers
   or "^Subject:.*specific words"     in headers  action "company2"
# A whole lot of more rules
# ...
# ...

# Any non matched message by sender or subject will
# match one of these depending on the account they come from:
match account "sdf"                               action "sdf"
match account "gmxch"                             action "gmxch"
match account "gmxnet"                            action "gmxnet"

    That’s pretty much everything for me. fdm -n will check if the syntax is correct. fdm fetch will get the messages. Add -v (or more v’s until -vvvv) before fetch to see what’s going on. And -k for getting the emails without deleting them from the server in case of doubt (this can generate local duplicates though).

    Some online resources that helped me with fdm:

    I also found useful peeking at other people’s conf files:

    Neomutt

    The structure is ready and full of our archived messages, and fdm is delivering the new ones to the corresponding folders. Thanks to maildir, we could already read our email with relative ease with just a text editor. Theoretically. And extremely cumbersome, but one never knows.

    Neomutt does the work of navagiting through the folders and mails. Its general usage is out of the scope of this entry, but here’s how I configured it to have something minimally comfortable.

    The main configuration resides in my ~/.config/neomutt/neomuttrc:

    # SOME GENERAL SETTINGS

set mbox_type = Maildir
set date_format = "!%d.%m.%Y"

# You can change this to vim or whatever:
set editor = nano

set trash = ~/Mail/.Trash

# We'll reply with the address they wrote to us:
set reverse_name = yes

# We won't use the name that used the sender to write us:
set reverse_realname = no

# Caching improves speed:
set header_cache = ~/.cache/neomutt
set message_cachedir = ~/.cache/neomutt

# Not vital, but always goot to have a visually pleasant interface
source ~/.config/neomutt/colors-solarized-dark-256
# Got that one from https://github.com/altercation/mutt-colors-solarized

# SETTINGS FOR THE MESSAGES LIST

# Haven't wrapped my mind around how this setting works yet
# Here's how the conditional date works:
# https://neomutt.org/feature/cond-date
set index_format = "%4C %Z %H %<[y?%<[m?%<[d?%[%H:%M       ]&%[!%a %d     ]>&%[%d.%b     ]>&%[%d.%m.%Y ]> %-15.15L » %s"

# I like to see the messages ordered by thread:
set sort = threads

# Group threads by subject if there's a "RE: " in it:
set sort_re

# Newer messages up:
set sort_aux = reverse-last-date-received

# SHOW MY FOLDERS IN A SIDEBAR
set sidebar_visible
set sidebar_format = "%B%?F? [%F]?%* %?N?%N/?%S"
# I use a very wide sidebar because I use neomutt fullscreen:
set sidebar_width = 60
set sidebar_short_path
set sidebar_folder_indent
set sidebar_indent_string = " "
set sidebar_sort_method = "alpha"
set mail_check_stats
# Show only folders that got new messages:
set sidebar_new_mail_only

# Navigating the sidebar
bind index,pager <f9> sidebar-toggle-visible # F9 toggles it
bind index,pager \CP sidebar-prev # Ctrl-Shift-P: Previous Mailbox
bind index,pager \CN sidebar-next # Ctrl-Shift-N: Next Mailbox
bind index,pager \CO sidebar-open # Ctrl-Shift-O: Open Highlighted Mailbox
macro index,pager <f10> "<enter-command>toggle sidebar_new_mail_only<enter>" # F10: Show all folders/only with new mail

# CHANGING INBOXES
# Copied from https://www.df7cb.de/blog/2010/Using_multiple_IMAP_accounts_with_Mutt.html
macro index <f4> '<change-folder>~/Mail/Sdf<enter>'
macro index <f3> '<change-folder>~/Mail/GMXch<enter>'
macro index <f2> '<change-folder>~/Mail/GMXnet<enter>'

# HOOKS THAT READ THE CORRESPONDING ACCOUNT CONFIG WHEN WE CHANGE INBOX
folder-hook '~/Mail/Sdf' 'source ~/.config/neomutt/acct-sdf-b.conf'
folder-hook '~/Mail/GMXch' 'source ~/.config/neomutt/acct-gmx-ch.conf'
folder-hook '~/Mail/GMXnet' 'source ~/.config/neomutt/acct-gmx-net.conf'

# I like to store my replies in the same folder of the original:
folder-hook . 'set record="^"'

# We I open neomutt I want to go directly to this account:
source ~/.config/neomutt/acct-sdf-b.conf

    The ^ in line 75 is an abbreviation for “the current folder”. There’s more named folders and abbreviations in the Mutt’s Wiki. I’m using + further down.

    Update: function keys stopped working.

    For some unknown reason I stopped being able to change inboxes with the function keys, so I changed to <f2> to <ESC>1 etc.

    The three account configs look more or less like this:

    ## Folders
set folder = ~/Mail/GMXnet
set spoolfile = +
set postponed = +.Drafts
set record = +.Sent

## Receive options
# Neomutt can handle the receiving directly,
# but in this case this is done by fdm.

## Other
set my_user = my_usual_address@gmx.net
# Same as with fdm, pass prevents having the password written here:
set my_pass=`pass gmxnet`
# I like to know in which inbbox I am by changing the status bar colour:
color status color33 color18
# To-do: find  reference for the colour names.

## Send options
# smtp url should *not* be smtps according to
# https://unix.stackexchange.com/a/97679
set smtp_url="smtp://$my_user@mail.gmx.net:587"
set smtp_pass = "$my_pass"
set ssl_force_tls = yes
set ssl_starttls=yes
set realname='guillem'
set hostname="gmx.net"
alternates ^my_usual_Address@gmx.net$ ^my_alias@gmx.net$
# We can specify a text file to include automatically after all our messages:
set signature = "~/.config/neomutt/acct-gmx-net.signature"

## Mailboxes
# First, unset all the mailboxes neomutt might be showing:
unmailboxes *
# Add this account's spoolfile to the mailboxes list:
mailboxes $spoolfile
# Add all the directories starting by a dot to the mailboxes list,
# so all our "Local Folders" will show up, plus .Trash.
mailboxes `find ~/Mail -maxdepth 1 -name ".[^.]*" -type d | awk '{printf "%s ", $0}'`

## A hook triggers the re setting of user and pass whenever we enter here
account-hook $folder "set imap_user=my_usual_address@gmx.net imap_pass=$my_pass"

    Neomutt should be able to show the inbox, but once you navigate to the imported folders…

    Problem: I can’t see the messages in the Archive

    I could see the folders I imported with the script above, they show the number of messages inside, but when I open them they look empty.

    After much shooting in the dark, I came up with a hacky solution: if any folder got a new unread message, all the rest of the messages turned visible for neomutt; so I copied the file corresponding to a new email of the inbox into the /new subdirectory of all the mailboxes. Of course with a script, otherwise it would have been tedious: https://gitlab.com/-/snippets/2090316.

    And of course afterwards I had to delete this message, but that’s as easy as pressing d to mark it for deletion and $ to purge.

    Dealing with html emails and attachments

    Neomutt uses a file called ~/.mailcap to decide what to do with different MIME types. Here is mine, made with the help of this guide:

    # PDF
# Okular is my GUI viewer, you can use another:
application/pdf;okular %s; test=test -n "$DISPLAY"
# If not on X, neomutt resorts to pdftotext:
application/pdf;pdftotext %s; copiousoutput

#HTML
# I use Firefox, but that can be changed:
text/html; firefox %s; edit=nano %s; test=test -n "$DISPLAY"
# If not on X, neomutt resorts to w3m
text/html ; w3m -dump -T text/html %s; nametemplate=%s.html ; copiousoutput

# OFFICE
application/vnd.openxmlformats-officedocument.wordprocessingml.document; libreoffice %s; test=test -n "$DISPLAY"
application/msword; libreoffice %s; test=test -n "$DISPLAY"
application/vnd.ms-excel; libreoffice %s; test=test -n "$DISPLAY"
application/vnd.ms-powerpoint; libreoffice %s; test=test -n "$DISPLAY"

    We just need to press v to view a list of attachments from which we can select what we want to open. We also tell neomutt to try to show the html emails directly, without having to open them as attachment. That’s just adding auto_view text/html to our neomuttrc.

    Printing emails

    It is possible to print a message directly, but I prefer to see it as a PDF first, and then print from my PDF viewer:

    set print_command = 'set -e; f=`mktemp`; muttprint -P A4 -p TO_FILE:"$f"; sleep 1; okular "$f"; rm "$f"'

    When I press p at a message, this gets a temporary file path, makes a DIN-A4 PDF in it, waits 1 second for it to be complete before opening it with Okular, then removes it.

    Training Spamassasin

    I copied the following macros from this thread to be able to mark a message as (S)pam or (H)am:

    macro index S '<enter-command>unset wait_key<enter><shell-escape>date >> ~/Mail/.sa-learn.log<enter><pipe-message>sa-learn --spam --no-rebuild -D >> ~/Mail/.sa-learn.log 2>&1 &<enter><copy-message>~/Mail/.Archive.Spam/ <enter><enter><delete-message><enter-command>set wait_key=yes<enter>'
macro index H '<enter-command>unset wait_key<enter><shell-escape>date >> ~/Mail/.sa-learn.log<enter><pipe-message>sa-learn --ham --no-rebuild -D >> ~/Mail/.sa-learn.log 2>&1 &<enter><enter-command>set wait_key=yes<enter>'
macro index E '<enter-command>unset wait_key<enter><shell-escape>date >> ~/Mail/.sa-learn.log<enter><shell-escape>sa-learn --rebuild -D >> ~/Mail/.sa-learn.log 2>&1 &<enter><enter-command>set wait_key=yes<enter>'

    Searching

    Mairix can be configured with ~/.mairixrc:

    base=~/Mail
database=~/.mairixdb
mfolder=.Search
maildir=Sdf
maildir=GMXch
maildir=.Archive*

    And used from neomutt with a couple of macros:

    macro generic ,f "<shell-escape>mairix " "search via mairix"
macro generic ,,f "<change-folder>=.Search" "load the search results mailbox"

    The .Search mailbox contains the results in the form of symbolic links to the actual messages, so we can e.g. read them but not delete them from there.

    Integration with khard

    Using and configuring khard is too much for this entry, but if it happens to be installed already, the following is enough for neomutt:

    set query_command = "khard email --parsable %s"
bind editor <Tab> complete-query
bind editor ^T complete

    Neomutt can be configured to save a mail contact into khard, but it’s not something I do.

    One last useful macro

    Sometimes a newsletters folder gets some messages one doesn’t read further than the subject line. To mark all the folder as read with just hitting A:

    macro index A \
    "<tag-pattern>~N|~O<enter><tag-prefix><clear-flag>N<untag-pattern>.<enter>" \
    "mark all new as read"

    Done

    This is more or less how my setup works at the time of writing this. I said before that I have a fourth account. The reason for that is that I didn’t want to just copy all the messages in my Thunderbird inboxes over to Neomutt, so I’m slowly sorting them with fdm. But this is material for a future entry.

    Snippet: scan folders for Qtile's maildir widget