My Next Personal Project

I don't like distractions. They're distracting.

I also don't like missing a mail from my boss. They're rare, so when they come, they're often important.

So, I shut down alerts in Thunderbird and wrote a tool called jBiff.

jBiff connects to your mail server via IMAP and finds the new mail (thus biff) that meets the standards you want to watch for, then sends an alert via XMPP, aka Jabber (thus j). As is, I'm mostly searching on sender, and I'm not sure, but that might be all that works.

I suppose I should put this on Github. I'm pretty sure that step's overdue. But there's a problem.

jBiff uses IMAP. There are certain mail servers that don't offer IMAP. This obliges me to start working in POP.

And, there's another possiblity. There are many places I receive mail but never check. There are two primary places I check mail, and if you know me, you know what they are. But having a tool to check the status of the others and notify me weekly or so, that would be good.

My Life Just Became Easier

I get mail off of three classes of mail server. The first is an old-school, all-in-one server where I can log on, read mail with a built-in mail client like PINE. I can set a .forward file to send all my mail anywhere, or run it through a filter like procmail. With procmail, you can do almost anything. You can configure it so that when you send an email with the subject "reboot webserver", procmail will run a script that shuts down and restarts your webserver. It is an awesomely powerful and useful thing.

Then there's Gmail. I have nothing bad to say about Gmail. You cannot use it to run abstract programs — I've been playing with the idea of using Google Calendar to set up a remotely-configurable cron/at/batch equivalent, but it isn't there yet — but you have great stability, great interface and filtering. At this point. I'm going to state my position: If you can't filter mail, mail is useless. You'll get squashed by the deluge.

Which brings us to non-Gmail webmail. Specifically, my employer's webmail. You have to pay to get IMAP access to Yahoo Mail, which is a dealbreaker for me. My employer's webmail had fine POP and IMAP access, but what you couldn't do is filter on the server. You could configure clients to filter, but that means you have to copy all your filters to each client, whether it's the web interface, Thunderbird on this machine, Thunderbird on that machine, and so on.

My solution was to write my own filters to run on my workstation via crontab. It worked, and it allowed me to develop a means to specifically alert on things I find crucial rather than Thunderbird's "Alert if new mail" approach. But we've moved to a Zimbra-based solution which seems to have everything I'd want. 

imap_filter.pl -- Mail Filtering via IMAP with Perl

Thunderbird is a big thing. It is an even bigger thing when you have several mailboxes. So, while I've not yet given up on Thunderbird, I have taken to not have it running unless I know I need it. That is where jBiff, my IMAP-to-XMPP thingee, comes in. I've even taken to using the web interfaces instead of Thunderbird on occasion. All is well in the world.

But there's a problem.

Lists. Lists and filters. Filtering lists. ( That makes it one problem again. )

I'm used to using slocal or procmail on Unix machines to filter as the mail comes in. I'm not nearly as happy with the interface to filtering in Gmail, but I do love the result, which is a cleaner inbox.

My work mail has filtering that's enabled when you use the web interface. I don't use the web interface often, and when I have all my filters in Thunderbird, nothing is filtered when I use the web. What I needed is a means to write my filters outside of a mail client and have them run regularly, so that the state of the inbox is always as it should be.

So, I wrote imap_filter.pl. It needs to be cleaned up a bit, and it is not feature-complete, but I'm able to move mail based on many good things, so I'm at a done point.

#!/usr/bin/perl
use 5.010 ;
use strict ;
use warnings ;
use lib '/home/jacoby/bin' ;
use Carp ;
use Data::Dumper ;
use Getopt::Long ;
use IO::Socket::SSL ;
use IO::Interactive qw{interactive} ;
use Mail::IMAPClient ;
use IdentConf ':all' ;
use subs qw{ imap_part xmpp_part } ;

$Data::Dumper::Indent = 1 ;
my $debug ;
my $imap_identity ;

#methods
my @from ;
my @to ;
my @cc ;
my @subject ;
my @to_or_cc ; # don't use or yet.
my $age = 0 ;

#actions
my $move ; #move to dir
my $forward ; #forward to this address
my $delete ; #delete this file
my $read ;
my $unread ;

GetOptions(
            'imap=s'    => \$imap_identity,
            'from=s'    => \@from,
            'to=s'      => \@to,
            'cc=s'      => \@cc,
            'or=s'      => \@to_or_cc,
            'subject=s' => \@subject,
            'age=i'     => \$age,

            'move=s'    => \$move ,
            'delete'    => \$delete ,
            'read'      => \$read ,
            ) or exit( 1 ) ;

exit if !defined $imap_identity ;
exit if length $imap_identity < 1 ;

for my $a ( @to_or_cc ) {
    push @to, $a ;
    push @cc, $a ;
    }

my $filter ;
$filter->{ from }    = \@from ;
$filter->{ subject } = \@subject ;
$filter->{ to }      = \@to ;
$filter->{ cc }      = \@cc ;
$filter->{ age }     = $age ;

$filter->{ move }    = $move ;
$filter->{ read }    = $read ;
$filter->{ delete }  = $delete ;

imap_part $filter ;
exit ;

# ====================================================================
#
# connect to and search your mail server via IMAP
#
# ====================================================================
sub imap_part {
    my ( $filter ) = @_ ;
    say { interactive } Dumper $filter ;

    my %creds = get_credentials( 'imap', $imap_identity ) ;

    my $socket = IO::Socket::SSL->new( PeerAddr => $creds{ server },
                                       PeerPort => $creds{ port },
                                       ) or die "socket(): $@" ;

    my $client = Mail::IMAPClient->new( Socket   => $socket,
                                        User     => $creds{ username },
                                        Password => $creds{ password },
                                        ) or die "new(): $@" ;

    if ( $client->IsAuthenticated() ) {
        $client->select( $creds{ directory } )
          or die "Select '$creds{directory}' error: ",
          $client->LastError, "\n" ;

        my $i = 1 ;
        for my $msg ( reverse $client->messages ) {
            my $flag    = 0 ;
            my $from    = $client->get_header( $msg, 'From' ) ;
            my $sender  = $client->get_header( $msg, 'Sender' ) ;
            my $date    = $client->date( $msg ) ;
            my $to      = $client->get_header( $msg, 'To' ) ;
            my $cc      = $client->get_header( $msg, 'Cc' ) ;
            my $subject = $client->subject( $msg ) ;
            my @flags   = $client->flags( $msg ) ;
            my $seen = 0 ; $seen = 1 if grep m{/Seen}mx , @flags ;
            next if grep m{\Deleted}mx , @flags ;

            # Subject
            if ( scalar @{ $filter->{ subject } } > 0 ) {
                for my $f ( $filter->{ subject } ) {
                    my $ff = $$f[ 0 ] ;
                    $subject =~ m{($ff)}mix ;
                    say $1 if defined $1 ;
                    $flag++ unless defined $1 ;
                    }
                }
            # From
            if ( scalar @{ $filter->{ from } } > 0 ) {
                for my $f ( $filter->{ from } ) {
                    my $ff = $$f[ 0 ] ;
                    $from =~ m{($ff)}mix ;
                    $flag++ unless defined $1 ;
                    }
                }
            # To
            if ( scalar @{ $filter->{ to } } > 0 && ! defined $to ) {
                $flag++ ;
                }
            if ( scalar @{ $filter->{ to } } > 0 && defined $to ) {
                for my $f ( $filter->{ to } ) {
                    my $ff = $$f[ 0 ] ;
                    $to =~ m{($ff)}mix ;
                    $flag++ unless defined $1 ;
                    }
                }
            # Cc
            if ( scalar @{ $filter->{ cc } } > 0 && ! defined $cc ) {
                $flag++ ;
                }
            if ( scalar @{ $filter->{ cc } } > 0 && defined $cc ) {
                for my $f ( $filter->{ cc } ) {
                    my $ff = $$f[ 0 ] ;
                    $cc =~ m{($ff)}mix ;
                    $flag++ unless defined $1 ;
                    }
                }
            # Age
                #GOT NADA

            next if $flag ;

            if ( 1 ) {
                my $title = 'New mail from ' . $from ;
                my $body  = $subject ;
                $body = join q{"}, '', $body, '' ;

                #say { interactive } "$title - $body" ;
                say { interactive } $i++ . "\t" . '=' x 40 ;
                say { interactive } $subject;
                say { interactive }   'From:     ' . $from ;
                say { interactive }   '  Sender: ' . $sender if $sender  ;
                say { interactive }   '      To: ' . $to if $to ;
                say { interactive }   '      Cc: ' . $cc if $cc ;
                say { interactive }   '    Date: ' . $date ;
                say { interactive }   join ' ' , ' ' , @flags ;
                if ( $filter->{ move } ) {
                    my $move = $filter->{ move } ;
                    say { interactive } '    Move: ' . $move ;
                    my $newUid = $client->move( $move , $msg )
                        or die "Could not move: $@\n";
                    }
                say { interactive } '' ;
                }
            }
        $client->logout() ;
        }
    else {
        say { interactive } 'FAIL ' . $! ;
        }
    }

# --------------------------------------------------------------------

Reading IMAP directories with Perl

I use and like Thunderbird. Not the greatest thing since sliced bread, surely, but better for managing most of my mailboxes than pine and the school's web interface, which might soon be replaced with Gmail. Plus, IMAP!

The thing that sucks is alerting. If a mailing list gets updated, don't tell me. I shunt that stuff off into a subdirectory for a reason. In fact, there are three classes of people whose emails I want alerts about:

• Family — mostly my wife and parents, but also my children, cousins, etc.
• Bosses and Co-Workers — If you're involved in me getting paid, of course I need to know when you contact me
• Friends — this is a low third, which is understandable, I hope.

Everyone else, all the spammers and mailing lists, I'll read your mail when I get around to it.

So, I looked into Perl's Mail::IMAPClient.

#!/usr/bin/perl
use Modern::Perl ;
use IO::Socket::SSL ;
use Mail::IMAPClient ;
use Getopt::Long ;
use Carp ;
use Data::Dumper ;

$Data::Dumper::Indent = 1 ;

my $server = 'imap.gmail.com' ;
my $username = 'me' ;
my $password = 'nope' ;
my @sender ;

GetOptions(
    'sender=s' => \@sender ,
    ) ;

my $sender = join '|' , @sender  ;

my $socket = IO::Socket::SSL->new(
   PeerAddr => $server ,
   PeerPort => 993,
   )
  or die "socket(): $@";

my $client = Mail::IMAPClient->new(
   Socket   => $socket,
   User     => $username ,
   Password => $password ,
   Ignoresizeerrors => 1 ,
   )
  or die "new(): $@";

if ( $client->IsAuthenticated() ) {
    $client->select( 'INBOX' )
    or die "Select 'INBOX' error: ", $client->LastError, "\n";
    #for my $msg ( reverse $client->messages ) {
    for my $msg ( reverse $client->unseen ) {
        #my $string = $client->message_string($msg)
            #or die "Could not message_string: $@\n";
        my $from = $client->get_header( $msg , 'From' ) ;
        my $to   = $client->get_header( $msg , 'To' ) ;
        my $subject = $client->subject($msg)
            or die "Could not subject $@\n";
        if ( $from =~ m{$sender}i ) {
            my $notify = '/usr/bin/notify-send' ;
            my $title = 'New mail from ' . $from ;
            my $body = $subject ;
            my $icon = '-i ~/Pictures/Icons/icon_black_muffin.jpg' ;
            my $time = '-t ' . (1000 * 10 ) ;
            $body  =~ s/(['"])/\\$1/g;
            $title =~ s/(['"])/\\$1/g;
            $title = join q{"} , '' , $title , '' ;
            $body  = join q{"} , '' , $body , '' ;
            qx{ $notify $title $body $icon $time } ;
            }
        }
    $client->logout();
    }

It takes one or more sender substrings. mail_notify.pl -sender '@purdue.edu' , for example, would pop notifications from all mail from Purdue University addresses. And the good thing is, you don't need to change much more than the server name, username and password to make it work on my work server.

Thinking about mail ...

I use Thunderbird for mail. I use multiple mailboxes to segment my life. I have one that my bosses use to mail me. I have one for some friends. I have one I put on my resumes. I have one (strictly speaking, it's at least 6, potentially more, but one mailbox) that I've been at forever, and it's where most of my mailing lists go, where my parents mail me, and where I can bring the awesome power of Procmail to trigger abstract code through a mail filter. Plus one from my cable company that I don't really use.

Anyway, I use Thunderbird, because it's easier to watch all those mailboxes with one thing than it is to have seventeen windows open. I have Procmail for one mailbox, GMail's server-side sorting for two others, and contempt and Thunderbird's built-in filter for the campus mailbox. So, when I get mail from mailing lists, it goes to the list. When I get spam, for the most part, it gets trashed. Very fine-grained control over what goes where.

And very coarse control over alerts.

I like mailing lists. I'm on a lot, on a few subjects where I'm not the master and not going to be soon. So I collect great gobs of mail and use search functions or grep to find exactly what I'm looking for. But the alert function has just on and off. I don't care if there's a flame war on the opensolaris list, or discussion on new forms of statistical analysis on the Bioconductor list, or a flame war on the guitar list, and the alert function that tells me about that is stealing my focus. What I want is alerts for the people in the office, my parents, my wife, my children, a select set of friends, and complete silence for everyone else.

I'm going to have to write a module, aren't I?