Cookie Notice

As far as I know, and as far as I remember, nothing in this page does anything with Cookies.

2016/04/12

Did I Mention I Hate Default Mail Notifications?

We live in a world of spam, of free email accounts and large mailing lists. You do not want to enable promiscuous notifications in such a world. That way lies madness.

But never knowing that the important people in your life — those you love, those who pay you, those who fix your guitars — are trying to contact you because you've turned off notifications is madness also. Perhaps a worse madness.

But Perl exists. CPAN exists. There is a way out.

I wrote a program for more general-purpose mail-handling, mostly clearing spam out of my work accounts, but decided to rewrite in order to handle the act of warning that I had new mail.

#!/home/jacoby/perl5/perlbrew/perls/perl-5.20.2/bin/perl

# specialized version of imap_task that handles just warnings. 
# problem with previous attempts is that it kept warning about
# new mail that matched until it was marked it read or deleted

# the goal is to do things once, with a data store independent 
# from IMAP that indicates if the warning has been sent. 

# YAML? JSON? Mongo? We'll try YAML.

use feature qw'say state' ;
use strict ;
use warnings ;
use utf8 ;

use Carp ;
use DateTime ;
use DateTime::Duration ;
use DateTime::Format::DateParse ;
use Getopt::Long ;
use IO::Interactive qw{interactive} ;
use IO::Socket::SSL ;
use Mail::IMAPClient ;
use YAML::XS qw{ LoadFile DumpFile } ;

use lib '/home/jacoby/lib' ;
use Locked ;
use Notify qw{ notify } ;
use Pushover ;
use Say qw{ say_message } ;

I have gone to perlbrew for most of my usable Perl, and I would normally use #!/usr/bin/env perl as my hashbang, but it is hard to tell crontab to use a perl other than system perl, so rather than trying, I specify the perl I want. Your usage will vary.

The next four non-code lines are my standard. That's mostly use Modern::Perl, I think, but I like being able to specify.

After that, there's a bunch of modules from CPAN. IO::Socket::SSL and Mail::IMAPClient are crucial for interacting with the mail server, YAML::XS is the better YAML module, according to Gabor Szabo. I like programs that give me verbose output when I run them, but don't clog my cron inbox when run via crontab, so I really overuse IO::Interactive. I am not sure that I need all the DateTime stuff I load for this purpose, but better safe than sorry.

Then there's the stuff that I wrote for purposes such as this. I have many programs that I want to behave differently if the computer is locked, which means I'm not at my standing desk, so I wrote Locked. I wanted to use notify-send on my Ubuntu machines to pop up notifications, so I wrote Notify. Net::Pushover wasn't written when I started this, so I wrote Pushover to interact with Pushover and should've put it on CPAN myself. Alas. And Say doesn't have to do with say(), but rather is a wrapper around eSpeak, a speech synthesizer.

my @sender ;
my $debug = 0 ;
my $task ;
$task = 'work_alert' ;

GetOptions(
    'debug=i' => \$debug,
    # 'task=s'  => \$task,
    )
    or exit(1) ;
# get the configuration
my $config_file = $ENV{HOME} . '/.imap/' . $task . '.yml' ;
croak 'No task set'  if length $task < 1 ;
croak 'No task file' if !-f $config_file ;

my $settings = LoadFile($config_file) ;
$settings->{debug} = $debug ;

# set a message if one hasn't been set
$settings->{message} = $settings->{message} ? $settings->{message} : 'You have mail' ;

my $has_spoken = 0 ;

say {interactive} '='x20;
my $warn_file   = $ENV{HOME} . '/.imap_warn.yml' ;
my $warnings = LoadFile($warn_file) ;
check_imap($settings) ;
DumpFile( $warn_file , $warnings ) ;
say {interactive} '-'x20;
exit ;

Here I establish a bunch of globals and everything up for check_imap(), the main part of this program.

There are two YAML files that this program uses. One is .imap_warn.yml, which is a hash where the key is "$FROM||$SUBJECT||$DATE" and the value is 1, so I can tell if I've been told about a certain email before, and .imap/work_alert.yml, which is the main configuration file, and looks like this:

---
server: mailserver.example.com
port: 993
username: username
password: you_dont_get_my_password
message: 'You have mail'
folders:
    INBOX: 
        alert:
            subject:
                - 'big data'
            from:
                # Family
                - jacoby
                # The Lab
                - boss@example.com

I have used a separate file to hold the specifications for my SMTP and IMAP servers, but here, having all the config in one place seemed right. Since it contains password information, it is especially important that permissions are set correctly, specifically only you can read it. I do not test permissions in this program.

As mentioned, this is adapted from a more general mail-handling program, which takes specific configuration files for the kind of work it does. This just has the one, so that has been commented out, leaving just the debug flag.

I have had issues with YAML empty-writing files, which is why I separated .imap_warn from work_alert.pl.
sub check_imap {
    my $settings = shift ;
    my $client ;
    if ( $settings->{port} == 993 ) {

        my $socket = IO::Socket::SSL->new(
            PeerAddr => $settings->{server},
            PeerPort => $settings->{port},
            )
            or die "socket(): $@" ;

        $client = Mail::IMAPClient->new(
            Socket   => $socket,
            User     => $settings->{username},
            Password => $settings->{password},
            )
            or die "new(): $@" ;
        }
    elsif ( $settings->{port} == 587 ) {
        $client = Mail::IMAPClient->new(
            Server   => $settings->{server},
            User     => $settings->{username},
            Password => $settings->{password},
            )
            or die "new(): $@" ;
        }

    my $dispatch ;
    $dispatch->{'alert'}          = \&alert_and_store_mail ;
    $dispatch->{'warn'}           = \&warn_mail ;

    if ( $client->IsAuthenticated() ) {
        say {interactive} 'STARTING' ;

        for my $folder ( keys %{ $settings->{folders} } ) {
            say {interactive} join ' ', ( '+' x 5 ), $folder ;
            $client->select($folder)
                or die "Select '$folder' error: ",
                $client->LastError, "\n" ;

            my $actions = $settings->{folders}->{$folder} ;

            for my $msg ( reverse $client->unseen ) {
                my $from = $client->get_header( $msg, 'From' ) || '' ;
                my $to   = $client->get_header( $msg, 'To' )   || '' ;
                my $cc   = $client->get_header( $msg, 'Cc' )   || '' ;
                my $subject = $client->subject($msg) || '' ;

                say {interactive} 'F: ' . $from ;
                say {interactive} 'S: ' . $subject ;

                # say { interactive } 'T: ' . $to ;
                # say { interactive } 'C: ' . $cc ;

                for my $action ( keys %$actions ) {

                    # say { interactive } '     for action: ' . $action ;

                    for my $key ( @{ $actions->{$action}->{from} } ) {
                        if (   defined $key
                            && $from =~ m{$key}i
                            && $dispatch->{$action} ) {
                            $dispatch->{$action}->( $client, $msg ) ;
                            }
                        }
                    for my $key ( @{ $actions->{$action}->{to} } ) {
                        if ( $to =~ m{$key}i && $dispatch->{$action} ) {
                            $dispatch->{$action}->( $client, $msg ) ;
                            }
                        }
                    for my $key ( @{ $actions->{$action}->{cc} } ) {
                        if ( $cc =~ m{$key}i && $dispatch->{$action} ) {
                            $dispatch->{$action}->( $client, $msg ) ;
                            }
                        }
                    for my $key ( @{ $actions->{$action}->{subject} } ) {
                        my $match = $subject =~ m{$key}i ;
                        if ( $subject =~ m{$key}i && $dispatch->{$action} ) {
                            $dispatch->{$action}->( $client, $msg ) ;
                            }
                        }
                    }
                say {interactive} '' ;
                }

            say {interactive} join ' ', ( '-' x 5 ), $folder ;
            }

   # $client->close() is needed to make deletes delete, put putting before the
   # logout stops the process.
        $client->close ;
        $client->logout() ;
        say {interactive} 'Finishing' ;
        }
    say {interactive} 'Bye' ;
    }

There are four things we can match on: from, to, cc and subject. I generally match on subject and from, but the code is there.

I have started but not finished Higher Order Perl by Mark Jason Dominus, but one of the things I got from that book (I think; if not, then from co-workers) is the concept of a dispatch table, where behavior of the program changes based on the data. I could simplify this a lot more, I'm sure, with more higher-order programming, but I'm reasonably happy with it right now.

# ====================================================================
# send to STDOUT without IO::Interactive, for testing
sub warn_mail {
    my ( $client, $msg ) = @_ ;
    say {interactive} 'warn' ;
    my $from = $client->get_header( $msg, 'From' ) || return ;
    my $to   = $client->get_header( $msg, 'To' )   || return ;
    my $subject = $client->subject($msg) || return ;
    my $date  = $client->get_header( $msg, 'Date' ) || return ;
    my $dt    = DateTime::Format::DateParse->parse_datetime($date) ;
    my $today = DateTime->now() ;
    $dt->set_time_zone('UTC') ;
    $today->set_time_zone('UTC') ;
    my $delta = $today->delta_days($dt)->in_units('days') ;
    say $from ;
    say $to ;
    say $subject ;
    say $dt->ymd ;
    say $delta ;
    }

# ====================================================================
# alert about new mail
sub alert_and_store_mail {
    my ( $client, $msg ) = @_ ;
    say {interactive} 'alert and store' ;
    my $date = $client->get_header( $msg, 'Date' ) || 'NONE' ;
    my $from = $client->get_header( $msg, 'From' ) || 'NONE' ;
    my $to   = $client->get_header( $msg, 'To' )   || 'NONE' ;
    my $subject = $client->subject($msg) || 'NONE' ;
    my $key = join '||' , $from , $subject , $date ;
    $key =~ s{\s+}{ }g ;
    my $title =  'Mail From: ' . $from ;
    chomp $title ;
    chomp $subject ;

    return if $warnings->{$key} ;
    $warnings->{$key} = 1 ;

    $from =~ s{\"}{}gx ;
    if ( is_locked() ) {
        pushover(
            {   title   => $title ,
                message => $subject
                }
            ) ;
        }
    else {
        say {interactive} $title  ;
        say {interactive} $subject ;
        say {interactive} defined $warnings->{$key} ? 1 : 0 ;
        say {interactive} 'has spoken: ' . $has_spoken ;
        if ( ! $has_spoken ) {
            say_message( { message => $settings->{message} , title => '' } ) ;
            }
        notify(
            {   title   => $title ,
                message => $subject ,
                icon    => '/home/jacoby/Dropbox/Photos/Icons/mail.png' ,
                }
            ) ;
        }
    $has_spoken = 1 ;
    return ;
    }

warn() is useful for debugging, but the work of the program is done in alert_and_store_mail(). $client is the IMAP connection, and $msg is the message itself. I find that I have to send both. I might be doing it wrong, though.

And here is where my modules come in. is_locked() returns a boolean, depending on which way you lock your screens. say_message(), pushover() and notify() share a format, a hashref containing title and message. say_message() tells me that I have notifications coming, and they show up on my desktop. And if I'm away from my desk, they show up on my tablet because of Pushover.

I'll put this into a repo on GitHub, including all the modules. I would like to get this into shape to be something like App::imap_warn or the like, but I'm not there yet. I'm sure there's interest, because default notifications suck.

2016/04/08

Purdue Perl Mongers - April 13 - "Starship Mongers"

I wrote a quick five-minute counter in Javascript just for this
I don't think I've mentioned it here, but I'm one of the core members of Purdue Perl Mongers, which I've wrangled into a SIG of Greater Lafayette Open Source Symposium (#GLOSSY) to try to reach out to others in the Open Source community.

I was going to talk about DBIx::Class and how it connects to Dancer, but I haven't learned nearly enough about DBIx::Class to talk about it, and didn't have enough open days to come up with a decent presentation, so I punted.

Thus Starship Mongers!

It's a variation on "Lightning Talks", which give speakers a strict five-minute window, but because we're a small group, I decided to add a wrinkle: "Everybody Talks! No one quits!"

(The next part of the quote seems a little too tough for a user group.)

This means that I intend that everyone should talk for five minutes on something. Doesn't have to be Perl. Doesn't have to be programming, or computing, or open source. Just has to be something you are interested in or have questions about. (But, remember your audience.)

I hope that this will charge up the group, bring up ideas for upcoming meetings. If nothing else, it'll give me time to get up to speed on DBIC.

2016/04/04

Diagnosing A Problem: OddMuse

I work in a lab in a large research-centered university. We use a wiki to serve as our lab notebook where we keep notes about the samples that go through. We're also a Perl shop, so we went with a Perl-based wiki named OddMuse (a fork of UseMod). This has been our platform of choice for nearly a decade.

Today, it was reported that a few pages would hang during loading. They gave up after 300, as we have a 5-minute timeout in our Apache config. I shame myself by saying that I went to the OddMuse IRC channel before I looked at /var/log/html/error_log, but that is what happened. The error log reported:  [Mon Apr 04 13:11:54 2016] [error] [client 142.68.31.21] (70007)The timeout specified has expired: ap_content_length_filter: apr_bucket_read() failed. I'm not strong in my Apache Fu, but I'm pretty sure this means that we hit the timeout, but it doesn't really say why we hit the timeout.

Which brings up a weirdness. Imagine example.com/wiki/SandBox is the page in question. You can get the page in it's full glory, before it's turned into HTML and spat out, at example.com/wiki/raw/SandBox, and that page always loads fast.

I "solved" the issue by editing and saving the file. I still don't know what's going on so that OddMuse can handle the data in raw form but cannot convert it to HTML. I am currently going down two roads of thought. The first is that there was a filesystem issue. We're working on a filesystem that is amazing in it's redundancy, size and the sheer number of connected nodes, but on occasion, we hit points where it falls down, giving us a several minute wait for commands such as ls or clear to run.

The other thought relates to how we actually use OddMuse. We wanted to have a front-end that behaved nearly like Word, so we use CKEditor and save HTML instead of wiki markup. At first, we saved samples in groups of up to 10, writing them to an unordered HTML list, but now we're pushing 400. The stub of each wiki page is created programmatically, and I am thinking that the higher numbers might be more than it can take.

This, I guess, gives me a thing I can test. Write a thing that starts with, say, 100 elements in a list, then builds it up until the page doesn't render. I can do that. And I will do that tomorrow, because it's after 5pm today.

Taking the Great Leap Forward with Dancer2

I am working on understanding a raft of technologies, including Dancer and Bootstrap, in order to make our web presence look more current and, more importantly, be more maintainable. 

I'm learning a lot, which is not the positive statement that it sounds like. Rip Van Winkle certainly learned a lot after he slept for twenty years and woke up in post-Revolution America. 

For most of my time as a web developer, when I needed to do authentication, I did it with Apache's built-in server authentication. The number of users I needed to handle was always small enough, and except for a few things where it was entirely for me, I was not the person in charge of creating and maintaining the password system.

I know and believe in a few points. I know that I as admin of a system should not have access to the plain-text passwords of the users. I know that it is common to have two password fields when creating/changing passwords, to ensure you have the right spelling. I'm not 100% bought into that one, but I understand it. I know you keep an email address for "Forgot Password" systems can use your email system as a factor to ensure you're authorized to change that password. And I know you should use encryption systems created by experts, rather than roll your own and create a system that's full of holes. 

I've been using Dancer2::Plugin::Auth::Extensible, trying to get the parts I'd want for a generic system before working on things that I'd want for the lab, and there's much I'm comfortable with. I can get people logged in. I can set roles and limit access to users with specific roles. I can store the date of the last login, which might be useful. And it's all backed by MySQL, which means that, even without an admin dashboard, I have the skills to change anything about a user profile that needs changing.

But we don't want that. We want the techs and the users to have the ability to set values for the user, if for no other reason than I want to be able to move on to other things. So I need to figure out, as a standard, how these things go together, so I can try to implement it. I have things that I'm getting together. I do have questions, though.
  • Clearly, lots consider the repeat-your-password thing as an important part of the password workflow, and clearly, this is a check that I need to at least be able to do. I'm seeing a huge task-duplication thing, because you want to be able to say "passwords don't match" on the client side before the user presses go, but you always want to check things on the server side before you click "submit", because the user might block Javascript. Is this something that Bootstrap can help with? Or will I have to write something like that? I'm willing and able, but with the layout stuff and the way of the future encouraging us to have CSS and JS that's combined and minified and gzipped and included on every page, I'd like to have that taken care of automatically by the framework than go custom.
  • It's not immediately clear in the docs how to enable password encryption. I do need to read that more. (Solved. It was in the docs. I need to read the docs.)
  • I'm hitting the concept of roles and finding that they'd make certain things very useful. I'd like to be able to handle things like unix groups instead, but as is, they allow certain things that will make the end result a lot easier.

    I found, however, that, while the tools to check and control access due to roles are solid, setting and removing roles is less so. I asked the Dancer2 IRC about it, and was told to make a Github issue. I did, and then I wrote something that, within context of my tooling, adds add_user_role and remove_user_role functions. So I have that covered and can move forward.
There's more than this. I could see us wanting a website that has static, CGI and Dancer2 paths, although I think that, when I wake up with a start at 3am, bathed in sweat with a racing heartbeat, this thought is what I was dreaming about. But I'll wait for a while before I have to worry about that.

And, with a parting shot from @perigrin.