Cookie Notice

As far as I know, and as far as I remember, nothing in this page does anything with Cookies.

2016/04/12

Did I Mention I Hate Default Mail Notifications?

We live in a world of spam, of free email accounts and large mailing lists. You do not want to enable promiscuous notifications in such a world. That way lies madness.

But never knowing that the important people in your life — those you love, those who pay you, those who fix your guitars — are trying to contact you because you've turned off notifications is madness also. Perhaps a worse madness.

But Perl exists. CPAN exists. There is a way out.

I wrote a program for more general-purpose mail-handling, mostly clearing spam out of my work accounts, but decided to rewrite in order to handle the act of warning that I had new mail.

#!/home/jacoby/perl5/perlbrew/perls/perl-5.20.2/bin/perl

# specialized version of imap_task that handles just warnings. 
# problem with previous attempts is that it kept warning about
# new mail that matched until it was marked it read or deleted

# the goal is to do things once, with a data store independent 
# from IMAP that indicates if the warning has been sent. 

# YAML? JSON? Mongo? We'll try YAML.

use feature qw'say state' ;
use strict ;
use warnings ;
use utf8 ;

use Carp ;
use DateTime ;
use DateTime::Duration ;
use DateTime::Format::DateParse ;
use Getopt::Long ;
use IO::Interactive qw{interactive} ;
use IO::Socket::SSL ;
use Mail::IMAPClient ;
use YAML::XS qw{ LoadFile DumpFile } ;

use lib '/home/jacoby/lib' ;
use Locked ;
use Notify qw{ notify } ;
use Pushover ;
use Say qw{ say_message } ;

I have gone to perlbrew for most of my usable Perl, and I would normally use #!/usr/bin/env perl as my hashbang, but it is hard to tell crontab to use a perl other than system perl, so rather than trying, I specify the perl I want. Your usage will vary.

The next four non-code lines are my standard. That's mostly use Modern::Perl, I think, but I like being able to specify.

After that, there's a bunch of modules from CPAN. IO::Socket::SSL and Mail::IMAPClient are crucial for interacting with the mail server, YAML::XS is the better YAML module, according to Gabor Szabo. I like programs that give me verbose output when I run them, but don't clog my cron inbox when run via crontab, so I really overuse IO::Interactive. I am not sure that I need all the DateTime stuff I load for this purpose, but better safe than sorry.

Then there's the stuff that I wrote for purposes such as this. I have many programs that I want to behave differently if the computer is locked, which means I'm not at my standing desk, so I wrote Locked. I wanted to use notify-send on my Ubuntu machines to pop up notifications, so I wrote Notify. Net::Pushover wasn't written when I started this, so I wrote Pushover to interact with Pushover and should've put it on CPAN myself. Alas. And Say doesn't have to do with say(), but rather is a wrapper around eSpeak, a speech synthesizer.

my @sender ;
my $debug = 0 ;
my $task ;
$task = 'work_alert' ;

GetOptions(
    'debug=i' => \$debug,
    # 'task=s'  => \$task,
    )
    or exit(1) ;
# get the configuration
my $config_file = $ENV{HOME} . '/.imap/' . $task . '.yml' ;
croak 'No task set'  if length $task < 1 ;
croak 'No task file' if !-f $config_file ;

my $settings = LoadFile($config_file) ;
$settings->{debug} = $debug ;

# set a message if one hasn't been set
$settings->{message} = $settings->{message} ? $settings->{message} : 'You have mail' ;

my $has_spoken = 0 ;

say {interactive} '='x20;
my $warn_file   = $ENV{HOME} . '/.imap_warn.yml' ;
my $warnings = LoadFile($warn_file) ;
check_imap($settings) ;
DumpFile( $warn_file , $warnings ) ;
say {interactive} '-'x20;
exit ;

Here I establish a bunch of globals and everything up for check_imap(), the main part of this program.

There are two YAML files that this program uses. One is .imap_warn.yml, which is a hash where the key is "$FROM||$SUBJECT||$DATE" and the value is 1, so I can tell if I've been told about a certain email before, and .imap/work_alert.yml, which is the main configuration file, and looks like this:

---
server: mailserver.example.com
port: 993
username: username
password: you_dont_get_my_password
message: 'You have mail'
folders:
    INBOX: 
        alert:
            subject:
                - 'big data'
            from:
                # Family
                - jacoby
                # The Lab
                - boss@example.com

I have used a separate file to hold the specifications for my SMTP and IMAP servers, but here, having all the config in one place seemed right. Since it contains password information, it is especially important that permissions are set correctly, specifically only you can read it. I do not test permissions in this program.

As mentioned, this is adapted from a more general mail-handling program, which takes specific configuration files for the kind of work it does. This just has the one, so that has been commented out, leaving just the debug flag.

I have had issues with YAML empty-writing files, which is why I separated .imap_warn from work_alert.pl.
sub check_imap {
    my $settings = shift ;
    my $client ;
    if ( $settings->{port} == 993 ) {

        my $socket = IO::Socket::SSL->new(
            PeerAddr => $settings->{server},
            PeerPort => $settings->{port},
            )
            or die "socket(): $@" ;

        $client = Mail::IMAPClient->new(
            Socket   => $socket,
            User     => $settings->{username},
            Password => $settings->{password},
            )
            or die "new(): $@" ;
        }
    elsif ( $settings->{port} == 587 ) {
        $client = Mail::IMAPClient->new(
            Server   => $settings->{server},
            User     => $settings->{username},
            Password => $settings->{password},
            )
            or die "new(): $@" ;
        }

    my $dispatch ;
    $dispatch->{'alert'}          = \&alert_and_store_mail ;
    $dispatch->{'warn'}           = \&warn_mail ;

    if ( $client->IsAuthenticated() ) {
        say {interactive} 'STARTING' ;

        for my $folder ( keys %{ $settings->{folders} } ) {
            say {interactive} join ' ', ( '+' x 5 ), $folder ;
            $client->select($folder)
                or die "Select '$folder' error: ",
                $client->LastError, "\n" ;

            my $actions = $settings->{folders}->{$folder} ;

            for my $msg ( reverse $client->unseen ) {
                my $from = $client->get_header( $msg, 'From' ) || '' ;
                my $to   = $client->get_header( $msg, 'To' )   || '' ;
                my $cc   = $client->get_header( $msg, 'Cc' )   || '' ;
                my $subject = $client->subject($msg) || '' ;

                say {interactive} 'F: ' . $from ;
                say {interactive} 'S: ' . $subject ;

                # say { interactive } 'T: ' . $to ;
                # say { interactive } 'C: ' . $cc ;

                for my $action ( keys %$actions ) {

                    # say { interactive } '     for action: ' . $action ;

                    for my $key ( @{ $actions->{$action}->{from} } ) {
                        if (   defined $key
                            && $from =~ m{$key}i
                            && $dispatch->{$action} ) {
                            $dispatch->{$action}->( $client, $msg ) ;
                            }
                        }
                    for my $key ( @{ $actions->{$action}->{to} } ) {
                        if ( $to =~ m{$key}i && $dispatch->{$action} ) {
                            $dispatch->{$action}->( $client, $msg ) ;
                            }
                        }
                    for my $key ( @{ $actions->{$action}->{cc} } ) {
                        if ( $cc =~ m{$key}i && $dispatch->{$action} ) {
                            $dispatch->{$action}->( $client, $msg ) ;
                            }
                        }
                    for my $key ( @{ $actions->{$action}->{subject} } ) {
                        my $match = $subject =~ m{$key}i ;
                        if ( $subject =~ m{$key}i && $dispatch->{$action} ) {
                            $dispatch->{$action}->( $client, $msg ) ;
                            }
                        }
                    }
                say {interactive} '' ;
                }

            say {interactive} join ' ', ( '-' x 5 ), $folder ;
            }

   # $client->close() is needed to make deletes delete, put putting before the
   # logout stops the process.
        $client->close ;
        $client->logout() ;
        say {interactive} 'Finishing' ;
        }
    say {interactive} 'Bye' ;
    }

There are four things we can match on: from, to, cc and subject. I generally match on subject and from, but the code is there.

I have started but not finished Higher Order Perl by Mark Jason Dominus, but one of the things I got from that book (I think; if not, then from co-workers) is the concept of a dispatch table, where behavior of the program changes based on the data. I could simplify this a lot more, I'm sure, with more higher-order programming, but I'm reasonably happy with it right now.

# ====================================================================
# send to STDOUT without IO::Interactive, for testing
sub warn_mail {
    my ( $client, $msg ) = @_ ;
    say {interactive} 'warn' ;
    my $from = $client->get_header( $msg, 'From' ) || return ;
    my $to   = $client->get_header( $msg, 'To' )   || return ;
    my $subject = $client->subject($msg) || return ;
    my $date  = $client->get_header( $msg, 'Date' ) || return ;
    my $dt    = DateTime::Format::DateParse->parse_datetime($date) ;
    my $today = DateTime->now() ;
    $dt->set_time_zone('UTC') ;
    $today->set_time_zone('UTC') ;
    my $delta = $today->delta_days($dt)->in_units('days') ;
    say $from ;
    say $to ;
    say $subject ;
    say $dt->ymd ;
    say $delta ;
    }

# ====================================================================
# alert about new mail
sub alert_and_store_mail {
    my ( $client, $msg ) = @_ ;
    say {interactive} 'alert and store' ;
    my $date = $client->get_header( $msg, 'Date' ) || 'NONE' ;
    my $from = $client->get_header( $msg, 'From' ) || 'NONE' ;
    my $to   = $client->get_header( $msg, 'To' )   || 'NONE' ;
    my $subject = $client->subject($msg) || 'NONE' ;
    my $key = join '||' , $from , $subject , $date ;
    $key =~ s{\s+}{ }g ;
    my $title =  'Mail From: ' . $from ;
    chomp $title ;
    chomp $subject ;

    return if $warnings->{$key} ;
    $warnings->{$key} = 1 ;

    $from =~ s{\"}{}gx ;
    if ( is_locked() ) {
        pushover(
            {   title   => $title ,
                message => $subject
                }
            ) ;
        }
    else {
        say {interactive} $title  ;
        say {interactive} $subject ;
        say {interactive} defined $warnings->{$key} ? 1 : 0 ;
        say {interactive} 'has spoken: ' . $has_spoken ;
        if ( ! $has_spoken ) {
            say_message( { message => $settings->{message} , title => '' } ) ;
            }
        notify(
            {   title   => $title ,
                message => $subject ,
                icon    => '/home/jacoby/Dropbox/Photos/Icons/mail.png' ,
                }
            ) ;
        }
    $has_spoken = 1 ;
    return ;
    }

warn() is useful for debugging, but the work of the program is done in alert_and_store_mail(). $client is the IMAP connection, and $msg is the message itself. I find that I have to send both. I might be doing it wrong, though.

And here is where my modules come in. is_locked() returns a boolean, depending on which way you lock your screens. say_message(), pushover() and notify() share a format, a hashref containing title and message. say_message() tells me that I have notifications coming, and they show up on my desktop. And if I'm away from my desk, they show up on my tablet because of Pushover.

I'll put this into a repo on GitHub, including all the modules. I would like to get this into shape to be something like App::imap_warn or the like, but I'm not there yet. I'm sure there's interest, because default notifications suck.

2016/04/08

Purdue Perl Mongers - April 13 - "Starship Mongers"

I wrote a quick five-minute counter in Javascript just for this
I don't think I've mentioned it here, but I'm one of the core members of Purdue Perl Mongers, which I've wrangled into a SIG of Greater Lafayette Open Source Symposium (#GLOSSY) to try to reach out to others in the Open Source community.

I was going to talk about DBIx::Class and how it connects to Dancer, but I haven't learned nearly enough about DBIx::Class to talk about it, and didn't have enough open days to come up with a decent presentation, so I punted.

Thus Starship Mongers!

It's a variation on "Lightning Talks", which give speakers a strict five-minute window, but because we're a small group, I decided to add a wrinkle: "Everybody Talks! No one quits!"

(The next part of the quote seems a little too tough for a user group.)

This means that I intend that everyone should talk for five minutes on something. Doesn't have to be Perl. Doesn't have to be programming, or computing, or open source. Just has to be something you are interested in or have questions about. (But, remember your audience.)

I hope that this will charge up the group, bring up ideas for upcoming meetings. If nothing else, it'll give me time to get up to speed on DBIC.

2016/04/04

Diagnosing A Problem: OddMuse

I work in a lab in a large research-centered university. We use a wiki to serve as our lab notebook where we keep notes about the samples that go through. We're also a Perl shop, so we went with a Perl-based wiki named OddMuse (a fork of UseMod). This has been our platform of choice for nearly a decade.

Today, it was reported that a few pages would hang during loading. They gave up after 300, as we have a 5-minute timeout in our Apache config. I shame myself by saying that I went to the OddMuse IRC channel before I looked at /var/log/html/error_log, but that is what happened. The error log reported:  [Mon Apr 04 13:11:54 2016] [error] [client 142.68.31.21] (70007)The timeout specified has expired: ap_content_length_filter: apr_bucket_read() failed. I'm not strong in my Apache Fu, but I'm pretty sure this means that we hit the timeout, but it doesn't really say why we hit the timeout.

Which brings up a weirdness. Imagine example.com/wiki/SandBox is the page in question. You can get the page in it's full glory, before it's turned into HTML and spat out, at example.com/wiki/raw/SandBox, and that page always loads fast.

I "solved" the issue by editing and saving the file. I still don't know what's going on so that OddMuse can handle the data in raw form but cannot convert it to HTML. I am currently going down two roads of thought. The first is that there was a filesystem issue. We're working on a filesystem that is amazing in it's redundancy, size and the sheer number of connected nodes, but on occasion, we hit points where it falls down, giving us a several minute wait for commands such as ls or clear to run.

The other thought relates to how we actually use OddMuse. We wanted to have a front-end that behaved nearly like Word, so we use CKEditor and save HTML instead of wiki markup. At first, we saved samples in groups of up to 10, writing them to an unordered HTML list, but now we're pushing 400. The stub of each wiki page is created programmatically, and I am thinking that the higher numbers might be more than it can take.

This, I guess, gives me a thing I can test. Write a thing that starts with, say, 100 elements in a list, then builds it up until the page doesn't render. I can do that. And I will do that tomorrow, because it's after 5pm today.

Taking the Great Leap Forward with Dancer2

I am working on understanding a raft of technologies, including Dancer and Bootstrap, in order to make our web presence look more current and, more importantly, be more maintainable. 

I'm learning a lot, which is not the positive statement that it sounds like. Rip Van Winkle certainly learned a lot after he slept for twenty years and woke up in post-Revolution America. 

For most of my time as a web developer, when I needed to do authentication, I did it with Apache's built-in server authentication. The number of users I needed to handle was always small enough, and except for a few things where it was entirely for me, I was not the person in charge of creating and maintaining the password system.

I know and believe in a few points. I know that I as admin of a system should not have access to the plain-text passwords of the users. I know that it is common to have two password fields when creating/changing passwords, to ensure you have the right spelling. I'm not 100% bought into that one, but I understand it. I know you keep an email address for "Forgot Password" systems can use your email system as a factor to ensure you're authorized to change that password. And I know you should use encryption systems created by experts, rather than roll your own and create a system that's full of holes. 

I've been using Dancer2::Plugin::Auth::Extensible, trying to get the parts I'd want for a generic system before working on things that I'd want for the lab, and there's much I'm comfortable with. I can get people logged in. I can set roles and limit access to users with specific roles. I can store the date of the last login, which might be useful. And it's all backed by MySQL, which means that, even without an admin dashboard, I have the skills to change anything about a user profile that needs changing.

But we don't want that. We want the techs and the users to have the ability to set values for the user, if for no other reason than I want to be able to move on to other things. So I need to figure out, as a standard, how these things go together, so I can try to implement it. I have things that I'm getting together. I do have questions, though.
  • Clearly, lots consider the repeat-your-password thing as an important part of the password workflow, and clearly, this is a check that I need to at least be able to do. I'm seeing a huge task-duplication thing, because you want to be able to say "passwords don't match" on the client side before the user presses go, but you always want to check things on the server side before you click "submit", because the user might block Javascript. Is this something that Bootstrap can help with? Or will I have to write something like that? I'm willing and able, but with the layout stuff and the way of the future encouraging us to have CSS and JS that's combined and minified and gzipped and included on every page, I'd like to have that taken care of automatically by the framework than go custom.
  • It's not immediately clear in the docs how to enable password encryption. I do need to read that more. (Solved. It was in the docs. I need to read the docs.)
  • I'm hitting the concept of roles and finding that they'd make certain things very useful. I'd like to be able to handle things like unix groups instead, but as is, they allow certain things that will make the end result a lot easier.

    I found, however, that, while the tools to check and control access due to roles are solid, setting and removing roles is less so. I asked the Dancer2 IRC about it, and was told to make a Github issue. I did, and then I wrote something that, within context of my tooling, adds add_user_role and remove_user_role functions. So I have that covered and can move forward.
There's more than this. I could see us wanting a website that has static, CGI and Dancer2 paths, although I think that, when I wake up with a start at 3am, bathed in sweat with a racing heartbeat, this thought is what I was dreaming about. But I'll wait for a while before I have to worry about that.

And, with a parting shot from @perigrin.

2016/02/15

Notes on a Refactor

I prefer to let sleeping dogs lie. There's old, bad code that could be improved, but there's no case to be rewriting old code unless there's a problem with it that breaks a thing. Call it Bug-Driven Development, if you will.

But every bug will have it's day, I suppose.

My lab does science, and in order to not do wasteful, expensive science on things not worthy of the time and money, we do quality assurance, and we display the output to show us and our clients whether the samples are worth using or not. I wrote code to take the output and put it into a form to put online where our techs and our clients can see and use it.

And it sucked.

It sucked, in part, because I have two loops, one to find which GEL image to deal with and one for the EGRAMs. I pull everything into one loop, using grep() to find which GEL and EGRAM to pull for any point.

When I wrote the old code, things like map() and grep() were foreign concepts, and I might think twice about using them if I felt there was a chance that a programmer that wasn't me would have to maintain this code. The new solution is not so much shorter, but I feel it is more straightforward.

It could be better. I felt the call, for example, to redo this in Template Toolkit, pulling away from CGI.pm, but that struck me as a waste until we move forward on a wholescale transfer from CGI to Dancer or another framework.

Anyway, here's the code.

2015/12/10

Trying to minimize in-page Javascript



An overstatement, but a funny one. When everything on the web was new, we put all our JS code into the page itself, because there wasn't much other choice. We put our triggers into our tags and in script blocks at the top of the page, as we did with our CSS in style blocks.

Eventually, we decided that this is bad. In part, consistency across several pages requires each page to have access, so we pulled our style into CSS files and our code into JS files, and it was good.

And then came tablesorter.

In my lab, I deal with data a lot, and I create many web pages where the data is put in tabular form. This is the use of tables that Zeldman wouldn't yell at me about. We use tablesorter to allow us to do cool things with the tables, and if we were doing it vanilla, we could just make tablestarter.js that reads something like $(function() { $('.sorted').tablesorter() } ), but instead, we parse dates into sortable form and set widget options and all sorts of stuff, which isn't necessarily going to transfer between tables. So, I have a script block that's ever-growing, mostly with config. I've set up a tablesorter_tools library that'll allow me to start pulling code out of the pages, but the config? I will have to find a better solution.

2015/12/03

Working with Perl Critic around Exporter

I am working on a framework for modern web APIs using CGI and old-school Perl objects, inspired by what I learned from perlbrew. This is more or less how the included modules go. This auto-exports every function that starts with 'api_', so that I could write helper_one() and helper_two() with no worries about them being exported, and without having to add api_butt, api_stump, and api_tail to @EXPORT, because it's striking me that the export list follows documentation as a pernicious form of code duplication.

package API_Proof ;use strict ;
use warnings FATAL => 'all' ;

use Exporter qw{import} ;

our $VERSION = 0.1 ;

our @EXPORT ;
for my $entry ( keys %API_Billing_0_1:: ) {
    next if $entry !~ /^api_/mxs ;
    push @EXPORT, $entry ;
    }

sub api_stub {
    # code goes here
    }

I intend to put the whole deal up on +GitHub eventually, but to avoid complication, I'll just get to the point where it's used, below. I'm exporting everything that starts with api_, so it's all available for $api->run() when we get there. (It's all in previous posts.)

#!/usr/bin/env perl

use strict ;
use warnings ;
use lib '/path/to/lib' ;

my $api = API->new( @ARGV ) ;
$api->run() ;

package API ;
use lib '/path/to/lib' ;
use base 'API_Base' ; use API_Stub ;

And here is where we run into Perl Best Practices and Perl::Critic. I've been running some modules through perlcritic -1 for sake of being complete, and it lead me to do some changes, and there's one that is keeping me from being clean. It's that I'm using @EXPORT. I should be using @EXPORT_OK or %EXPORT_TAGS instead, it says. This means, that first code block should have something like this instead.

my @export ;
for my $entry ( keys %API_Bard:: ) {
    next if $entry !~ /^api_/ ;
    push @export, $entry ;
    }

$EXPORT_TAGS{ api } = [ @export ];
our @EXPORT_OK = ( @{ $EXPORT_TAGS{ 'api' } } ) ;

And then use API_Stub qw{:api}. I am not quite convinced. I'm open, though. I guess I would just like to know what the problem with export by default is, but this isn't in PBP.

2015/12/01

Thinking Aloud about Testing a Module

I have a module that handles getting a DBI object connecting to MySQL databases. With some work, it could be made to talk to others, but since I don't really have any Oracle or PostgreSQL databases to connect to, it hasn't been worth it.

I have another module that handles calling the first module, managing credentials so they don't show up in code, getting and holding the DBI object and doing the actual calls to the database. This way, I can set a variable saying this DB is named 'foo' and say "I want an arrayref containing the result of this query" without dealing with all the nooks and crannies of DBI.

I now have a module that uses that second module to do work. Get information from the database, throw it into an object and kick it out to be converted to JSON later. And I want to write tests for it. But I have thought and continue to think that having a test that queries a database in a way that requires a dynamic resource to be static is stupid.

Is the way to test it to feed in a "this is a test" flag somewhere, stick a bunch of canned output into the functions to output if the flag, because what I should be testing here is that the module dumps things the right way, right?

2015/11/12

Considering committing Clever with jQuery XHR code

A code snippet

    if ( protecting_namespaces.data.hasOwnProperty( 'xhr' ) ) { 
        submission.data.xhr.abort() ; 
        }
    protecting_namespaces.data.xhr = $.ajax({
        url: url ,
        data: object ,
        method: 'POST',
        dataType: 'json'
        }) ;

Seems that Syntax Highlighter is happy with JavaScript. OK.

I'm using jQuery's .ajax() interface, and what this does is, should the function that does this get called again, aborts an unfinished AJAX call. This is wonderful when this will be called from one place, but it sucks if you call it from several places to do a few different things.

And, by wonderful, I mean useful for the UX. If you tell the web server to do a thing, you cannot go back and say "No, I didn't mean it." You can tell it "Do something else to get us back to the first state", but you cannot stop it once it has started, unless you're logged into the server and can kill -9.

So, I am considering making xhr into an object, based on a unique identifier for what's actually being done, which should give me protecting_namespaces.data.xhr[id], so I could have several XHR tasks going on in parallel.

2015/11/06

Trying to Read MongoDB dump files using BSON

I've been looking at increasing the amount of MongoDB we use at work, and this includes backing up the data. Due to my own confusion, I had a little issue getting mongodump to work, but I have been able to dump from one Mongo database and restore to another.

mongodump writes in a format called Binary JSON, or BSON. I installed BSON.pm with the intention of reading the BSON file and ensuring it works. with small tests, I was able to put objects into BSON, write to file, read from file, and use Data::Dumper to display the object is what I wanted it to be.

But, I find I cannot read the file, because BSON.pm reports it as having an incorrect length.


I fully expect that I'm doing something subtly stupid, but what it could be isn't immediately obvious. the contents of $bson should be exactly as written to file, and mongorestore was happy with it. encode() and decode() had worked acceptably, but admittedly on a much smaller dataset than the one I'm working with here, which contains several months of status updates.

I suppose I don't need to read the BSON files, but I do like being able to check the contents of a database before restoring.