But Wait! There's More! Extendable DRY API Code

The previous post wasn't built in a vacuum.

I was chasing down ideas for a purpose. I've been building JSON APIs for the lab for quite some time, each with as much knowledge about them as I had at that time, which means I have a number of bad APIs out there, and I'm in a position now where I'd like to unify them into either task-specific or access-level-specific groups.

That last piece of code was me finally able to take the lesson I learned from perlbrew about the symbol table being usable as a dispatch table. But it had a problem.

That problem was that, to use it, I would have to use or require whatever module I want to access into that one module. This is a problem because then I'd need a different API.pm for each version of the API I wanted to make. This is not good.

I remembered code that I used to use to download Dilbert comics. I subclassed HTML::Parser, adding functions that were specific to the task of grabbing images. Which is exactly what I do here.

#!/usr/bin/env perl
# the application code

use feature qw{ say } ;
use strict ;
use warnings ;
use lib ;

my $api = API->new( @ARGV ) ;
$api->run() ;

package API ;
use base 'API_Base' ;
use lib ;
use API_PED ;

Above, I make a version of API that includes API_PED.

API_Base is conceptually very perlbrew-influenced, but I totally dropped the "I think you mean" because this is a JSON API, not a command-line program.

package API_Base ;
use feature qw{ say } ;
use strict ;
use warnings ;
use CGI ;
use Getopt::Long ;
use Data::Dumper ;
use JSON ;

# Yes, I still use CGI. I kick it old-school.

sub new {
    my ( $class, @argv ) = @_ ;
    my $self ;
    my $cgi = CGI->new() ;
    %{ $self->{param} } = map { $_ => $cgi->param($_) } $cgi->param() ;
    ( undef, @{ $self->{pathinfo} } ) = split m{/}, $cgi->path_info() ;
    return bless $self, $class ;
    }

sub run {
    my ($self) = @_ ;
    $self->run_command( $self->{pathinfo}, $self->{param} ) ;
    }

sub run_command {
    my ( $self, $pathinfo, $param ) = @_ ;
    my $command = $pathinfo->[0] || 'test' ;
    my $s = $self->can("api_$command") ;
    unless ($s) {
        $command =~ y/-/_/ ;
        $s = $self->can("api_$command") ;
        }
    unless ($s) {
        $self->fail( $pathinfo, $param ) ;
        exit ;
        }
    unless ( 'CODE' eq ref $s ) { $self->fail( $pathinfo, $param ) }
    $self->$s( $pathinfo, $param ) ;
    }

sub fail {
    my ( $self, $pathinfo, $param ) = @_ ;
    say 'content-type: application/json' ;
    say '' ;
    say encode_json {
        status   => 'fail',
        param    => $param,
        pathinfo => $pathinfo
        } ;
    print STDERR 'INAPPROPRIATE USAGE: '
        . 'desired path = '
        . ( join '/', '',@$pathinfo ) ;
    }

sub api_test {
    my ( $self, $pathinfo, $param ) = @_ ;
    say 'content-type: application/json' ;
    say '' ;
    say encode_json { status => 1, param => $param, pathinfo => $pathinfo } ;
    }
1 ;

We go into the symbol table twice. one to export api_PED, which would become api.cgi/PED, and one to make subroutines named ped_* into a dispatch table, allowing api.cgi/PED/test, api/PED/mail and api.cgi/PED/lookup.

package API_PED ;
use feature qw{ say } ;
use warnings ;

use Exporter qw{import} ;
use JSON ;

# PED is our campus LDAP server
use lib '/depot/gcore/apps/lib/' ;
use PED qw{ purdue_ldap_lookup } ;

our @EXPORT ;
for my $entry ( keys %API_PED:: ) {
    next if $entry !~ /^api_/ ;
    push @EXPORT, $entry ;
    }

# The goal here is to do as much as we can without repetition
# we export api_PED so that API accept this as a command without
# having to write it into API.pm

# api_PED checks for any subroutine starting with 'ped_' and
# runs it

# so in essence, exporting a sub starting with api_ adds it to the
# API dispatch table, and writing a sub starting with ped_ adds it
# to this module's dispatch tableee 

sub api_PED {
    my ( $self, $pathinfo, $param ) = @_ ;
    my %commands ;
    shift @$pathinfo ;
    foreach my $entry ( keys %API_PED:: ) {
        next if $entry !~ /^ped_/ ;
        $commands{$entry} = 1 ;
        }
    my $sub_name = shift @$pathinfo ;
    my $command  = 'ped_' . $sub_name ;
    if ( $commands{$command} ) {
        my $full = join '::', 'API_PED', $command ;
        &{$full}( $pathinfo, $param ) ;
        exit ;
        }
    else {
        say 'content-type: application/json' ;
        say '' ;
        say encode_json { c => \%commands, p => $pathinfo, e => $command } ;
        }
    }

sub ped_test {
    my ( $pathinfo, $param ) = @_ ;
    say 'content-type: application/json' ;
    say '' ;
    say encode_json { result => 1 } ;
    exit ;
    }

sub ped_mail {
    my ( $pathinfo, $param ) = @_ ;
    my $name   = $pathinfo->[0] ;
    my $lookup = purdue_ldap_lookup($name) ;
    say 'content-type: application/json' ;
    say '' ;
    say encode_json {
        status => ( scalar keys %$lookup ? 1 : 0 ),
        mail => $lookup->{mail},
        } ;
    exit ;
    }

sub ped_lookup {
    my ( $pathinfo, $param ) = @_ ;
    my $name   = $pathinfo->[0] ;
    my $lookup = purdue_ldap_lookup($name) ;
    say 'content-type: application/json' ;
    say '' ;
    say encode_json {
        status => ( scalar keys %$lookup ? 1 : 0 ),
        lookup => $lookup,
        } ;
    exit ;
    }
1 ;

I'm not the happiest. Each sub handles encoding the output itself. I normally use JSON, but I could imagine exporting JSONP, XML, CSV or something else, and I could imagine passing back the data and an indication as to how it should be handled. I think I might have that somewhere, like I had the base code in a web comic reader from the late 1990s.

To sum up, I'm in the middle panel of this:

Three Panel Soul. I think the artist turned this into a t-shirt.

Filling a Dispatch Table For Ease of Development and DRY

I hate magic.

Don't get me wrong. If the night's entertainment is going to see Penn & Teller, I'll enjoy myself like anyone else, but when I have to work with it, I hate magic.

I hate magic because there's an implicit promise of "this will always work", and when it doesn't work, because eventually it won't, I won't know how to fix it because I was never told how it works in the first place.

On the other hand, I hate repeating myself.

Consider this code:

package Me ;
use Exporter qw{import} ;
our @EXPORT =  qw{add} ;

# adds two variables
sub add {
    my ($i,$j)=@_;
    return $i + $j ;
    }
1;

It was recently suggested to me that documentation like the above is essentially code duplication, and if I made substantial changes, assuming a more complex concept than addition, then the comments are out of date. And because writing in code and writing in English are different, switching between the two can be taxing, so developers often say "I'll get to it later" and forget.

But there's another point of code duplication.

Not only are we saying add() is a subroutine with sub add(), we're saying it with our @EXPORT = qw{add}. I don't want that code repetition. So, what can we do to make it better?

Look at the symbol table!

package Me ;
use Exporter qw{import} ;
our @EXPORT ;
for my $entry ( keys %Me:: ) {
    next if $entry !~ /^my_/ ;
    push @EXPORT, $entry ; 
    }

# we do our own addition because I don't trust the built-in
sub my_add {
    my ($i,$j)=@_;
    return $i + $j ;
    }
1;

I've changed the documentation to why I've done such a silly thing as re-implemented add(), given add() a name much more likely to avoid namespace collisions, and made @EXPORT auto-filling. If I wanted to created my own subtraction, I'd just write my_sub() and away we'd go. Additionally, if I wanted a non-exported subroutine log_usage() to add to these subroutines and see how often my code is used, I could just write and use that and it won't be exported.

But, because I see how @EXPORT is filled, it will be apparent when I return to maintain this code how that works, so something else will be the biggest "WTF?" in the code base. No magic. No Three of Clubs. No Jack of Spades spitting cider into my ear. More than likely, it'd be "Why did I roll my own buggy addition? What was I even thinking?"

Which is good. Because, while I love clever, I hate magic.

Now I Am A C# Developer

It started with Instagram.

I wanted to be able to have my most recent Instagram picture become my background.

For my Android, tablet, that was easy. I just used IFTTT to connect the Instagram channel to the Android Device channel and that was that.

I wrote a Perl module, which I should put on GitHub and maybe then CPAN one of these days, that interacts with the Instagram API, and wrote a program that downloads the latest and uses gsettings to set it as background. That was easy, and I suppose I could use Strawberry Perl to make it all work on Win32, but I thought "Hey, Microsoft just told me that Visual Studio Community was free to use, and I just moved my laptop to Win10, so why not learn a thing?"

So, I learned a thing. I have used Visual Studio to write four C# programs:

• Hello World, because you need to learn how to do I/O and get the compiler to run
• Iterative Fibonacci, because you need to understand how to use control structures in a language
• Recursive Fibonacci, because you need to understand how to use subroutines in a language
• SetBackgroundImage, which used inputs and SystemParametersInfo to allow me to type a path to an image and set that as my background.

Before: picture of my whiteboard

After: another picture of my whiteboard

The only part of the source that I had more than minor interaction with is Program.cs, and that isn't much. I'm not a fan of default bracing style, having my own preferences, but knowing that Ctrl-K Ctrl-D (I think) will tidy whatever I write into correct formatting is a good thing.

I'm a vi man, so living la vida IDE is a stretch for me. I had to Google to find out how to turn on line numbers, but to be fair, I was not born with the knowledge of :se nu, either, so that's fair. I generally use Windows machines that give me a browser and PuTTY, which let me connect to real computers, so this is me learning how Windows does Windows.

Anyway, that's my code, up there on GitHub. If you have suggestions for program #5, I'm all ears.

Perl Module for interacting with Globus

Ever heard of Globus?

No?

I'm kinda not surprised. It's a bit of an obscure thing.

Imagine you have research data. A collaborator wants to analyse it. Thing is, you're where you are and the collaborator is on the other side of the continent. There are many ways to move things, but many of them involve you having to be fairly technically sophisticated in order to make them work. Or your collaborator, who is an expert in your domain but not necessarily in file transfer. Plus, there's permissions and security and all that.

Or, you can use Globus.

We're changing how we handle data sharing here in the lab. We just moved file systems, moving to one that supports access control lists (ACLs), which means that, instead of playing games with symbolic links in order to share a researcher's work, we could simply setfacl -R -m 'u:user:r-x' and go from there.

That was fun, but because an ACL has to be set for every file, changes take a long time, so testing takes a long time, and updates take a long time. But it means that local researchers have access on the Big Iron.

But not everyone we want to share with is local. Some are on the other side of the continent. Thus Globus.

And, soon, Net::Globus.

It uses their first API, which is exposed via SSH. This is not my preferred interface, but I was able to get it to do the subset I needed fast enough, and I'll go back and make a thing work with their REST API in my copious free time. I mean, I could just use the Python API, right?

If you were to take my Globus.pm and copy it into your ~/lib directory, and you had the right stuff set up with SSH keys, it'd work. For the subset that isn't stubtastic, it's there.

I started of this project with dzil new, which is as much of Dist::Zilla as I know how to use. (I didn't start this paragraph intending an Elvis Costello reference, but there you go.) I know how to use git, how to make pull requests and merge them. My contribution graph is more white than green, but GitHub is not integral to my workflow, containing more my toys than my work. So, I'm pretty close to OK for the managing of a repo, but the steps toward getting it onto CPAN will involve me reading and asking a lot of stupid questions irc.perl.org.

Plus, the tests. Oh, man, the tests....

Perl is good because Perl is tested, and the tests I know how to write are tests I consider stupid. So, I expect to spend some time in chromatic's book of testing and in the tests for Net::Twitter and the like, trying to figure out what needs to be tested and how to write them.

So, I'm not ready to put it up to CPAN yet, but I see the path, because I finally have a module I could imagine someone else needing.

So, if this is a thing you think you have a use for, try it. If you think I did something stupid, I'm sure I did. Suggestions are great; pull requests are better. And thanks.

I Need To Write This Up To Understand It, or The Epic Battle between Wat and Derp!

I've signed up for Neil Bowers' Pull Request Challenge, as I've blogged about before.

This month, I'm working with System::Command. It's neat, a step up from system calls, and as I understand it, integral to handling Git programmically, within Git::Repository. If I wasn't elbow-deep, I might try to use it as part of my Net::Globus project.

But the challenge isn't to use a module, it's to commit changes to a project. I wrote book, and he suggested writing a test to catch a bug that shows up in Windows. There's lots of "what even?" that's been going along with this, the great majority has a lot to do with the difference between coding for your self and your lab and coding for a community. I mean, I really do not get what's going on here, and you know that old joke? "Doctor, it hurts when I do this!" "Don't do that, then!" I keep thinking "If this is breaking your code, do something else!"

But, that sort of thinking doesn't get your code merged. So, I'm proceeding, my mind becoming the ring for an epic cage match between Wat and Derp.

One of the points of Derp (thanks #win32 on irc.perl.org) is that warnings are not exceptions. Since the error as given has a distressingly consistent ability to lock up Powershell session, I hadn't thought of that. It's lead me to think about cutting back to the least amount of code you can include to get a behavior you want to test for. In this case, it isn't just giving the wrong result, it's kicking up a lot of errors. Or, rather, warnings.

Among the questions on the $64,000 Pyramid are:

• "Why does Git Bash shell show the IPC::Run::Win32Pump errors, after the next prompt shows up, while Powershell doesn't?" 
• "Is that significant?"
• "What, then, is killing the great Powershells of my laptop?"
• "How do you trap that?" 
• "Isn't that more a test issue or a program issue?"
• "How, then, do you turn this mess into a meaningful test?"

That last one is what I'll be sleeping on tonight. On the one hand, turn this into a .t file and any warning, any at all, would be enough to trigger the test, as it's demonstrated that no errors occur in Linux. On the other, a test that says "Yeah, this kicks up errors" seems less-than-helpful, so specifying the GEN7 and GEN11 seems like the kind of thing that could break on another system.

I suppose I could write both a general and specific error.

Any comments, either supportive ("Yeah, that's exactly the kind of test you need!"), inquisitive ("Can I use BEGIN blocks in my other work?") or dismissive ("_That_ is a stupid way to test this out!") will be read and learned from. 

Meanwhile, I wrote this after my bedtime, so good night.

I Am Unlikely To Move To Perl6

A camel in Busch Gardens in St. Louis.

I'm working on a project, which I will blog otherwise. This project is my hope to get into CPAN and level up my Perl game. I am still in progress, so it sits in the earliest stages of construction, with the bare minimum calling dzil new Foo::Bar will give you.

A friend asked me to rewrite it in Perl6.

He has also started putting out challenges, saying "Do this task in Perl5 without modules and I'll do it with Perl6.

I have several problems here. This project started with code I wrote to do something for work, and first and foremost, it had to do the work thing. That one part of the code could be useful to others is a perk. We're a Perl5 shop, so spending more than a few moments trying to do this task in Perl6 is just as counterproductive as trying to do it in Python, in Haskell, in Visual Basic or in x86 assembly. The library doesn't do everything, but it does everything I need, and the move from being a Perl developer to a CPAN developer is steep enough, thanks.

As for the challenges, I find them fundamentally unjust. If you have language A and language B, where feature X is available built into A but exists as a module in B, a module that comes with the core distribution of B, and this might not be a thing you use in most programs, then, to me, A puts unnecessary code into memory, while B is respectful of your requirements by not including things you don't need. My friend thinks the inclusion of feature X into A is a thing to be celebrated, a selling point he's trying to use to build excitement, while I regard it as a stacked deck.

But even as I write that, and intellectually agree with the logic I created, it isn't my fundamental objection. I could probably make a reverse argument if the tables were turned, and in the age where we have multiple gigabytes of RAM, and machines I have access to have a quarter-terabyte available, being so parsimonious with memory is out of style. As the saying goes, you cannot reason a person out of a position they didn't reason themselves into.

I started out as a Computer Science student, getting a second degree because my Journalism degree gave me no job. My school taught C as a subset of C++, using gcc on Solaris systems. This involved the use of stdio.h, and strings as arrays of characters. Intellectually, I get that it still is, but as a practical matter, it was not easy and it was not fun.

I worked helping to maintain the websites for a part of the school, and at first wrote HTML and documents in HTML, and also maintained the documentation library for the system administrators. Physical library. This is where I learned to love O'Reilly and Amazon. And, as I learned to program by learning CGI by learning Perl, where I learned that dealing with strings as scalars instead of arrays pointers to arrays of characters was a lot easier. This very much was where I learned to love Perl5.

This was an age of underpowered computers, and the computers I worked with were, in general, more underpowered than most. So weak, actually, that they could not run a web browser, which makes working as a web developer difficult. This is why I learned about X redirection. And, the minute I added any Javascript code to do anything on a page, this meant that the pages became unusable on the (for the time) beefy servers I was running the browsers on. I had reasoned myself into the position "pages + Javascript = slow and unusable".

Two things changed. The first was Time + Moore's Law = better computers. The second thing was the discovery of XMLHttpRequest, aka XHR or AJAX. This meant that Javascript was capable of more than display hacks, that you could do actual things with it. So, I grew to like Javascript, almost but not quite to the extent I like Perl.

A few years later, I had graduated and been hired by a local medical clinic. This was more IT than development, and I should've left early than I did, but while I was there, many of my friends had started to move to Python.

There was a computer lab where I had spent a great amount of time, and someone I knew had code that would identify who was at which SPARC5 machine, and which machines were empty, so you could know if it was worth your time to even go to that part of campus. I didn't need that information anymore, but what I needed was working Python code so I could learn how the language worked.

However, the program was indented with tabs. Python, for those unfamiliar with the Dynamic Language Wars, is a language where control structures are determined by indentation, instead of brackets, like most C-related languages. "Significant White Space" is the Ypres of the war between Perl and Python, the point where neither side will give ground.

The problem with tabs is that they tend to hide spaces. Which is to say, unless you have your editor set to show you the difference, SpaceTab looks exactly like Tab. This was exactly the problem that I found myself having. I was expecting, even wanting, to learn how if statements and for loops look in Python, and instead I found myself having to learn how Python error messages work.

The argument for white space is that, if you allow code to look like a one-line hairball, coders will make it look like a one-line hairball, but if your language design enforces coding standards, you will have readable code. I like reading readable code, and, as I said, I came from a Journalism background where I learned the rules to make printed documents more readable. It should be easy to reason me into appreciating Python, but having spent years learning Perl made me defensive, and spending hours struggling with with code entrenched my disgust. I viscerally dislike Python. I use it; my FitBit code is written in it because I found working through OAuth in Perl to be difficult. I'm willing to suck it up when necessary. But it is not my first choice of languages, and likely will never be, for entirely emotional reasons.

When I first started hearing about Perl6, I liked the idea of it. I wanted it to succeed. Perl is complex, they said. Only Perl can parse Perl, which isn't good. To hack Perl, to improve the base language, you need to know a bunch of domain-specific hackery, which means that the number of people who can make meaningful changes to the language is small and getting smaller. If we back up and start again, we can remove a bunch of cruft. This is fine.

And, since we're starting from scratch, we'll identify things we want, things that'll make the language better, and build those in. Logically, this works.

The brains of the Perl community went to work on Perl6, on the various platforms they tried to build it on. I can't create a strong argument against any individual technical decision. Use a VM to make it portable across many languages? Sure. Make our own VM system so we aren't reliant on someone else with interests unaligned with our own? Yeah, that's probably smart. But taken as a whole, that's a lot of things you need to do, with a significant chance of failure. The charitable might take it as a gamble worth taking, as a thing worth trying, even if it doesn't work right.

Not everyone is that charitable.

Meanwhile, those of us trying to use the language to do things to do things were left with, basically, a dead language. That is, until a bunch of us started to say "No!" We decided that, as we can, the good things about Perl6 will be taken and backported to Perl5, and that, while the world may identify the moment Jon Orwant's coffee mug hit the wall as the last moment they gave a shit about Perl, we have an enormous collection of pre-existing wheels called CPAN, which is considered the standard by which other languages' library systems are judged and found wanting.

So, to me, Perl5 is the language that saved me from C-type strings, the language that opened up computing for me and that has provided the majority of my paychecks for well over a decade. Perl6 is the language that has made me feel like a laughingstock for a good portion of that time. Perl5 is the a language that supported me, and Perl6 is the language that betrayed me.

Intellectually, I see that Perl6 has a real release date, that we're one community, that Perl6 isn't just a skunkworks for features to pull into Perl5. But I do not foresee my current workplace moving from Perl5 to Perl6, and I do not expect most place I might get hired into to be enthusiastic in embracing the change either.

I'm willing to change my mind. This is me trying to work through my issues enough to be open to Perl6. But it remains to be seen if I can change my heart.

What Language Should I Learn? Three Answers

A friend of mine, who works in IT but is not a developer, asked me a question during lunch today.

"I want to learn to program. What language should I learn first?"

This is a common question, one I have answered before. But, because I've blogged a lot recently, I've decided to write it up and post here.

I told him I have three answers.

The first is somewhat sarcastic. "I'm going to Europe. What language should I learn?" There just is not enough information to really answer that question, because many languages are very context-specific. If you're hoping to get into programming apps for the iPhone, your best choice is Objective C. If you want to code for Arduino microcontrollers, you'll want to start with the Arduino IDE and it's very C-like language. And, of course, there's JavaScript, your only choice on web browsers.

Where you want to go determines what you should learn.

But there's more to it than that.

There's a thing called the Church-Turing Theory, which states that any real-world calculation can be computed. Turing postulated using a Turing Machine, while Church referenced the Lambda calculus.

We get to a concept called Turing Completeness. A thing that can be computed in one Turing-Complete machine can be simulated in another Turing-Complete machine. The first real use of this was the creation of compilers, of higher-level languages that developers can use which compile to machine code that the hardware itself can run. What it means, for those learning, is that it doesn't really matter what language you learn, that anything one language does can be done by another language.

So the second answer is, Alan Turing would tell you it just doesn't matter which language you choose, that what you do and learn in one language can be simulated or applied in another language. So it doesn't really matter which you choose.

When Jeff Atwood of Coding Horror coined Atwood's Law -- any application that can be written in JavaScript, will eventually be written in JavaScript -- he didn't know the half of it. He knew that graphical applications were starting to be done within web browsers, like Gmail, He didn't know that web server applications and even command-line applications could be written in JavaScript via Node.js. He didn't know that a framework for creating cross-platform mobile applications using web technologies including JavaScript called Cordova would come along. He didn't know that Microsoft would allow developers to create Windows applications using HTML and JavaScript. He didn't know that Open Source microcontrollers such as Arduino would be developed, and frameworks such as Johnny Five would come to allow you to develop Internet of Things projects and even robots with Javascript. It might be a bit more complex to set it up to do these things with JavaScript, but they are possible.

Plus, if your code plans are more functional and computer-theoretical, you'd be glad to know that JavaScript is a Lisp.

If you want to code Objective-C, you need a Mac and the Apple development tools. If you want to code C#, you'll need to install Visual Studio tools from Microsoft (or Mono on Linux). If you want to code JavaScript, you'll need a text editor (and one comes with your computer, I promise) and a web browser (and one comes with your computer, I promise), plus there are places like CodeBin where you can enter your code into the browser itself .

If you're going to be writing an operating system, device drivers, you will want something that compiles to native machine code. If you're looking to get into a specific project, you'll want to know the language of that project. But the corners of the development landscape where JavaScript is the wrong choice are small and shrinking. So, the third answer is, it might as well be JavaScript.

This rubs me a bit wrong. I've set my rep as a Perl Monger, and I always feel like that's where you should start. But while my heart feels that, my mind argues the above, that the greater forces of modern computing are pushing to give JavaScript a front-row seat in the language arena.

But I'm willing to be wrong, and if I am, I want to know. Where am I wrong? What would you tell someone wanting to learn to program?